Achieving Grid Interoperability The ETSI Approach

1. Achieving Grid InteroperabilityThe ETSI Approach Stephan Schulz, Ph.D. ETSI Centre for Testing and Interoperability

2. Presentation Outline • On achieving interoperability • Conformance versus interoperability testing • ETSI test specification development • Requirement extraction • Test purposes • Conformance Tests • Interoperability Tests • Creation of test specifications at ETSI • Centre for Testing and Interoperability • Specialist Task Forces • On grid testing • Grid STF • Review of Byte IO and Grid RPC test specifications

3. Interoperability is … • The ultimate aim of standardisation in Information and Communication Technology • The red thread running through the entire standards development process, it’s not an isolated issue • Not something to be somehow fixed at the end • ETSI approach • ETSI produces with the required degree of parallelism Base Standards and Test Specifications • Base standards should be designed with interoperability in mind • Profiles can help to reduce potential non-interoperability • Two complementary forms of testing • Conformance testing • Interoperability testing (more formal IOT”) • Testing provides vital feedback into the standardization work

4. Device Reqs. About Conformance Testing • Is Black-Box testing • Stimulation and Response Point of Control and Observation (PCO) TestSystem

5. IUT = Implementation Under Test Upper Tester MMI API L3 L3 Test SUT L2 System d i g i t a l PHY 1 2 3 4 5 6 7 8 9 8 # * Lower Tester SUT = System Under Test Conformance is Usually Tested Layer-by-Layer Upper Adapter L3 Test Lower Adapter i i t a l

6. Characteristics of Conformance Testing • Tests IUT against requirements specified either in a base specification or profile (standard) • Gives a high-level of confidence that key components of a device or system are working as they were specified and designed to do • Usually limited to one requirement per test • High IUT control and observability • Can explicitly test error behaviour • Can provoke and test non-normal (but legitimate) scenarios • Can be extended to include robustness tests • Test execution can be automated and repeated • Requires a test system development (and executable test cases) • Can be expensive (e.g., radio-based test system) • Tests under ideal conditions • Tests are thorough and accurate but limited in scope • At level of detailed protocol messages, service primitives, or procedure calls

7. Limitations of Conformance Testing • Does not prove end-to-end functionality (interoperability) between communicating systems • Conformance tested implementations may still not interoperate This is often a specification problem rather than a testing problem! Need for minimum requirements coverage or profiles • Tests individual system components • But a system is often greater than the sum of its parts! • Does not test the user’s ‘perception’ of the system • Standardised conformance tests do not include proprietary ‘aspects’ • Difficult to test via proprietary interfaces, e.g., user APIs • Can however be done by a manufacturer with own conformance tests for proprietary requirements

8. Device2 Device1 Devicen About Interoperability Testing • End-to-End Interoperability of devices • What is being tested? • Assignment of verdicts? • Need to distinguish between • More formal interoperability testing(= test specifications) versus • Informal interoperability events used to validate/develop technologies (e.g., ETSI Plugtests like events)

9. 1 2 3 1 2 3 4 5 6 4 5 6 7 8 9 7 8 9 8 # 8 # * * MMI MMI SUT API API QE = Qualified Equipment EUT = Equipment Under Test L3 L3 L2 L2 PHY PHY ETSI Methodology for Interoperability Testing QE EUT Means of Communication (MoC) Interoperability Testing ( of terminals)

10. Characteristics of Interoperability Testing • It is system testing • Tests a complete device or a collection of devices • Shows that (two) devices interoperate • But within a limited scenario • Tests at a ‘high’ level (perception of users) • Tests the ‘whole’, not the parts or layers • E.g., protocol stacks integrated with applications • Tests functionality • Shows function is accomplished (but not how) • Does not usually involve test systems • Uses existing interfaces (standard/proprietary) • Interoperability testing looks at end-to end functionality • Less thorough but wide in scope • Gives a high-level of confidence that devices (or components in a system) will interoperate with other devices (components)

11. Limitations of Interoperability Testing • Does not prove that a device is conformant • Interoperable devices may still interoperate even though they are non-conformant to a base specification or standard • Cannot explicitly test error behaviour or unusual scenarios • Invalid conditions may need to be forced (lack of controllability) • Has limited coverage (does not fully exercise the device) • Are executed manually (usually) • Difficult to automate, tests use more time to be prepared than executed. • Does not prove 100% interoperability with other implementations with which no testing has been done • ‘A’ may interoperate with ‘B‘ and ‘B’ may interoperate with ‘C’. But it doesn’t necessarily follow that ‘A’ will interoperate with ‘C’.

12. Interop Testing ConformanceTesting Interoperability Conformance and Interoperability Testing are Complementary • ETSI experience • As you move up a system stack the emphasis should change from conformance to interoperability testing • Lower layer protocols, infrastructure • Emphasis on conformance • Middleware, enablers • Combination of conformance andinteroperability testing • Services, applications, systems • Emphasis on interoperability testing • Conformance testing can be viewed as a pre-requisite to interoperability testing

13. Iterative feedback Interoperability Testing Industry (Unit) Conformance Testing Interoperability Test Specifications Iterative feedback Conformance Test Specifications Standards Body Typical ETSI Standards Development Process Certification Products mature from prototypes to commercial products time ETSI: Development of base standards Testing provides vital feedback, e.g., holes, errors, or inconsistencies, in base standards

14. Stages in ETSI Test Specification Development Base Standard or Profile Specification Requirement Catalogue Implementation Check List (ICS/IFS) Identification of Test Group Structure (TSS) Specification of Test Purposes (TP) Specification ofTest Descriptions (TD) Specification ofTest Cases (TC) Validation of Test Cases

15. Requirements Cataloguing • Each requirement is identified, extracted, and catalogued as follows (example from IPv6): • Requirement type: • Mandatory (MUST, MUST NOT, SHALL, SHALL NOT) • Recommended (SHOULD, SHOULD NOT) • Optional (MAY, MAY NOT, COULD) • Requirement role (tested entity): • E.g., Host, Router, Node (Host or Router) • Requirement context • Requirement text • Base document citation and reference • Functional groupings • E.g., Process Fragmented packet, Generate ICMPv6 Error Type etc.

16. Test Purposes • Precise descriptions of the purpose of a test in relation to a particular (base standard) requirement • Meaningful to a larger audience than technical experts • Define the functionality being tested, i.e., WHAT? • Should be formulated using a similar level of information content and wording as the corresponding requirement description • Should only mention directly relevant aspects of interactions but not describe them in detail • Reference a test configuration (i.e., architecture) • It is possible to have several test purposes per requirement • Specified at ETSI in • Natural language, or • ETSI’s Test Purpose Language (TPLan) (www.tplan.info) • But definitely not coded!

17. TPLan Example from Digital Public Mobile Radio TP id: TP_PMR_0406_01summary: 'Header frame acknowledges connect request'TPtype: conformanceRQ ref: RQ_001_0406 –- Catalogue IdentifierIUT Role: CSF-- Configured Service Function (CSF)config ref: CF_dPMR_CSF_01 -- CSF Implementation Under-- Test (IUT) and TESTER with{ IUT in standby} ensure that{when{IUT receives a Connection_Request }then{IUT sends an Acknowledgement_Frame}

18. Interoperability Test Description • Specifies detailed steps to be followed to achieve stated test purpose, i.e., HOW? • State steps clearly and unambiguously without unreasonable restrictions on actual method: • Example: • Answer incoming call NOT • Pick up telephone handset • Written in a structured and tabulated natural language so tests can be performed manually • In theory it is also possible to increase automation in with test scripts, e.g., with TTCN-3 • Proprietary nature of user APIs makes this in practice difficult • Requires additional adapter development from equipment vendors

19. Example Interoperability Test Description Response to ‘ping’:

20. Conformance Test Suite • A collection of detailed test cases or scripts that implement test purposes • Can be compiled and executed, e.g., when using TTCN-3 • Specifies HOW to test • Implements also handling of, e.g., unexpected or background behaviour, or returning the SUT to the initial state after a test • Composition of test cases • Each individual test has a preamble, test body (i.e., implementation of the Test Purpose), and postamble • Test components may be used clearly separate interactions with different logical IUT interfaces during a test • Assigns test verdicts • Configurable at run-time, e.g., SUT address • A test suite is not a test systems, but • ETSI ensures all tests can be compiled • And validated by executing the tests against one or more real implementations on at least one test system

21. What is TTCN-3? www.ttcn-3.org • Testing and Test Control Notation Version 3 • Internationally standardized language developed specifically for executable test specification • Specified by ETSI MTS Technical Committee • Is independent of a specific IUT or IUT interfaces • Is independent of a test execution environment • Standard available at portal.etsi.org via ETSI programme • Allows unambiguous implementation of tests • Look and feel of a regular programming language • Good tool support (today 6 commercial tools available) • Successfully deployed at ETSI and industry in a variety of application domains • e.g., telecom, automotive, software, etc. • Good text book available • http://www.wiley.com/legacy/wileychi/ttcn-3/

22. Example TTCN-3 Test Case testcase TC_COR_0047_01() runs on Ipv6Node system EtherNetAdapter { f_cf02Up(); // Configure test system for HS->RT // No preamble required in this case f_TP_HopsSetToOne(); // Perform test // No postamble required in this case f_cf02Down(); // Return test system to initial state } function f_TP_HopsSetToOne() runs on Ipv6Node { var Ipv6Packet v_ipPkt; var FncRetCode v_ret := f_echoTimeExceeded( 1, v_ipPkt ); if ( v_ret == e_success and v_ipPkt.icmpCode == 0 ) { setverdict(pass);} else { setverdict(fail); } } function f_echoTimeExceeded(in UInt8 p_hops, out Ipv6Packet p_ípPkt ) runs on Ipv6Node return FncRetCode { var Ipv6Packet v_ipPacket; var FncRetCode v_ret; ipPort.send( m_echoReqWithHops(p_hops) );alt { [] ipPort.receive( mw_anyTimeExceeded ) -> value p_ipPkt { return e_success } [] ipPort.receive { return e_error } } }

23. Validation of (Conformance) Tests • Typical test suite validation requires • That TTCN-3 tests compile on several tools (5 at ETSI) • Tests executed on one or more platforms against various implementations of the standard (usually by ETSI members) • Requires very close co-operation with test tool suppliers, test platform providers, equipment manufacturers etc. • Strict and well-defined processes between • Test lab (who execute tests[, certify] and provide feedback) • And ETSI (who updates the tests) • Timely availability of stable base specifications, test platforms (tools) and implementations is essential • ETSI does not certify products!

24. CTI Test Specification Data Base Base Standard Database Test Cases(TTCN3) RequirementsCatalogue Test Description Test Purposes • Web interface, browsing by function, user-defined search and filter, traceability, document generation, …!

25. Example: ETSI IP Testing Library http://www.ipt.etsi.org/STF295-ph1/

26. Test Specification Development at ETSI • ETSI uses a variety of resources for test specification development • ETSI Technical Committee on Methods for Testing and Specifications(TC MTS) • Composed of testing experts from major telecom companies, tool vendors research institutes and academia • Works on guidelines and languages for ETSI specification development, e.g., TTCN-3, TPLan, IP Testing framework • ETSI Plugtests Service • Organizes interoperability events • ETSI Centre For Testing and Interoperability (CTI) • ETSI Specialist Task Forces (STF)

27. About ETSI CTI • In-house team of experts providing direct support as a service to ETSI Technical Bodies (e.g., ETSI GRID TC) on the use of formal techniques in standards • Supports ETSI test specification development - conformance and interoperability (60%) • Assistance in development of protocol standards and profiles, e.g., application of modern specification techniques (35%) • ‘Standards Engineering’ • Validation of standards (5%) • E.g., testbed activities (IPv6, SIP/H.323) • ETSI Plugtests™ Service • Validation of standards through interoperability events

28. Areas ETSI CTI Involvement • Cellular: GSM and 3G (UMTS) terminals • WiFi: HiperMAN, HiperACCESS, WiMax • VoIP: H.323, SIP, SIGTRAN • Service Creation: OSA/Parlay (API, IDL) • IPv6: Core, Security, Mobility, v4-v6 • Cordless phones: DECT • Radio communications: TETRA, DMR, dPMR • Access terminals: FSK, SMS • Broadband: ISDN, DSL • Smartcards: Readers, cards, security modules • Intelligent Transport Systems (ITS): DSRC • Early NGN/IMS • Future: More Security, GRID ...

29. About ETSI STFs • Team of experts seconded mostly from the ETSI membership • 25-30 experts in 2006 • Supported and often lead by CTI experts • Funded either by ETSI STF budget, European Union, or by commercial agreement • Can range from small (2 pm) to large (58pm over 2-3 years) • Approx. 2,7M€ of expert resource per year • Produce deliverables which are turned into freely and publicly available standards after approval from ETSI members • Requirement catalogue, test purposes, test descriptions, test cases • Frameworks, specification infrastructure, guidelines • Some validation • etc.

30. Upcoming GRID STF • ETSI GRID Technical Committee has created an STF • Completely funded by EC DG ENTR • Purpose is to identify gaps in grid specifications and develop GRID testing framework • At this point focus is on analysis of available grid specifications • Review of other GRID testing frameworks, e.g., ETICS project • Actual tests will come are the next step • Does not make sense to specify tests without clear scope • Call for experts is out (CL07_2532) • Application deadline May 27th !! • Interview of candidates on June 11th • Start of work expected in Q3 • Any ETSI or OGF member can apply – do so today 

31. Results of GRID Test Specification Analysis • ETSI CTI has been asked by ETSI TC GRID and OGF members to review Byte IO and Grid RPC test specifications Dec 2006 • Summary of observations • Lack of references to the specification, e.g., document clause or quotation of specification text • Test cases mix discussion of mandatory from optional features • Inconsistent use of terminology, e.g., “test case” • Grid RPC corresponds to ETSI test purpose(Grid RPC “test code” corresponds to ETSI test case) • Byte IO corresponds more to ETSI test description • Missing isolation of test purpose in Byte IO test specification makes understanding of test objective and verdict criteria difficult • Both documents from ETSI point of view specify conformance tests but claim to be interoperability tests (Byte IO specifies byte strings!) • Test cases should be more implementation language agnostic

32. Thank You! Feel free to contact us: • e-mail Stephan.Schulz@etsi.org • WWWwww.etsi.org/ptcc