Interoperability in the DNA of standards: Ensuring interoperability between building blocks

1. Interoperability in the DNA of standards: Ensuring interoperability between building blocks Anthony Wiles, ETSI Protocol and Testing Competence Centre (PTCC) @metis Kick-off meeting 30-31 March 2005

2. Test Specs Specification Testing Development In Pursuit of Interoperability • Ultimate aim of ICT standardisation is interoperability • Likelihood of interoperability is increased with • Well-defined, accurate and unambiguous standards • Systematic testing of products based on those standards • ETSI produces Base Standards and Test Specifications

3. Unique Resources Available to ETSI Technical Bodies • TC MTS (Methods for Testing and Specification) • Develops specification methodologies and techniques • http://portal.etsi.org • ETSI PTCC (Protocol and Testing Competence Centre) • Supports ETSI Technical Bodies with the development of: • technology standards: protocols, services, APIs etc. • test specifications • http://www.etsi.org/ptcc • ETSI Plugtests Service • Organises interoperability events • http://www.etsi.plugtests

4. How Does the PTCC Help? • Assist ETSI Technical Bodies on the use of state of the art techniques for • Specification, validation and testing • Good working practices (Standards Engineering) • Pragmatic and flexible approach • Based on experience • Help to develop usable methodologies • For ETSI’s current and emerging needs • Knowledge transfer • Quality through Continuity

5. Who PTCC Supports • Technical Bodies (TB) • Technical Committees • ETSI Projects • Partnership Projects etc. • Chairmen, Rapporteurs, Individuals • Working Groups (WG) • STFs (Specialist Task Forces) • PTCC budget for test specifications (15-20 STFs per yr) • ETSI Secretariat

6. Industry Products mature from prototypes to commercial products Interoperability Testing (Unit) Conformance Testing Interoperability Test Specifications Conformance Test Specifications ETSI TB: Development of base standards ETSI PTCC: Support on Protocol Specification and Testing Good practice, pragmatic application of methodology, text complemented by the use of specification languages: UML, SDL MSC, ASN.1, IDL, XML etc. Planning, methodology, test specifications, TTCN, ISO-9646, TS 102 237 PTCC Support in Standards Development Certification Specification, validation and simulation

7. Typical PTCC Areas of Activity • GSM and 3G (UMTS) terminals • Wi-Fi: HiperLAN/2, HiperMAN, HiperACCESS • Cordless phones: DECT • Access terminals: FSK, SMS • DSL Splitters • ISDN, Broadband ISDN • OSA (API, IDL) – web services • NGN • VoIP: H.225, H.248, H.245 (ITU), SIP (IETF) • SIGTRAN • Smartcards • DECT • IPv6: Core, IPSEC, Mobility, v4 -> v6 • DSRC (ITS) • DMR • ... future: More telematics/ITS, more security?

8. Design for Interoperability • Interoperability takes place on external interfaces • Normative (mandatory) features • Options • Consequences must be clear • Abnormal behaviour should be well-defined • Robustness • Never specify how product is to be implemented internally • Different levels of abstraction e.g., 3-stage approach • Methods like SDL, ASN.1, XML, MSC and UML used to specify these interfaces • Data transferred on protocol interfaces • syntax and encodings (ASN.1, XML) • Behaviour of modules as seen from these interfaces • TTCN and test methods used for black box testing • No need to know how product is implemented • External interface is tested according to specification

9. Software Interfaces: APIs • APIs are external interfaces to software • specification of external interface: • Protocol used at interface • Data exchanged on interface • Behaviour displayed by interface • API can be specified in high-level language like Java, C++ etc. • No specification of software system as a whole • Interoperability is difficult without well-defined external interfaces or connectors • Standardised test procedures required to demonstrate compliance with interface definition • Use of TTCN (TTCN-3)

10. Profiling and Interworking • Many standards are ‘open’ • Include many options • Interoperability requires profiling • e.g. ETSI TISPAN NGN • Options are screwed down (dropped) • Works with ‘similar’ technologies • Implement and test the profile • Significantly different technologies • Interworking functions • e.g., SIP – H.323 • General reluctance to specify ‘profiles’ • Indications that situation is changing

11. Conformance Network Integration Robustness Performance Interoperability RF/EMC Different Kinds of ETSI Test Specifications

12. Conformance Testing vs Interoperability Testing

13. Characteristics of Conformance Testing • Is unit testing • Tests a single ‘part’ of a product (e.g., a protocol layer) for conformity to the requirements specified in the base standard (specification) • Tests over standardised interfaces • May not be available to ‘normal’ user • Works at a 'low' level • Tests at the protocol (message/behaviour) level (bits) • High control and observability • Means we can explicitly test error behaviour and other 'difficult' scenarios • Repeatable • Requires a test system (and executable test cases) • Can be expensive (e.g., radio-based test system) • Conformance Testing is DEEP and NARROW

14. Limitations of Conformance Testing • Does not prove end-to-end functionality (interoperability) between communicating systems • Conformance tested implementations may still not interoperate • this is often a specification problem rather than a testing problem! • need minimum requirements or profiles • Does not test a complete product • Tests the parts, not the whole • often a product is more than the sum of its parts • Does not test proprietary features, functions etc. • but this may well be done by a manufacturer with own conformance tests for proprietary item

15. Characteristics of Interoperability Testing • Do not confuse interoperability testing with interoperability demos • Shows that two products interoperate • Within what may be a limited scenario • Tests Functionality (of an ‘entire’ product) • Tests the ‘whole’, not the parts • e.g, protocol stacks ... • or protocol stacks + applications • High-level (as perceived by users) • Shows function is accomplished (but not how) • Does not necessarily require a test system • Uses existing interfaces (standard/proprietary) • Usually lower-cost than conformance • But can (should be) automated => test drivers • Interoperability Testing is BROAD and SHALLOW

16. Limitations of Interoperability Testing • Does not prove interoperability with other implementations with which no testing has been done • Does not prove that a product conforms • Interoperable products may still be non-conformant • Cannot explicitly test error behaviour or unusual scenarios • Or other conditions that may need to be forced (lack of controllability) • Has limited coverage • May not be repeatable

17. Interop Testing Conformance Testing Interoperability Conformance Testing and Interoperability Testing complement each other • Not always a case of either ... or ... • Though sometimes it has to be for practical or economic reasons • But for the best probability of interoperability between products – do both!

18. Summary • Standards can be designed for interoperability • Standards should be engineered • Plan for testing (early) • Do the right kind of testing and test in parallel

19. Interoperability in the DNA of standards: Ensuring interoperability between building blocks The End Thank you for your attention