1 / 56

Overview of ETSI Testing Methodology ICT-OSA/Parlay Workshop Brazil, March 2006

Overview of ETSI Testing Methodology ICT-OSA/Parlay Workshop Brazil, March 2006. Anthony Wiles Manager ETSI Protocol and Testing Competence Centre. Interoperability is …. The ultimate aim of ICT standardisation

saman
Download Presentation

Overview of ETSI Testing Methodology ICT-OSA/Parlay Workshop Brazil, March 2006

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Overview of ETSI Testing Methodology ICT-OSA/Parlay WorkshopBrazil, March 2006 Anthony Wiles Manager ETSI Protocol and Testing Competence Centre

  2. Interoperability is … • The ultimate aim of ICT standardisation • The red thread running through the entire standards development process, it’s not an isolated issue • Not something to be somehow fixed at the end • The ability of systems and products to interwork is fundamental • ETSI approach • ETSI produces with the required degree of parallelism • Base Standards and • Test Specifications • Base standards should be designed with interoperability in mind • Profiles designed to reduce potential non-interoperability • Protocol conformance and Interoperability statements • Two complementary forms of testing • Conformance testing • Interoperability testing (formal IOT) • Testing provides vital feedback into the standardization work Testing Methodology - ICT Workshop, Brazil March 2006

  3. ETSI Resources for Interoperability • Technical Committee MTS (Methods for Testing and Specification) • Development of methodologies, techniques and languages (including TTCN3) http://portal.etsi.org • ETSI’s Protocol and Testing Competence Centre (PTCC) • Supports ETSI committees on the application of formal techniques in standards on a daily basis • Development of test specifications (conformance and interop) http://www.etsi.org/ptcc • ETSI Plugtests™ Service • Validation of standards and prototypes through interoperability events http://www.etsi.org/plugtests • These resources are highly complementary Testing Methodology - ICT Workshop, Brazil March 2006

  4. Industry Interoperability Testing (Unit) Conformance Testing Interoperability Test Specifications Iterative feedback Conformance Test Specifications Typical Standards Development Process in ETSI Certification Products mature from prototypes to commercial products time ETSI: Development of base standards Testing Methodology - ICT Workshop, Brazil March 2006

  5. Conformance Network Integration Robustness Performance Interoperability RF/EMC Different Kinds of ETSI Test Specifications Testing Methodology - ICT Workshop, Brazil March 2006

  6. Progressive Testing • All engineering disciplines accept that testing should be applied to progressively complex units • From individual components to entire systems • This demands a number of different testing solutions • There is no silver bullet • In the two extremes • Just testing the components is not enough • Just testing the system is not enough • Limitations are economic, not technical • What can I afford to test? • What can I afford not to test? • What should be covered by standardised test specifications • What is the right kind of testing for the job • What kind of ‘thing’ is being tested? Device? System? Service? • What is the most economic method(s)? • What resources are (or must be made) available? • Tools, test scripts, testbeds etc. Testing Methodology - ICT Workshop, Brazil March 2006

  7. Interop Testing Conformance Testing Interoperability Conformance and Interoperability Testing are Complementary • ETSI experience • As you move up a system stack the emphasis should change from conformance to IOT • Lower layer protocols, infrastructure • Emphasis on conformance • Middleware, enablers • Combination of Conformance + IOT • Services, applications, systems • Emphasis on IOT • Conformance testing as a pre-requisite to IOT Testing Methodology - ICT Workshop, Brazil March 2006

  8. Device Reqs. Tester Conformance Testing • Is Black-Box testing • Stimulation and Response Point of Control and Observation (PCO) Testing Methodology - ICT Workshop, Brazil March 2006

  9. IUT = Implementation Under Test Upper Tester MMI API L3 L3 Test SUT L2 System d i g i t a l PHY 1 2 3 4 5 6 7 8 9 8 # * Lower Tester SUT = System Under Test Conformance Testing Tests is Usually Layer-byLayer i i t a l Testing Methodology - ICT Workshop, Brazil March 2006

  10. Characteristics of Conformance Testing (1) • Is unit testing • Tests a single ‘part’ of a device (e.g., a protocol layer) • Often incrementally – layer-by-layer • Tests against well-specified requirements • For conformance to the requirements in a base specification or profile (standard) • Usually limited to one requirement per test • Tests at a 'low' level • At the protocol (message/behaviour) level (bits) • Or service primitive, API, Interface • Tests over standardised interfaces • May not be available to ‘normal’ user • Requires a test system (and executable test cases) • Can be expensive (e.g., radio-based test system) • Tests under ideal conditions Testing Methodology - ICT Workshop, Brazil March 2006

  11. Characteristics of Conformance Testing (2) • High control and observability • Means we can explicitly test error behaviour • Can provoke and test non-normal (but legitimate) scenarios • Can be extended to include robustness tests • It is usually automated and tests are repeatable • Conformance Testing is DEEP and NARROW • Thorough and accurate but limited in scope • Gives a high-level of confidence that key components of a device or system are working as they were specified and designed to do Testing Methodology - ICT Workshop, Brazil March 2006

  12. Limitations of Conformance Testing • Does not prove end-to-end functionality (interoperability) between communicating systems • Conformance tested implementations may still not interoperate • This is often a specification problem rather than a testing problem! Need minimum requirements or profiles • Does not test a complete system • Tests individual system components, not the whole • A system is often greater than the sum of its parts! • Does not test functionality • Does not test the user’s ‘perception’ of the system • Standardised conformance tests do not include proprietary ‘aspects’ • Though this may well be done by a manufacturer with own conformance tests for proprietary requirements Testing Methodology - ICT Workshop, Brazil March 2006

  13. logging and analysis Test System Product Implementation Under Test testing Executable Test Suite (e.g., C++) Test Report Industry validation implementation compilation Base Standard (or Profile) Conformance Testing Documentation and Process: ISO 9646 Methodology Req. checklist ICS Test Purposes TSS & TP Test Suite (Test Cases) ATS Testing Methodology - ICT Workshop, Brazil March 2006

  14. Implementation Conformance Statement (ICS) The ICS is a checklist of the capabilities supported by the Implementation Under Test (IUT) Provides an overview of the features and options that are implemented in any given product The ICS can be used to select and parameterise test cases and can be used as an indicator of basic interoperability between different products. Conditional support e.g., Qn must be supported if Q1 supported then otherwise not. Testing Methodology - ICT Workshop, Brazil March 2006

  15. Profile ICS A profile ICS reflects how a (protocol) profile restricts the scope of a base standard by making certain options mandatory or excluding certain options. Testing Methodology - ICT Workshop, Brazil March 2006

  16. Selecting Tests with the ICS A filled-in ICS can be used to select tests. For example, a test for an OPTIONAL feature which is not supported in a given product will be deselected from the test suite. Testing Methodology - ICT Workshop, Brazil March 2006

  17. Parameterizing Tests with eXtra Information for Testing (IXIT) A filled-in IXIT can be used to parameterize tests. The Value column is used to specify explicit IUT or test system dependent values. Testing Methodology - ICT Workshop, Brazil March 2006

  18. Test Purposes (TP) Test Purposes (TP) are natural language, precise descriptions of the purpose of the test in relation to a particular (base standard) requirement They are very focussed. They are not code. A TP is WHAT to test not HOW to test TPId: SIP_SS _PR_CE _V_012 Selection:Mandatory [Q2] Precondition:Registration of both simulated UA to the IUT and an initiated session Ref:5.1.1 [1], 17.3.6 [1], 17.4 [1] , 10.46.6 [1] Purpose:Ensure that the IUT on receipt of a Server Internal Error (500 Server Internal Error) server failure response, sends an ACK request, with the same field Via and Branch parameter as in the previous INVITE, to the UAS and forwards it to the UAC. Testing Methodology - ICT Workshop, Brazil March 2006

  19. Test Suite Structure (TSS) Registration Registrant Valid Invalid Registrar : Session Originating Endpoint Call Establishment TP_S_OE_CE_01 TP_S_OE_CE_02 TP_S_OE_CE_03 : Call Release : Terminating Endpoint Proxy Redirect Server : • Test Purposes are grouped into a logical Test Suite Structure (TSS) according to suitable criteria • e.g., basic interconnection, error handling, functionality etc.). • The complete document is usually called the Test Suite Structure and Test Purposes (TSS&TP) Testing Methodology - ICT Workshop, Brazil March 2006

  20. Abstract Test Suite • A Test Case is the ‘implementation’ of the test purpose • Is the HOW to test not WHAT to test • The Test Suite (ATS) is the collection of all Test Cases • Most ETSI Test Suites are written in TTCN • Predominantly in TTCN-2 • Progression to TTCN-3 for new work • Not RF testing (generally not physical layer) • TTCN-3 • Testing and Test Control Notation • Allows tests to be specified in detail (test code) that is independent of the (eventual) real test system on which it may be run • i.e., TTCN-3 will run on any test system that supports a standardised TTCN-3 execution environment Testing Methodology - ICT Workshop, Brazil March 2006

  21. Executable Test Suites • Executable test suites are compiled from the abstract • Either direct to binary, or • More commonly to some intermediate programming language (C++, Java, etc.) • ETSI does not deliver executables, but • We try to ensure all code can be compiled • And validated by executing the tests against a real implementation on at least one test system • 5 test systems in the case of UMTS, for example Testing Methodology - ICT Workshop, Brazil March 2006

  22. Device2 Device1 Devicen Interoperability Testing • End-to-End Interoperability of devices • What is being tested? • Assignment of verdicts? • Need to distinguish between • Formal interoperability testing • And informal interoperability events used to validate/develop technologies Testing Methodology - ICT Workshop, Brazil March 2006

  23. 1 2 3 1 2 3 4 5 6 4 5 6 7 8 9 7 8 9 8 # 8 # * * MMI MMI SUT API API QE = Qualified Equipment EUT = Equipment Under Test L3 L3 L2 L2 PHY PHY Conformance Monitoring Interoperability Testing Tests Functionality Between Devices QE EUT Means of Communication (MoC) Interoperability Testing ( of terminals) Testing Methodology - ICT Workshop, Brazil March 2006

  24. Characteristics of Interoperability Testing • Is system testing • Tests a complete device or a collection of devices • Shows that (two) devices interoperate • Albeit within a limited scenario • Tests at a ‘high’ level (as perceived by users) • Tests the ‘whole’, not the parts • e.g, protocol stacks + applications • Tests functionality • Shows function is accomplished (but not how) • Does not necessarily require a test system • Uses existing interfaces (standard/proprietary) • Interoperability Testing is BROAD and SHALLOW • Less thorough but wide in scope • Gives a high-level of confidence that devices (or components in a system) will interoperate with other devices (components) Testing Methodology - ICT Workshop, Brazil March 2006

  25. Limitations of Interoperability Testing • Does not prove interoperability with other implementations with which no testing has been done • A may interoperate with B and B may interoperate with C. But it doesn’t necessarily follow that A will interoperate with C. • Combinatorial explosion • Does not prove that a device is conformant • Interoperable devices may still interoperate even though they are non-conformant • Cannot explicitly test error behaviour or unusual scenarios • Or other conditions that may need to be forced (lack of controllability) • Has limited coverage (does not fully exercise the device) • Not usually automated and may not be repeatable Testing Methodology - ICT Workshop, Brazil March 2006

  26. Interoperability Test Methodology • ETSI TS 102 237-1 • Uses best principles of ISO 9646 and adapts them for interoperability testing • Definitions • EUT – Equipemt Under Test • QE – Qualified Equipment • Concepts • IFS (like the PICS) • Interoperable Functions Statement • Generic testing architecture • Test Descriptions (in prose, tabular format) • If suitable interfaces (API/MMI) are exposed then can automate the tests by running TTCN-3 scripts • Processes • Basic test specification development process Testing Methodology - ICT Workshop, Brazil March 2006

  27. Test Report Product 1 Qualified Equipment Product 2 Equipment Under Test logging and analysis testing implementation implementation Base Standard (or Profile) Interoperability Testing Documentation and Process: TS 102 237 Methodology Interop Test Cases Interop Test Purposes IFS (functional checklist) Testing Methodology - ICT Workshop, Brazil March 2006

  28. Test Purpose Language (TPLan) • ETSI language, not necessarily restricted to IOP testing • Keywords and syntax provide clear and consistent structure • Keywords chosen for communications applications (sends,receivesetc.) • Text between keywords not part of syntax so free expression possible • A TP’s basic structure (corresponding keyword): • Header (TP id) • Pre-conditions (with) • Stimulus (when) • Expected response (then) Testing Methodology - ICT Workshop, Brazil March 2006

  29. TPLan Example for Interoperability TP id : TP_COR_1719_02 Summary : 'EUT sends packet to All-RT LL M/cast address' RQ ref : RQ_COR_1719 Config : CF_021_I TD ref : TD_COR_1719_02 with { QE1'configured as a router' andQE2'configured as a router'} ensurethat { when { EUT is requested to 'send packet to All-RoutersLink-Local M/cast addr' } then { QE1 indicates'receipt of the packet' andQE2 indicates'receipt of the packet' } } Testing Methodology - ICT Workshop, Brazil March 2006

  30. Interoperability Test Descriptions • Specify detailed steps to be followed to achieve stated test purpose • Steps are specified clearly and unambiguously without unreasonable restrictions on actual method: • Example: • Answer incoming call NOT • Pick up telephone handset • Written in a structured and tabulated natural language so tests can be performed manually • Can be automated using TTCN‑3 when EUT has software interfaces Testing Methodology - ICT Workshop, Brazil March 2006

  31. Example Test Description Testing Methodology - ICT Workshop, Brazil March 2006

  32. AT Commands AT Commands Automated IOP Testing Using TTCN-3 Test Controller (TTCN-3) Test Driver (TTCN-3) Test Driver (TTCN-3) Testing Methodology - ICT Workshop, Brazil March 2006

  33. PoC RADIUS SIP DIAMETER Test Driver2 Test Driver1 Network Integration Testing (NIT) • End-to-end testing of the system • Components in the network must interoperate SUT Testing Methodology - ICT Workshop, Brazil March 2006

  34. Example NIT for IMS Testing Methodology - ICT Workshop, Brazil March 2006

  35. Testing Methodology - ICT Workshop, Brazil March 2006

  36. TTCN-3 can be used for Conformance and Interoperability Testing Testing Methodology - ICT Workshop, Brazil March 2006

  37. Benefits of TTCN-3 • Specifically designed for testing • Concentrates on the test not the test system • Commonly understood syntax and operational semantics • Constantly maintained and developed • Off-the-shelf tools and TTCN-based test systems are readily available • Single language for many (all?) testing activities • Education and training costs can be rationalized • Maintenance of test suites (and products) is easier • Allows the application of a common methodology and style, both on a corporate level and within standardization Testing Methodology - ICT Workshop, Brazil March 2006

  38. Main Capabilities of TTCN-3 • Dynamic concurrent testing configurations • Various communication mechanisms (synch and asynch) • Data and signature templates with powerful matching mechanisms (including regular expressions) • Specification of encoding information • Display and user-defined attributes • Test suite parameterization • Control of Test Case execution and selection mechanisms • Control of complex test configurations • Assignment and handling of test verdicts • Harmonized with ASN.1 (XML and IDL coming) • Different presentation formats • Well-defined syntax, static - and operational semantics Testing Methodology - ICT Workshop, Brazil March 2006

  39. Text format Tabular Format Graphical Format PresentationFormat3 PresentationFormatn The Core Languageand Other Presentation Formats • Core format is text based (most popular) • TTCN-3 can be edited or viewed in other formats • Tabular format (for TTCN-2 people) • Graphical format (good for visual overview) • Other standardized formats in the future? • Proprietary formats possible TTCN-3 Core Language Testing Methodology - ICT Workshop, Brazil March 2006

  40. Example Core (Text) Format function PO49901(integer FL) runson MyMTC { L0.send(A_RL3(FL,CREF1,16)) TAC.start alt { [] L0.receive(A_RC1((FL+1) mod 2)) { TAC.cancel verdict.set(pass) } [] TAC.timeout { verdict.set(inconc) } [] any.receive { verdict.set(fail) } } } Testing Methodology - ICT Workshop, Brazil March 2006

  41. Example Graphical Format Testing Methodology - ICT Workshop, Brazil March 2006

  42. Example Tabular Format Testing Methodology - ICT Workshop, Brazil March 2006

  43. ASN.1 Types & Values IDL Types & Values XML Types & Values Other types & Valuesn Use of TTCN-3 With Other Languages • TTCN can be integrated with other languages • Fully harmonized with ASN.1 (1997) • Harmonized with other languages • IDL, XML, C/C++ TTCN-3 Core Language Testing Methodology - ICT Workshop, Brazil March 2006

  44. Test Suite Test Data Types Test System Architecture Actual Test Data Test Behaviour Major Elements of a TTCN-3 Test Suite • Test components • Test ports • Connections • between test components • to System Under Test • Configurations • static • dynamic • Actual test data (values) used during testing • Message templates • (Remote) signature templates • Matching mechanisms • values, lists, wildcards, • regular expressions etc. • Encoding information • Definition of data types for • Protocol Messages and • Information elements (fields, parameters) • Internal data (e.g., for computation) • Test coordination messages etc. • Predefined simple types • Integer, boolean, float • bitstring, hexstring, octetstring • charstring, universalcharstring • ... and complex types • record, recordof, set, set of • union, enumerated • ... and special types such as • verdict, address test cases test steps default behaviour management of test components sending/receiving messages verdict assignment computation (e.g., checksums) timers and timeouts test execution and control etc. Testing Methodology - ICT Workshop, Brazil March 2006

  45. stimulus response Simple TTCN-3 Architecture Implementation Under Test Black-Box TTCN-3 based Test System Point of Control and Observation (PCO) Test System IUT Test port Test port Testing Methodology - ICT Workshop, Brazil March 2006

  46. Parallel Test Component (PTC) Parallel Test Component (PTC) IUT Main Test Component (MTC) TTCN-3 Architecture – Test Components Test System Internal Communication Communication to IUT Testing Methodology - ICT Workshop, Brazil March 2006

  47. Module Definitions Module Control Attributes TTCN-3 Module module Example { // Definitions part control { // Control part } } with {encode "BER"} Module (…) Testing Methodology - ICT Workshop, Brazil March 2006

  48. Definitions Part module Example { group TestArchitecture { // Port and Test Component definitions } group MessageTypes { // Defintions of message types } group ActualMessages { // Templates for instances of actual messages } group TestCases { // Test Case definitions } } module Example { // Port and Test Component definitions // Defintions of message types // Templates for instances of actual messages // Test Case definitions } Testing Methodology - ICT Workshop, Brazil March 2006

  49. Specifying Test Messages type record MsgType { integer msgId charstring msgName, charstring msgInfo } template MsgType INVITE { msgId 0 msgName "INVITE", msgInfo "For a good Brazilian Dinner" } template MsgType ACCEPT { msgId 1 msgName "ACCEPT", msgInfo * } Testing Methodology - ICT Workshop, Brazil March 2006

  50. P INVITE ACCEPT or DECLINE TTCN-3 Behaviour testcase TC1() runs on PTC1 { P.send(INVITE) T1.start(30E3) alt { [] PCO.receive(ACCEPT) {verdict.set(pass)} [] PCO.receive(DECLINE) {verdict.set(inconc)} [] PCO.receive {verdict.set(fail)} [] T1.timeout {verdict.set(fail)} } } Testing Methodology - ICT Workshop, Brazil March 2006

More Related