1 / 37

ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing. Nikolay Pakulin Contributors: Ina Schieferdecker Jens Grabowski. Agenda. STF 430: TTCN3 Evolution TTCN3 Overview Advanced TRI Continuous Signals STF 433: TTCN3 Tool Conformance Testing Motivation Methodology

claude
Download Presentation

ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing NikolayPakulin Contributors: Ina Schieferdecker Jens Grabowski

  2. Agenda • STF 430: TTCN3 Evolution • TTCN3 Overview • Advanced TRI • Continuous Signals • STF 433: TTCN3 Tool Conformance Testing • Motivation • Methodology • Challenges • Requirements-driven Test Development ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  3. TTCN-3 Evolution TTCN-3 User Conference2012, Bangelore, India Nikolay Pakulinon behalf of STF430

  4. STF administrative data • STF 430: April 2011 – December 2011 (78 mDays) • Members: • Gyorgy Rethy, Ericsson • Jens Grabowski, University of Goettingen • Ina Schieferdecker, Fraunhofer FOKUS • Jacob Wieland, Testing Technologies • STF 430 sessions in 2011 • Session 1: 24-27 May • Session 2: 27 June - 1 July • Session 3: 26 - 30 September • Session 4: 28 November - 2 December ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  5. TTCN-3 Technology Overview ASN.1 IDL XSD Languagemappings Documentationt3doc Advancedparameteri-zation Behaviour types Staticconfiguration Real-time support Continuoussignalsupport AdvancedTRI Extensions TTCN-3 Structuring: Imports, Groups, Attributes TTCN-3 Behaviour Core language TTCN-3 Data Java C C++ C# XML TRI/TCI mapping ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  6. TTCN-3 language evolution • Tasks • Resolution and implementation of CRs • Prepare the drafts for ETSI publication: Parts 1 and 4 to 10 of the ES 201 873 series The 4 extension packages (ES 202 781, ~782, ~784, ~785) • Cooperation with STF 160 on language requirements of LTE testing • Two new language extension packages • Advanced TRI • Continuous signal support ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  7. The version forest • Approved new versions • TTCN-3 Part-1: Core Language V4.4.1 (44 CRs) • TTCN-3 Part-4: Operational Semantics V4.4.1 (1 CR) • TTCN-3 Part-5: TTCN-3 Runtime Interface V4.4.1 (2 CRs) • TTCN-3 Part-6: TTCN-3 Control Interface V4.4.1 (8 CRs) • TTCN-3 Part-7: Using ASN.1 with TTCN-3 V4.4.1 (1 CR) • TTCN-3 Part-8: Using IDL with TTCN-3 V4.4.1 (2 CR) • TTCN-3 Part-9: Using XML with TTCN-3 V4.4.1 (12 CRs) • TTCN-3 Part-10: TTCN-3 docum. tags V4.4.1 (1 CRs) • TTCN-3 Extension: Behaviour Types V1.2.1 (3 CRs) • TTCN-3 Extension: Configuration and deployment V1.2.1 (1 CR) • TTCN-3 Extension: Real-Time and Performance Testing V1.2.2 (2 CRs) • Not republished: • TTCN-3 Extension: Advanced Parameterization V1.3.1 • New: • TTCN-3 Extension: Advanced TRI V1.1.1 • TTCN-3 Extension: Continuous signal support V1.1.1 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  8. New features in Part 1 • 5417: Support of parametrized map/unmap varMyConfigTypecfg:={ option := 1, lock := false}; ... map(mtc:Port4, system:PCO2) param (cfg); • 5262: Partially constrained structured types • 5938: Type restriction by template listtype MyRecord MyRecordSub5 ( {f1 := *, f2 := "user", f3 := pattern "password|Password" }, { f1 := (1 .. 10), f2 := "User“, f3 := ? }) • 5937: Generalized annotation of attributes to declarations/members ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  9. New: Advanced TRI 1(3) • Historically, TTCN has been used to test communication protocols which typically use encoded messages. This has been reflected in the TRI SA and TCI CD design of TTCN-3 by encoding and decoding messages to/from bitstrings. • However, TTCN-3 also supports signature-based communication for which the transformation of objects into bitstrings and vice versa is cumbersome. • Therefore, an alternative API is being proposed along which TTCN-3 values can be directly passed to/from the SUT. • Old and new TRI can be used in combination! ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  10. New: Advanced TRI 2(3) • Redefinition of TRI Data Interface • TriMessageType use Value • TriAddressType use Value • TriAddressListType use ValueList • TriSignatureIdType unchanged • TriParameterType unchanged (not being used) • TriParameterPassingModeType unchanged (not being used) • TriParameterListType TciParameterListType (by that, the TciParameterPassingMode and Values for parameters will be used) • TriExceptionType use Value ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  11. New: Advanced TRI 2(3) • Changes to Message-based communication operations • triSend  xtriSend • triSendBC xtriSendBC • triSendMC xtriSendMC • triEnqueueMsg xtriEnqueueMsg • New • Value xtriConvert(in any value, in Type typeHypothesis)This operation shall be called whenever the TE has to convert a value. The TE might convert immediately after reception of the value, or might for performance considerations postpone the conversion until the actual access to the value. • Changes to Procedure-based communication operations • triCall xtriCall • etc. • Changes to Miscellaneous operations • triExternalFunction  xtriExternalFunction ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  12. TTCN-3 Extension: Continuous signal support 1(4) • This new package defines concepts for testing systems using continuous signals. • The package introduces • notions of time and sampling, • notions of streams, stream ports and stream variables, and • definitions allowing an automaton alike control flow structure supporting the specification of hybrid behavior (i.e., combined definition of discrete and continuous behaviors). ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  13. TTCN-3 Extension: Continuous signal support 2(4) • The following items shall give an impression about the contents of the package. Details and further operations can be found in the package definition. • Concepts for time and sampling: • access to global time: now operator • Setting stepsize for sampling: stepsize operator • Streams • New port type for streams, e.g.,: • type port StreamIn stream { in float } • type port StreamOut stream { out float } • Operations for accessing data streams: • retrieving or setting actual values: value operation • retrieval of time related information: timestamp operation • Retrieval of actual stepsize of a port: delta operation ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  14. TTCN-3 Extension: Continuous signal support 3(4) • Streams (cont.) • navigation on streams • previous event: prev operation • event at a certain point in time: at operation • stream extraction • Retrieval of (partial) history: history operation • Modes • define states in hybrid automata. Modes may • be composed sequential or parallel • have entry and exit conditions • have invariant parts • have a duration • be parameterisable • etc. ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  15. TTCN-3 Extension: Continuous signal support 4(4) 15 ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  16. A Conformance Test Suite for TTCN-3 Tools Bogdan Stanca-Kaposta, Testing Technologies Andras Kovacs, Broadbit Nikolay Pakulin, ISPRAS

  17. Motivation • Many TTCN-3 compilers (~7 commercial, 2 internal) • High interest in the language • Complex standard: hundreds of pages • No way to check the compliance of TTCN-3 tools • Quality of a testtooliscritical • Relevance: • For users: • Is my tool standard compliant? • Can I switch tools easily / vendor lock-in? • For tool vendors: • Where can I improve the standard compliance of my tool? • Possibly later: compliance certification • For standard developers: • Where is need for clarification? ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  18. STF 433 • Specialist Task Force initiated by ETSI MTS • Manpower: 3 persons for 100 days total • Dates: November 2011 – Early 2012 • Work is based on the existing framework. • Task: • Update the existing TTCN3 ATS to the newer TTCN-3 revision • Extend the conformance test suite for TTCN-3 tools. • Three deliverables: • ETSI TS 102 950-1 V1.2.1 (2012-05) – ICS • ETSI TS 102 950-2 V1.2.1 (2012-05) – TSS & TP • ETSI TS 102 950-3 V1.2.1 (2012-05) – ATS & IXIT • Approved by MTS in May 2012. • Downloadablefromthe ETSI downloadarearightnow! ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  19. Methodology (1/7) • Main steps: • Validate existing tests against new revision of the standard. • Develop conformance tests. • Follow the priority list discussed with MTS and vendors. • Develop adapters for selected tools. Validate and fix tests. • Automate test execution. ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  20. Methodology (2/7) • Adjusting Conformance ATS from TTCN3 rev. 4.2.1 to rev. 4.3.1: • Was: 733 tests, 130 pp. of test purposes • Validate that test purposes still hold • Update modified test purposes • Revoke outdated tests • Update ATS folder structure • Approximately 30% of the STF 433 effort • New tests development • Clause priority list: • Not all clauses can be covered. Selection was made: Vendor feedback + most basic functionality. • There is no perfect selection! ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  21. Methodology (3/7) • Development of conformance tests: • The ATS is not the test automation, but the test input! • In most cases: 1 module = 1 conformance test. • Test automation provided for certain vendors • Elvior and Testing Technologies • Test suite structure: organized according to clauses. • Expected output for verdict determination: • @verdict pass accept/reject [expectedoutput] • Examples: • @verdict pass reject • @verdict pass accept, ttcn3verdict:pass • @verdict pass accept, noexecution • Purpose specification: • @purpose documentreference, description • Documentreference: part:clause • Examples: • @purpose 1:5, Ensure that when the IUT loads a module containing some definitions before the module declaration then the module is rejected. ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  22. Methodology (4/7) • Development of conformance tests (cont.): • Complete TTCN-3 test input example: /*************************************************** ** @purpose 1:5.2.2, Ensure that the IUT correctly handles the uniqueness of variable names in its scope. ** @verdict pass accept, ttcn3verdict:pass ***************************************************/ module Sem_050202_Uniqueness_001 { importfrom Sem_050202_Uniqueness_001_import { constall; } typecomponentGeneralComp {} functionf_funcScope() { varbooleanrepeatedIdentifier := true; if(repeatedIdentifier==true) { setverdict(pass); } } testcase TC_Sem_050202_Uniqueness_001() runsonGeneralComp { f_funcScope(); } control { execute(TC_Sem_050202_Uniqueness_001()); } } module Sem_050202_Uniqueness_001_import { // repeated indentifier // from imported module // is allowed constinteger repeatedIdentifier := 0; } ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  23. Methodology (5/7) • Creation of documentation: • Framework already provided by STF393 (Proforma). • Little time for documentation. • Working on a single source saves time. • Solution: • Write test purposes and expected tool outputs directly into the test inputs. • Generate tables for the documentation from the ATS: • Test purpose tables. • Test suite structure. • Much easier to maintain! • Documentation generation tools developed as part of STF409. ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  24. Methodology (6/7) • Validation and fixing: • Validation and fixing took more than half of the development time! • STF internal validation with 2 tools: • TTworkbench: compile-time and execution. • TestCast: compile-time and execution. • Developed necessary adapters • Loopback, proc. call • Lots of feedback from TTech and Elvior! ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  25. Methodology (7/7) • Executing the ATS: TTCN-3 File TTCN-3 File TTCN-3 File Sem_060101_TopLevel_001.ttcn ATS ** @verdict pass accept, ttcn3verdict:pass Test input Sem_060101_TopLevel_001.ttcn NegSem_060303_component_types_001.ttcn … Expected output TTCN-3 Tool (SUT) Tool output Rejection as invalid Rejection to execute Execution with results (e.g., pass/inconc/fail) Tool output evaluation pass fail ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  26. Discussion pass TTCN-3 Conformance ATS TTCN-3 Tool Meaning? • Compliance does not imply safety from a vendor lock-in! • Testing is never complete! • With v1.1.1 of the ATS, about 32% of the core standard clauses are covered with at least one test. • Ideally, we should have many tests for each clause. • Compliance means less likelihood for vendor lock-in. • With better coverage and higher number of test cases, we improve the quality of this statement. • Compliance reduces the likelihood for major tool errors • Tests range from simple arithmetic operations to communication operations and verdict handling. • ATS checks syntax and semantics. • ATS does not check the correctness of every single tool message, but of the overall output. ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  27. Challenges (1/2) • Various standard clarifications were necessary: • 13 CRs for TTCN-3 core languages have been written. • Unclarified test cases have been excluded from the delivery ("on-hold"). • Tools sometimes handle the standard differently. • Tests need to be expressed as tool independent as possible. • Test Automation: • No standard command line interface for batch run • Tool-specific adapters • No common test log format • TTCN3 semantics • TTCN3 standard part 1 defines both static and dynamic semantics • No definition of well- and ill- formed programs • What to check at compile time is tool-dependent ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  28. Challenges (2/2) • Synchronize ATS with new TTCN3 revisions • Time consuming manual activity • As ATS grows the effort will grow as well • To which core language standard version does the conformance test suite relate to? • How to maintain compatibility across different language versions? • Against which version should the conformance test suite be developed in the future? • Which TTCN3 revision to support • Should we certify newest TTCN3 revision only? ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  29. Conclusion • The conformance test suite is available now! • ATS Metrics: • Total number of conformance tests: 1077 (340 new) • 20 tests fixed • 9 removed since deprecated or duplicates • Number of positive syntactic conformance tests: 138 • Number of negative syntactic conformance tests: 82 • Number of positive semantic conformance tests: 517 • Number of negative semantic conformance tests: 340 • Number of clauses (at least partially) covered: 134 (24 new) ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  30. 30 TTCN-3 CONFORMANCE TEST SUITE STF 433 STATUS REPORT NikolayPakulin Requirements tracking case study ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  31. Case study in a nutshell • 4 chapters: total 7,5 pages • 12 Declaring messages, • 13 Declaring timers, • 23 Timer operations, • 24 Verdict operations; • 56 requirements elicited, • 7 requirements refined into 34 test purposes; • covered by 174 test cases • Average 7,5 requirements per page, 3 test cases per requirement ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  32. Requirements tracking case study • ISP RAS contributed to the project requirements management tool Requality • Hierarchical grouping of requirements • Requirements elicitation from DOC/HTML documents • Requirements are connected to certain locations in the documents • Requirements locations are highlighted • Requirements refinement through test purposes • Test purposes define specific cases to be tested in order to treat corresponding requirement ‘covered’ • Requirements coverage by test cases report • Source document change management (beta) • Update requirements when the text of the standard is modified • Requality is an open-source plugin to eclipse • Integrated with version control software (SVN) ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  33. Sample view: Requality in TTWorkbench ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  34. Requirements Tracking (1/2) Fully qualified name of the requirement /*************************************************** ** @author STF 433 ** @version 0.0.1 ** @purpose 1:12, Ensure timer can be declared in components ** @verdict pass accept, ttcn3verdict:pass ***************************************************/ /* * #reqname /Requirements/12 Declaring timers/Timers can be declared in component type **/ module Sem_12_toplevel_timer_001 { typecomponentTComp{ timert_timer:=10.0; } testcase TC_Sem_12_toplevel_timer_001() runsonTComp{ t_timer.start; if (t_timer.running){ setverdict(pass) } else { setverdict(fail) } } control{ execute(TC_Sem_12_toplevel_timer_001()) } } ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  35. Requirements Tracking (1/2) • Requirements coverage report • Integrated into Eclipse • Evaluates coverage of requirements and test purposes ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  36. Suggested use case • Add requirements markup to TTCN3 standard • Manual • Introduce references between requirements and TTCN3 test cases • Manual • Track requirements changes between TTCN3 standard revisions • Automated (beta) • Track required changes in TTCN3 ATS • Manual, to be automated ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

  37. Thank You! Questions? ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

More Related