1 / 41

Theory and Practice of TTCN-3

Theory and Practice of TTCN-3. Curricula – 2 weeks (2L / week). Why do we test?. Errare humanum est! (to err is human) Even with formal specification and program generation... Specification could be wrong Generator doesn‘t work as expected Platform doesn‘t work as expected ...

byron
Download Presentation

Theory and Practice of TTCN-3

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Theory and Practice of TTCN-3

  2. Curricula – 2 weeks (2L / week)

  3. Why do we test? • Errare humanum est! (to err is human) • Even with formal specification and program generation... • Specification could be wrong • Generator doesn‘t work as expected • Platform doesn‘t work as expected • ... • Testing checks that the Software • ... does what it should do and ... • ... doesn‘t what it shouldn‘t do • But always remember that ... • ... „testing can only show the presence of bugs, not their absence“!!! E.W. Djikstra

  4. Validation & Verification Test Level Construction Acceptance Test Requirements Analysis System Test Architecture Integration Test Unit Test Design Coding When do we test? Test process Activities of a test team during the development cycle: • Identification of test cases – analysis • Specification of test cases – design • Implementation of test cases – coding • Execution of test cases – testing • Initial (manual) testing of new Test Cases • Continuous regression

  5. Testing Today • Is... • Important • Expensive • Time critical • But... • Only rarely practiced • Unsystematic • Performed by hand • Error-prone • Uncool ("If you are a bad programmer you might be a tester!") • Destructive

  6. Testing is... • a technical process, • performed by experimenting with a software product, • in a controlled environment, • following a specified procedure, • with the intent of observing one or more characteristics of the product, • by demonstrating the deviation of the product's actual status from the required status/specification.

  7. About TTCN-3... • TTCN has been developed and is maintained by the Methods for Testing and Specification Technical Committee (TC-MTS) at ETSI. This group is composed of leading testing experts from industry and academia as well as members of ETSI's own Centre for Testing & Interoperability (CTI). • TTCN-3 was born out of TTCN-2 which had been maintained by the same group. • TTCN testing technology has been applied widely and successfully in European industry, ETSI standardization, and certification for more than a decade.

  8. Why TTCN-3? • TTCN-3 is the Testing and Test Control Notation version 3 • Internationally standardized testing language for formally defining test scenarios – design purely for testing • developed from 1999 – 2002 at the European Telecommunications Standards Institute (ETSI). • A programming language that has been used for more than 15 years in standardization as well as industry • Triple C • Configuration: Dynamic concurrent test configurations with test components • Communication: Various communication mechanisms (synchronous and asynchronous) • Control: Test case execution and selection mechanism • Applicable for all kinds of black-box testing for reactive and distributed systems.

  9. Main Aspects of TTCN-3 • Triple C • Configuration: Dynamic concurrent test configurations with test components • Communication: Various communication mechanisms (synchronous and asynchronous) • Control: Test case execution and selection mechanisms • Features • Well-defined syntax, static and operational semantics • Different presentation formats • Module concept • Extendibility via attributes, external function, external data • Harmonized with ASN.1

  10. Some of the Contributing Organizations... Danet Davinci CommunicationEricsson Expert Telecoms France Telecom Fraunhofer Gesellschaft (FhG)Fraunhofer FOKUS Motorola NMG Telecoms Nokia Nortel PlenwareSiemensTechnical University Berlin Tektronix Telelogic Testing Technologies University of GoettigenOpenTTCN University of Luebeck (Institute for Telematics) VTT Technical Center of Finland etc ...

  11. TTCN-3 Success stories • At ETSI • Used for development of any new conformance test suite, e.g., SIP (VoIP), IPv6 (Core, Mobility, Security), HiperMAN / WiMax, 3GPP IP Multimedia Subsystem, … • In industry • Applied in a variety of application domains, e.g., telecom, automotive, financial, … (see www.tt-medal.org) • Ericsson reported 1,000 active licenses at TTCN-3 User Conference 2006 • Nokia experiences captured in IEEE Software 23(4) 2006 • Motorola reports doubling of testing productivity • Also used beyond Europe • Strong community in China

  12. Application Areas New application areas: • Unified testing approach for software and protocol testing • IP based protocols • Text-based protocols ... Additional communication paradigm: • Message-based communication • Procedure-based communication Area of Testing • Conformance and Functionality Testing • Pre-integration Testing • Integration and Interoperability Testing • Regression Testing • Load/Stress Testing • Dynamic concurrent testing configurations

  13. TTCN-3 Test Case Port.send(Stimulus) Port.receive(Response) Port System Under Test Black-box testing with TTCN-3 • Assignmentof aTest Verdict

  14. Main Capabilities of TTCN-3 • Dynamic concurrent testing configurations • Various communication mechanisms (synch and asynch) • Data and signature templates with powerful matching mechanisms (including regular expressions) • Attributes for encoding, display or user-defined information • Test suite parameterization • Control of Test Case execution and selection mechanisms • Control of complex test configurations • Assignment and handling of test verdicts • Harmonized with ASN.1 (XML and IDL coming) • Different presentation formats • Well-defined syntax, static - and operational semantics

  15. TTCN-3 test systems in a nutshell • TTCN-3 specifies a test but a test system is needed for test execution • TRI and TCI standards define test system architecture • TTCN- 3 tools are required to support internal interfaces • Allows reuse of test platforms with different tools but also for different SUTs • A test system requires • A TTCN-3 tool = TTCN-3 compiler and execution environment • A test platform for a specific device under test Note: Tools come with default Test Control & Logging Test System Executor TTCN-3 Test System Test Control Logging Codecs TCI [compiled] TTCN-3 Executable TRI SUT Adapter Platform Adapter System Under Test (SUT) TCI = TTCN-3 Control InterfaceTRI = TTCN-3 Runtime Interface

  16. Text format Tabular Format Graphical Format Presentation Format3 Presentation Formatn The Core Language and Other Presentation Formats • Core format is text based (most popular) • TTCN-3 can be edited or viewed in other formats • Tabular format (for TTCN-2 people) • Graphical format (good for visual overview) • Other standardized formats in the future? • Proprietary formats possible TTCN-3 Core Language

  17. Example Core (Text) Format testcase TC_resolveEtsiWww() runson DnsClient { timer t_ack; serverPort.send(m_dnsQuestion("www.etsi.org")); t_ack.start(1.0); alt { [] serverPort.receive(mw_dnsAnswer("172.26.1.17")) {setverdict (pass); } [] serverPort.receive { // any other message setverdict(fail); } [] t_ack.timeout { setverdict(inconc); } } t_ack.stop; }

  18. Example Graphical Format testcase TC_resolveEtsiWww() runson DnsClient mtc serverPort DnsClient DnsPort timer t_ack m_dnsQuestion("www.etsi.org") t_ack alt mw_dnsAnswer("172.26.1.17") pass ? fail t_ack inconc t_ack

  19. Example Tabular Format

  20. ASN.1 Types & Values IDL Types & Values XML Types & Values Other types & Valuesn Use of TTCN-3 With Other Languages • TTCN can be integrated with types systems of other languages • Fully harmonized with ASN.1 (1997) • Harmonized with other languages • IDL, XML, C/C++ TTCN-3 Core Language

  21. Minimal Test Configuration • All test behavior is executed on one (main) test component TTCN-3 Test System TCI Test Control TTCN-3 Executable Logging MTC Communication with IUT Codecs TRI SUT Adapter Platform Adapter Communication with SUT System Under Test (SUT) Communication with IUT Implementation Under Test

  22. Example Concurrent Test Configuration • A test involves execution of many parallel test components • Dynamic instantiation of components and communication links TTCN-3 Test System TCI Test Control TTCN-3 Executable MTC Logging InternalCommunication PTC PTC Codecs TRI SUT Adapter Platform Adapter System Under Test (SUT) Implementation Under Test

  23. Main elements of TTCN-3 • Module covers declarations and control • module name as file name • two (optional) parts in each module • definition part • control part – which test cases to execute (at the end of the module) • Testconfigurations • Ports • communication takes place through ports • FIFO Queue for incoming communication • head of the queue can be accessed • Components • test behaviour is executed on component instances • contain ports, timers, constants, variables • special components (Main test component: mtc, Test system interface: system)

  24. SUT start create TC TC TC MTC TCs TCs TCs TC TC TC create start start create Test Configurations (1) TTCN-3 Test Case MTC – Main Test Component TC – Test Component

  25. Test System Connected Ports INOUT TC2 OUT IN OUT IN Mapped Ports OUT IN Real Test System Interface Real Test System Interface SUT TC1 Test Configurations (2) Abstract Test System Interface

  26. P1 (out) P2 (in) P1.send (Msg) P2.receive (Msg) TC1 TC2 Main elements of TTCN-3 • Communication / Messages • Test components communicate via ports • A test port is modeled as an infinite FIFO queue • Ports have direction (in, out, inout) • There are three types of port • message-based, procedure-based or mixed

  27. Verdict returned by the test case when it terminates P F F I I MTC PTC1 P PTCN setverdict(fail) setverdict(inconc) setverdict(pass) Test verdicts • Test verdicts: none, pass, inconc, fail, error (set by runtime system) • Each test component has its own local verdict, which can be set (setverdict) and read (getverdict). • A test case returns a global verdict

  28. Data Types Test Data Test Configuration Test Behavior Ingredients of TTCN-3 • Built-in and user-defined generic data types (e.g., to define messages, service primitives, information elements, PDUs). • Actual test data transmitted/received during testing. • Definition of the components and communication ports that are used to build various testing configurations. • Specification of the dynamic test system behavior. TTCN-3

  29. Module Definitions Module Control Structure of TTCN-3 specs – TTCN-3 Modules • Modules are the building blocks of all TTCN-3 test specifications. • A test suite is a module. • A module has a definitions part and an (optional) control part. • Modules can be parameterized. • Modules can import definitions from other modules. Module

  30. Data Types Constants Signatures Data Templates Signature Templates Communication Ports Test Components Functions Altsteps Test Cases Structure of TTCN-3 specs – Module definitions part • Definitions are global to the entire module. • Data Type definitions are based on TTCN-3 predefined and structured types. • Templates define the test data. • Ports and Components are used in Test Configurations. • Functions, Altsteps and Test Cases define behavior.

  31. Module Definitions Module Control Structure of TTCN-3 specs – Module control part 1 (2) • Module control is the ‘dynamic’ part of a TTCN-3 specification where the test cases are executed. • Local declarations, such as variables and timers may be made in the control part. • Basic programming statements may be used to select and control the execution of the test cases. Module

  32. Structure of TTCN-3 specs – Module control part 2 (2) module … { … control{ var integer count; if(execute(SIP_UA_REC_V_001()) == pass) { // Execute test case 10 times count := 0; while( count <= 10) { execute(SIP_UA_REC_V_002()); count := count + 1; } // end while } // end if } // end control } // end module

  33. Reuse of existing code – The import mechanism • Import allows to import • single definitions • definitions of a certain kind, and • groups of definitions from other modules. • Definitions may be imported recursively. • If several definitions are addressed, certain definitions can be excluded by using an except directive. import form ModuleOne { modulepar ModPar2; type RecordType_T2 } import from ModuleTworecursive { testcase T_case } import from ModuleThree all except { template all }

  34. Data Types Constants Signatures Data Templates Signature Templates Communication Ports Test Components Functions Altsteps Test Cases Data types and test data – Constants, types, templates 1 (2) • TTCN-3 data types: • integer, char, universal char, float, boolean, objid • enumerated • bitstring, hexstring, octetstring, charstring, universal charstring, • record, record of, set, set of, union • anytype • verdicttype • address • Useful types • IEEE754float, long, byte, … • Iso8859string, utf16string, …

  35. Data types and test data – Constants, types, templates 2 (2) const integer int_Const := 7; // Normal constant external const boolean bool_Const; // External constant type record Request { // Structured type definition RequestLine requestLine, ReqMessageHeader reqMessageHeader optional, charstring crlf, charstring messageBody optional } template Request Invite := { // template for the Request type requestLine := Request_Line("INVITE"), reqMessageHeader := Req_Mes_Header("INVITE"), crlf := ?, messageBody := omit }

  36. Test Case Specification So far... • Message types to define communication • Communication via ports • Behavior is executed on components • Templates to express sending and receiving of values • Communication statements to communicate • Ordinary statements to express control flow • Verdicts to record the outcome Now, let’s put these together...

  37. Test Case Specification • Test Cases are a special kind of function. • In the module control part the execute statement is used to start test cases. • The result of an executed test case is always a value of type verdicttype. • Every test case shall contain one and only one MTC. • The behavior part defines the behavior of the MTC. • A test case header has two parts: • interface part (mandatory) - runs on • test system part (optional) - system TSI Type MTC Type Test Case Definition testcase Hello_Stephan runs on ClientComponent system SUT() { map(self:port, system:port); port.send(“Hello Stephan“); alt{ []port.receive(“Hello Colin“) {setverdict(pass)}; [else] {setverdict(inconc)} // Stephan asleep! } } Mapping the Ports Communication Test Verdicts

  38. Advantages of using TTCN-3 • Systematic product testing • Large percentage of testing coverage • Parallel testing and develop processes • Automated testing so it can be run without user intervention • Complex and multiple scenarios implementation • Clear and exact bug reports • Easy management of errors and bug fixing • Reusable tests and consistent version validation • Automated regression tests on all levels • Control and exact status of product quality in all development stages Testing strategies: • Validation of product basic functionality using SMOKE TEST • Deep and complete functional testing based on MSCs

  39. Conclusions • TTCN-3 is the current method of choice for the implementation of messages and test case logic of an automated test system. • TTCN-3 allows a high re-use but this requires a clever test system design. • TTCN-3 provides standardized interfaces to integrate the test cases to a complete test system. • TTCN-3 can be a good choice for implementing a future-oriented solution with a higher degree of test automation! • TTCN-3 is suitable for test automation in the embedded domain. Test the test system before it comes to test execution!

  40. TTCN-3 Project - overview Test Framework Serial PASS Power 0101111001101010111 0101111001101010111 MOST Msg - 540020010101 MOST Testcases TTCN-3 Adapter Java Blueooth.Enable.Status(On) MOST Msg - 5400200C0101 0101111001101010111 0101111001101010111 Bluetooth.Enable.Set(On) Codec Java System Under Test

  41. Literature on TTCN-3 • Standard documents: (can be found on the TTCN-3 homepage: http://www.ttcn3.org/ or http://www.etsi.org/ptcc/ptccttcn3.htm) • ES 201 873-1: TTCN-3 Core Language • ES 201 873-2: TTCN-3 Tabular Presentation Format (TFT) • ES 201 873-3: TTCN-3 Graphical Presentation Format (GFT) • ES 201 873-4: TTCN-3 Operational Semantics • ES 201 873-5: TTCN-3 Runtime Interface (TRI) • ES 201 873-6: TTCN-3 Control Interface (TCI) • ES 201 873-7: Using ASN.1 with TTCN-3 • ES 201 873-8: Using IDL with TTCN-3 • ES 201 873-9: Using XML with TTCN-3 • ES 201 873-10: Using C/C++ with TTCN-3 (planned) • Example test suites: • See: http://www.etsi.org/ptcc/ptccttcn3.htm • E.g., http://www.etsi.org/ptcc/ptccsip_osp.htm

More Related