1 / 20

Making TTCN-3 work!

Making TTCN-3 work!. Issues and strategies for its use in product development. Martin Botteck, Thomas Deiß, Colin Willcock Nokia Research Center Bochum, Germany. TTCN-3 adoption today:. Generally, uptake is slower than anticipated. Speed-up could be possible through :

ordell
Download Presentation

Making TTCN-3 work!

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Making TTCN-3 work! Issues and strategies for its use in product development Martin Botteck, Thomas Deiß, Colin Willcock Nokia Research Center Bochum, Germany Making TTCN-3 work

  2. TTCN-3 adoption today: • Generally, uptake is slower than anticipated. Speed-up could be possible through : • Handling variations of the testing task • Integrating TTCN-3 into existing test environments • Using TTCN-3 in the hardware development process Making TTCN-3 work

  3. Overview Variations in Test Systems TTCN-3 in existing test environments TTCN-3 for testing hardware Conclusion/Summary Making TTCN-3 work

  4. Variations in Testing • Within a set of test cases there often are several commonalities • Example: 3G “Call” • several types of calls (voice, data,…) and calling routes • behaviour will in many cases be similar, but not the same • “intuitive” solution: copy/paste code from one test case to another “similar” one • -> maintenance problem when the 3G specification is updated • alternative solution: add a parameter in the test case invocation • -> code will become hard to read Making TTCN-3 work

  5. Types of variations • Value Variants • Variations that may be expressed by altering a parameter inside a test case or message sent to the SUT • seems trivial but soon becomes difficult due to • vast amount of parameters and combinations • typically not all combinations make sense • dependancies between the values is typically not documented and implicit • a language or coding style solution needs to be found for this Making TTCN-3 work

  6. Types of variations (ctd) • Behavioural Variants • Variations that may be expressed by a different sequence of messages between the SUT and the tester • several messages or sets of sequences may remain the same • some are different • decomposition of the SUT may reveal common abstract test case structures with respect to the SUT components • but what about complex composition variants of such components? • -> Abstract Modelling of the tester? (UML testing profile, Test Purpose Language) Making TTCN-3 work

  7. Types of variations (ctd) • Variants of the System under test • product customisation • evolution of conformance standards • Might be expressed by altering a parameter (e.g. “feature_x=OFF”, “call_type=‘rich_call’”) • in combination with reference to a set of message sequences • Typically, a vast amount of such combinations will be needed • How to solve this without copy/paste of code but still avoiding combinatorial logic to the user? Making TTCN-3 work

  8. Overview Variations in Test Systems TTCN-3 in existing test environments TTCN-3 for testing hardware Conclusion/Summary Making TTCN-3 work

  9. The dilemma of choice • Nowadays most R&D departments already perform testing: • Conformance testing, hardware testing, product testing, module testing, unit testing… • Proprietary test systems (“home grown” or “tailor made”) • Multitude of vendors with closed environments • Trying out a system is a lot of effort. So, shortcomings of any choice being made show up too late for reversing the choice Making TTCN-3 work

  10. Thomas Deiß: Usually, in the TTCN-3 world, the platform and the SUT adapter are considered to work side by side, not the PA on top of the SA. MBo: Better this way? It does not become clear to me, what are the stages: Tools, Testware (development), Test sytem (development?), Test execution. But 'tools' is not a stage. MBo: Indeed. Let´s talk Editorial comment: Test system <-> Test Execution, note different capitalization of the second word Run Time System Executable Test System (ETS) Testware Test System Test Development Tools Test Execution Codec Platform Adapter Control Log Analysis Editor Compiler Codec Generator … Codec Router Test cases SUT SUT Adapter The testing tool chain • for each stage components from different vendors need to interoperate • clearly defined interfaces are needed Making TTCN-3 work

  11. Run Time System Executable Test System (ETS) Testware Test Execution Test Development Tools Test System Codec Platform Adapter Control Log Analysis Test cases Codec Router Editor Compiler Codec Generator … SUT SUT Adapter The testing tool chain interfaces • TTCN-3 core notation language for exchange of test cases • Run Time IF for Adaptors and Codecs • Test Execution Control and logging through TCI TCI/Log TRI Making TTCN-3 work

  12. Further work on interfaces • Tool vendors have a notion to “close shop” and not support public interfaces • Codec interface part in TCI leads to non-interchangeable codecs • realisation of a given specification in IDL depends on the target language. • Java based tools and C based tools require different codecs • performance problems when separating the codec to a separate executable • Standardisation sometimes lags behind (e.g. TCI) • It is also desirable to generate as many test cases and parts of the environment as possible directly from the behavioural description of the IUT • -> modelling representations (e.g. XMI) need to be supported in the tools Making TTCN-3 work

  13. Overview Variations in Test Systems TTCN-3 in existing test environments TTCN-3 for testing hardware Conclusion/Summary Making TTCN-3 work

  14. Algorithmic description informal model, e.g. C source code Decomposition (Block diagram) Highest Level behavioural model (e.g. SystemC) Notion of “Time” Bit accurate Hardware model (e.g. VHDL) Notion of “Clock Cycles” Introduction of physics (layout, transistor, …) Physical Model (e.g. RTL, Spice,…) Hw design flow • Description consistent through all stages/levels • Automatic generation of tests (“Test vectors”) • Back annotation integral part of the process Making TTCN-3 work

  15. use case description formal model, e.g. UML UC Decomposition (package diagram) Highest Level behavioural model (e.g. UML sequence diagram, state chart) Refinement, Introduction of “Signals” more detailed behavioural model (e.g. UML sequence diagram, state chart) Introduction of programming language syntax (C, C++, Java,…) executable (e.g. binary, script,…) Sw design flow • Rather “novel” approach • Automatic generation of tests (“Test vectors”) still in experimental phase • Back annotation is not formally specified Making TTCN-3 work

  16. Algorithmic description informal model, e.g. C source code use case description formal model, e.g. UML UC Decomposition (package diagram) Decomposition (Block diagram) Highest Level behavioural model (e.g. SystemC) Highest Level behavioural model (e.g. UML sequence diagram, state chart) Notion of “Time” Refinement, Introduction of “Signals” Bit accurate Hardware model (e.g. VHDL) more detailed behavioural model (e.g. UML sequence diagram, state chart) Notion of “Clock Cycles” Introduction of programming language syntax (C, C++, Java,…) Introduction of physics (layout, transistor, …) Physical Model (e.g. RTL, Spice,…) executable (e.g. binary, script,…) “Traditional” HW-SW integration HW Prototype SW release Integration SW release HW prototype Integration Tests Making TTCN-3 work

  17. Integration tests are not determined by formal derivation from HW/SW high level models Integration typically is the 1st stage where HW and SW meet Integration results feed back “somewhere” in the process in SW HW and SW tests only derive from HW and SW models respectively How about utilisation of SW models (e.g. High level behavioural) for generation of HW tests? including behavioural HW models (SystemC) in High level behavioural SW models? generation of tests directly from the model representation? Common nodes in the HW and SW design flow Making TTCN-3 work

  18. Overview Variations in Test Systems TTCN-3 in existing test environments TTCN-3 for testing hardware Conclusion/Summary Making TTCN-3 work

  19. Summary/Conclusions • Further deployment of TTCN-3 needs convincing advantages: • integration into existing test environments • interfaces, replaceable components from heterogenous test systems • handling of variants through specific coding constructs • automated generation of test cases from abstract models • cover HW as well as SW through adequate representation in SW models Making TTCN-3 work

  20. Thank you! Questions? Acknowledgement: The authors would like to thank Mr. Risto Teittinen from Nokia Networks for his contributions to this topic Making TTCN-3 work

More Related