MOGENTES 3 rd and Final Review Reporting Period January 2010 – March 2011 Cologne, 26 May 2011

1. MOGENTES 3rd and Final ReviewReporting Period January 2010 – March 2011 Cologne, 26 May 2011 Scientific Results – UML/(OO)AS track Harald Brandl, Elisabeth Jöbstl (TUG)

2. Outline • Introduction • Models and Transformations • Test Case Generation and ioco • Empirical Evaluation of Test Selection Strategies • TCG Results for Demonstrators • Symbolic TCG • Conclusions

3. model Model-Based Mutation Testing Requirements fault models mutated models conformance test case generator conformance conformance SUT (black box) test case executor verdicts

4. Used Models and Transformations • UML: • class diagrams & state charts • hierarchical, parallel states • constraints over quantors etc. • OOAS/AS: • object-oriented action systems • action systems • tool Argos • LTS: • labelled transition systems • tool Ulysses |[ var v : T := init methods M1;...;Mn actions A1 = g1-> v := e1; ...; Am = gm -> Mi(ei); do A1 ☐ A2 ; A3 // Am od ]| : MI

5. Test Case Generation Action System Model IOLTSS discriminating test case ioco ? for every mutant Mutated Action Systems IOLTSM

6. Input Output Conformance (ioco)

7. Input Output Conformance (ioco)

8. Mutation-Based Test Case Generation

9. Ulysses Tool • Mutation-based Test Case Generator • Implemented in SICStus Prolog • Modes: 1) interpretation, 2) compilation to plain Prolog • Animator to step through a model • Generation of random test cases • Support for hybrid systems  Qualitative Action Systems • Extension of Hybrid Action Systems by Rönkkö et al. • Integration of qual. simulation tool ASIM by Bandelj et al. • Discrete event interpretation of continuous processes

10. Test Case Selection Strategies Fault-based strategies: A1 Brute-force mutation killer A2 Target-oriented mutation killer A3 Adaptive mutation killer A4 Lazy mutation killer A5 Lazy ignorant mutation killer A6 Random first mutation killer linear test cases adaptive test cases State of the art (for comparison): A7 Random A8 Test purposes (TGV)

11. Test Case Generation Metrics Ford „Car Alarm System“

12. Assessment of Test Suites • Classicalmutationtesting: • 38 faulty implementations • Test caseexecutionresults: • Comparisonto UPPAAL track: • UPPAAL TCs kill 80% of UML mutants

13. TCG Results FFA-CAS – 10 actions

14. TCG Results RELAB – 137 actions

15. TCG Results PROL

16. TCG Results TRSS/ELEKTRA

17. TCG Results TRSS/ELEKTRA More complexmodel: • 10 trainroutes • 118 objects (reducibleto 64) • Tool limitations: • statespaceexplosion (memory) • Ulysses random TCG with 30 GB RAM: • 1 TC of depth 30 + 1 TC of depth 32 (118 objects) • 1 TC of depth 33 (64 objects) • 1 - 6 hoursfor1 TC • Online testing • Symbolictestcasegeneration

18. Symbolic Test Case Generation • no explicit state enumeration • implemented translation • conformance check via constraint solver: • National project: TRUFAL • THALES, AIT, TUG, AVL

19. Conclusions • Formal model-based mutation testing: • object-oriented and hybrid systems • Action system modelling language: • object-oriented and qualitative extensions • Tools: • Argos • Ulysses • Empirical studies • works well for usual embedded systems

20. Publications 1/2 • H. Brandl, G. Fraser, F. Wotawa: A report on QR-based testing (QR 2008) • H. Brandl, G. Fraser, F. Wotawa: Coverage-based testing using qualitative reasoning models (SEKE 2008) • B. Aichernig, M. Weiglhofer: Unifying input output conformance (UTP 2008) • B. Aichernig, H. Brandl, W. Krenn: Qualitative action systems (ICFEM 2009) • W. Krenn, R. Schlick, B. Aichernig: Mapping UML to Labeled Transition Systems for Test-Case Generation - A Translation via Object-Oriented Action Systems (FMCO 2009) • B. Aichernig, H. Brandl, E. Jöbstl, W. Krenn: Model-based Mutation Testing of Hybrid Systems (FMCO 2009) • B. Aichernig, H. Brandl, F. Wotawa: Conformance testing of hybrid systems with qualitative reasoning models (MBT 2009) • W. Krenn, B. Aichernig: Test case generation by contract mutation in Spec# (MBT 2009)

21. Publications 2/2 • M. Weiglhofer, B. Aichernig, F. Wotawa: Fault-based conformance testing in practice. Int. Journal of Software and Informatics, 3(2–3):375–411, Jun/Sep 2009. Chinese Academy of Science. • H. Brandl, M. Weiglhofer, B. Aichernig: Automated Conformance Verification of Hybrid Systems (QSIC 2010) • B. Aichernig, H. Brandl, E. Jöbstl, and W. Krenn: UML in Action - A Two-Layered Interpretation for Testing (UML&FM 2010) • E. Jöbstl, M. Weiglhofer, B. Aichernig, F. Wotawa: When BDDs fail: Conformance testing with symbolic execution and SMT solving (ICST 2010) • B. Aichernig, H. Brandl, E. Jöbstl, and W. Krenn: Efficient Mutation Killers in Action (ICST 2011) • E. Jöbstl: Symbolic Model-Based Mutation Testing. Research Abstract (Doctoral Symposium at FM 2011)

22. Appendix

23. Test Models • Systems are specified with UML • Because of semantic variations of UML  Action Systems • Action Systems are defined via weakest precondition semantics • Actions are guarded commands • Parallel composition • Refinement between action systems • Extensions for hybrid systems • Labeling of actions  execution yields a Labeled Transition System (LTS)

24. Car Alarm System

25. Car Alarm System: 76 Mutations, e.g.

26. Linear Test Cases • A1: inconclusive pass product tree with set of unsafe states product graph with set of unsafe states linear test case(s) • A2: • like A1 but directly on product graph

27. Adaptive Test Cases • A3: inconclusive unsafe pass fail 1 adaptive test case per unsafe state product graph with set of unsafe states

28. Adaptive Test Cases • A4: • like A3 but no new TC if existing TC covers a given unsafe state • A5: • like A3 but no new TC if existing TC kills the given mutant • A6: • combination of A5 and random testing