PVR Murthy Andreas Ulrich Siemens AG, Corporate Technology

1. Essential Criteria on MBT to Ensure Quality of Software in Industry PVR Murthy Andreas Ulrich Siemens AG, Corporate Technology PVR Murthy Andreas Ulrich Siemens

2. Agenda • Model-based testing (MBT) • Case study • Effective criteria for evaluation of MBT tools • Analysis and discussions • Summary

3. Test Suite Test Suite Executable Scripts Executable Scripts Introduction to Model Based Testing Modeling Generating Executing Analyzing Results Coverage Algorithm Generate Test Cases Mental Model Create Executable Requirement or Design Documentation and/or Generate Executable Oracles • Decide whether to • Generate more tests • Modify the model • Stop testing • Estimate • Reliability & other • quality measures Test pass or Fail Verify Results Run Scripts Execute Manually Application Under Testing

4. Model-Based Testing • Model-based testing (MBT) is an evolving test generation technique. • Uses design artifacts (models) of a system under test. • Generate test cases automatically. • Suitable for adequateand thorough testing of critical and complex software.

5. MBT tools under study • A number of MBT tools are available • Commercial tools, e.g. T-VEC, Qtronic, Spec Explorer etc. • Academic tools, e.g. NModel, TGV/CADP etc. • We selected two tools for our study • Conformiq’sQtronic V2.0 • Microsoft’s Spec Explorer2010 V3.0 • They represent two different modeling paradigms • Qtronic uses both textual and graphical notations while describing the system model (based on hierarchical state charts). • Spec Explorer uses guarded state update rules in C# (based on abstract state machines, called model programs) and a coordination specification.

6. Conformiq’s Qtronic • Models are expressed as a collection of textual files and/or graphical models. • The textual notation is defined using QML • Graphical models are created using either Conformiq Modeler or a third party UML editor. • Generation of test cases based on state space analysis • Can translate test cases into executable test scripts

7. Microsoft’s Spec Explorer • Uses Visual Studio programming environment to model system behavior. • Input is a set of .NET model assemblies and a coordination script. • Generates test cases based on exploration of the model. • Exploration systematically discovers all possible states and transitions. • Exploration can be visualized through a state space graph. • Test cases are a collection of C# files executable in MSTest

8. We have considered an application from Siemens medical domain. Considered SUT:An automated exposure control software. Make large images with an automatic sequence of many small images Modeled as a UML state chart with 11 system states, 24 transitions and 72 transition paths. Digital radiographic system Case Study

9. Evaluation Criteria • For a Siemens industrial project in the healthcare domain • High quality, reliable and robust software is indispensable • It therefore requires rigorous and effective testing • Effective and rigorous testing of critical and highly reliable software requires a good MBT tool / methodology. • We have identified eight important parameters for evaluating the considered MBT tools. • Based on the Siemens requirement in the domain of highly critical and reliable software. • Identified parameters are subjective.

10. Criterion 1: Model Representation • Aspects considered • Type of model (tester or design model). • Kind of notation (standard or proprietary). • Ease of creation and editing (expressiveness, reusability, etc) • Modeling advanced features (hierarchy, concurrency). • Findings

11. Criterion 2: Model Validation • Aspects Considered • Detecting requirement defects. • Consistency checking. • Checking design errors in the models. • Findings

12. Criterion 3: Test Generation Strategy • Aspects considered • Tool support for choosing test coverage criteria. • Efficiency of test case generation algorithm. • Test generation control to avoid test case explosion • Findings

13. Criterion 4: Test Data Generation • Aspects Considered • Automation in test data generation. • Accuracy, thoroughness, cost of testing. • Findings

14. Criterion 5: Concurrency support • Aspects Considered • Ability to test concurrent system behavior. • Findings

15. Criterion 6: Testing Levels • Aspects Considered • Unit, integration, or system testing. • Findings

16. Criterion 7: Regression testing • Aspects Considered • Capability to generate test cases for regression testing. • Test case prioritization, Test case optimization • Findings

17. Criterion 8: Usability • Aspects Considered • Initial learning curve, • Availability of documentation and on-line help. • User friendliness. • Findings

18. Relevance of criteria for considered case study • Selection of relevant criteria for applying MBT in large industrial projects • Leads to an informed selection of the right MBT approach and tool • Highlights weak points of the selected tooling

19. Outlook – Suggested improvements for future generation tools • Integration of different testing levels in a single framework. -- Unit, Integration testing and System testing • Need more rigorous automatic test case selection techniques. -- Sophisticated constraint specification over models • More sophisticated automatic test data generation. --Applying right combination of standard techniques for effective fault revealing data -- Handling primitive as well as structured data types. • Improved tool interoperability for different test artifacts. -- Migration from one environment to another

20. Summary • In addition to the criteria mentioned already as a basis • for understanding MBT approaches for application in • industry, criteria for effective MBT may also include • fault modeling support for applications in different domains • model debugging support • guiding modelers towards incorporating safety and liveness properties of applications in models • facilitating expression of test design , test verdict and data concepts from UML Testing profile. • support for modeling non-functional requirements

21. Thank You – Any Questions?