1 / 28

Semantic Mutation Testing

Semantic Mutation Testing. John A. Clark, Haitao Dan, and Robert M. Hierons. Andreas Voraberger 13.01.2014. Content. Introduction Traditional Mutation Testing The i dea Mutation count Possibilities Advantages / Disadvantages Semantic Mutation Testing The idea

hestia
Download Presentation

Semantic Mutation Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Semantic Mutation Testing John A. Clark, Haitao Dan, and Robert M. Hierons Andreas Voraberger 13.01.2014 AK Softwaretechnologie 1 WS 2013/14

  2. Content • Introduction • Traditional Mutation Testing • The idea • Mutation count • Possibilities • Advantages / Disadvantages • SemanticMutation Testing • The idea • Implementing semantic mutation • Advantages • Semantic Error Model • Common misunderstandings • Refinement • Translation • Example (Cruise-control system) • Conclusion AK Softwaretechnologie 1 WS 2013/14

  3. Introduction • Testing is important but expensive • 50% of development budget • Test automation • More efficient/effective • Mutation Testing • Distinguishing between descriptions (N) • Semantic Mutation Testing • Distinguishing between semantics (L) AK Softwaretechnologie 1 WS 2013/14

  4. Introduction • Code defined by the pair: (N, L) • N: the description (source code) • L: thesemanticsofthelanguage AK Softwaretechnologie 1 WS 2013/14

  5. Traditional Mutation Testing AK Softwaretechnologie 1 WS 2013/14

  6. The idea • Mutants are classes of faults • Produce mutants (N’) • Making changes to the program • Apply mutation operators • {+, -} • {<=, <} • {deleteparts} • (N,L)  (N‘,L) • Test Cases should distinguish N/N‘ (mutant killed) AK Softwaretechnologie 1 WS 2013/14

  7. Mutation count • Single mutation operator creates large number of mutations • Even in small programs • Only use “first-order mutants“ • Producedbyapplication ofoneoperator • Hypothesis: all “first-order mutants“ kill most “higher-ordermutants“ • Equivalentmutant: not kill able AK Softwaretechnologie 1 WS 2013/14

  8. Possibilities • Judge test data against the mutants created • Produce test data to kill all mutants • A test set that is good in distinguishing N from N’ is good at finding faults AK Softwaretechnologie 1 WS 2013/14

  9. Advantages • Target particular classes of faults • Optimal for small slips or typos • Simulate other test criteria • Mutants lead to errors  provide code coverage AK Softwaretechnologie 1 WS 2013/14

  10. Disadvantages • High number of mutants, even with • Only “first-order mutants“ • Small programs • Solution: “selective mutation“ • Equivalent mutants • Leads to manual effort • Increases costs • Doesn’t aim on semantic mistakes AK Softwaretechnologie 1 WS 2013/14

  11. Semantic Mutation Testing (SMT) Overview AK Softwaretechnologie 1 WS 2013/14

  12. The idea • Produce mutants (L’) • (N,L)  (N,L‘) • Discover misunderstandings related to semantic mistakes • Requires description language with a semantics that allows manipulation AK Softwaretechnologie 1 WS 2013/14

  13. Implementing semantic mutation • Parameterisable system • Mutate through changing the parameters • Express semantics that can be manipulated • A set of rewrite rules • Simulate a mutation of the semantics • Changes to the syntax throughout the description AK Softwaretechnologie 1 WS 2013/14

  14. Advantages • Lead to far fewer mutants • Fewer equivalent mutants • Tests translation, refinement, migration, porting code • Eliminates misunderstandings AK Softwaretechnologie 1 WS 2013/14

  15. Semantic Mutation Testing (SMT) Scenarios AK Softwaretechnologie 1 WS 2013/14

  16. Semantic Error Model • Change of description in development process • Abstract  Concrete • Requirements • Specification • Design • Code • Machinecode • Misunderstandings in everytransformation AK Softwaretechnologie 1 WS 2013/14

  17. Semantic Error Model AK Softwaretechnologie 1 WS 2013/14

  18. Semantic Error Model • Refinement • Transformation to different abstraction level • Translation • Transformation on the same abstraction level • Basis formisunderstandings • Solution: SMT AK Softwaretechnologie 1 WS 2013/14

  19. Common misunderstandings • Find set of semantic misunderstandings • For different languages • Based on studies or experience • Implement tests, finding them • Set of operators should reflect the environment • Different mistakes depending on programmer skills AK Softwaretechnologie 1 WS 2013/14

  20. Refinement • Semantic mistakes between abstraction levels • Problems on copying syntax • Maybe different semantics • Examples • Truncation rules • Precedence rules • Datatypes • Floating point precision • Binary representation AK Softwaretechnologie 1 WS 2013/14

  21. Migration / Porting (Translation) • Migration / Porting to a different language • Maybe different semantics • Use a semantic test case generation tool • Suits of semantic mutation operators for common combinations • Point to common failures • Example: C to Z • Division of negative numbers • Branchingstructures • floating-point comparison AK Softwaretechnologie 1 WS 2013/14

  22. Semantic Mutation Testing (SMT) Example AK Softwaretechnologie 1 WS 2013/14

  23. Example • Shows • cruise-control system • Not only slips have to be tested • Developers can make semantic mistakes AK Softwaretechnologie 1 WS 2013/14

  24. Statechart for a cruise-control system AK Softwaretechnologie 1 WS 2013/14

  25. Cruise-control system (Case 1) • State: no_vehicle_in_front • Events: • brake • level=increase • What happens? AK Softwaretechnologie 1 WS 2013/14

  26. Cruise-control system (Case 2) • State: no_vehicle_in_front • Events: • Vehicle detected • level=increase • What happens? AK Softwaretechnologie 1 WS 2013/14

  27. Conclusion • Semantic Testing is necessary • SMT is a working practice • Captures a different failure type than MT • Focuses on misunderstandings • Normal MT still necessary AK Softwaretechnologie 1 WS 2013/14

  28. Quelle • John A. Clark, Haitao Dan, and Robert M. Hierons. Semantic mutation testing. In Third International Conference on Software Testing, Verification, and Validation Workshops, pages 100-109. IEEE, 2010 AK Softwaretechnologie 1 WS 2013/14

More Related