Mutation-Based Testing

1. Mutation-Based Testing Kyle Mundt February 3, 2010

2. What is Mutation-Based Testing? • Richard Lipton, 1971 • A way of testing your tests • Alter your code in various ways • Check to see if tests fail on altered code • Becoming popular again due to increased computing power • Strongest use seems to be with Fortran and C (procedural languages)

3. Terms • Mutant: code resulting from applying a mutation operator • Mutation (or Fault) Operator: rule used to create a mutant • Killing a Mutant: when tests fail on a mutant • Mutation-Adequate: tests kill all mutants • Equivalent mutant: produces same output as original (can’t be killed) • Mutation score: killed / (total – equivalent)

4. Example Mutant if (a != 10) b = 3; else b = 5; Original if (a == 10) b = 3; else b = 5;

5. The Process • Apply mutation operators to code (only one change in each mutant) • Run mutants through test cases • Check mutants that do not fail • Equivalent mutants • Change test cases to catch mutant • Repeat

6. When to Perform? • Don’t begin too early • Code should already be written • Tests should be reasonably thorough • Begin with good set of test cases and code that passes tests • Iterative process • Create mutants, run tests, fix, repeat

7. Examples of Mutation Operators • Traditional Mutation Operators • Statement deletion • Invert logical operators • Replace arithmetic operators • Replace variable with another in same scope • Class-Level Mutation Operators • Object oriented • Change containers • Concurrency

8. Why is it Important? • If test results are the same • There is “dead code” • Mutant is equivalent to original • Test cases not complete • Code coverage not good enough • Functional testing not enough • Need way to analyze test effectiveness • Useful for hardware verification • Based on coupling effect: “ test data set that detects all simple faults in a program is so sensitive that it also detects more complex faults” • Ties into automatic test generation

9. Code Coverage • Hitting every line doesn’t guarantee good tests • Doesn’t ensure defects detected if they occur • Not good measure of verification effectiveness

10. Functional Testing • Testing functionality of program • Deals with how program should work • Mostly ignores how it shouldn’t • Functional tests are subjective • Also bad measure of verification quality

11. Analyzing Tests • Looking at, improving tests • Convinces us tests will detect defects if they occur • Similar to techniques done manually on hardware • Need way to automate it to be practical

12. Hardware verification • Verify testbed • Gives data to be used in improving chip testing • Hardware defects can cause unexpected behavior • Must ensure this behavior is caught • Attempt to show defects found by tests

13. Problems • Adequate automation • Huge (maybe infinite) possible number of mutants • Fault classification and prioritization • Which operators to use • How to apply to OOP, concurrency, etc.

14. Selective Mutation • Mothra – 22 operators • 6 operators account for 40-60% of mutants • Redundant mutants (can be killed by same test) • Research shows 5 operators gives good data • Gives 99% total mutant score and reduces number of mutants by 77%

15. Five Operators • ABS – make value of each arithmetic expression be 0, positive, and negative • AOR – replace arithmetic operator with valid operators • LCR – replace each logical operator • ROR – replace relational operators • UOI – insert unary operators in front of expressions • Note this is used on Fortran, a procedural language

16. Weak Mutation • Running whole program takes time • Don’t really care about what happens after mutant line hit • So, don’t execute whole mutant • Compare states of original and mutant after mutated line executed • Regular mutation requires three things • Mutant be reached • Mutant creates incorrect state in program • Program state reflected as difference in test output • Weak Mutation only requires the first two • Related to automatic test generation

17. Schema-Based Mutation • Put all mutants into “metaprogram” • One large program to compile and link • Mothra turns program into intermediate form • Modifies intermediate form to create mutants • Interprets intermediate forms (interpreting is slow) • Schema-based is much faster than interpretive systems

18. Other Optimizations • Can reduce number of test cases executed • Once a test case is killed, remove it from further testing • No reason to kill it twice • Random selection of mutants appropriate in some cases

19. OO Mutation • Easy to come up with operators for simple types • User types much harder • Operators have been developed for things like • Inheritance • Polymorphism • Method visibility • Harder to mutate based on meaning of object • Some research applied to Java OO mutation

20. Changing Object State • Methods proposed to alter at code level • Allows changes to things like attributes • Mutants made this way may not compile • Could cause integration errors • Still doesn’t look at meanings • Can write operators specifically for classes • Probably not practical in almost all situations • How to actually change object state?

21. One Approach • Mutate commonly used Java libraries • Again operators picked to try and simulate common mistakes • One solution found was to mutate Containers, Iterators, and InputStream • Java reflection can also be used to examine object fields at runtime

22. Some Suggested Operators • Collection Interface (most apply to List or Vector as well) • Make collection empty • Remove some element (first, last, random) • Reorder elements • Mutate element type • Iterators: • Skip element • InputStream • Skip bytes of data

23. Interface Mutation • First described for C programs • Designed for integration testing, mutate interface between modules • Designed to scale to larger systems • Actions taken to control number of mutants • Only look at integration errors • Tests only connections between pairs of subsystems • Mutates only module interfaces (eg. Function calls, return values)

24. Implementations • MuClipse • Open source mutation testing plug-in for Eclipse • µJava (muJava) • Heckle (Ruby) • Insure++ (C++) • Nester (C#)

25. Nester • C# programs for Visual Studios 2005 • Only supports NUnit Framework • Highlights code • Killed mutations in green • Surviving mutations in red • Code not covered by tests in blue • Allows use of XML-based grammar to define own transformation rules

26. Insure++ • Different approach • Used to find bugs in source code • Creates functionally equivalent mutants • Lines changed without changing expected results • Tests should all still pass • If a test fails, something wrong with code • Insure++ reports faults and lines responsible

27. Conclusion • More complicated programs equals greater need for quality tests • Need way to ensure/measure test effectiveness • Mutation-Based testing can fill this role • Still needs further research • Not a lot of implementations or examples of use on large scale • Getting attention in hardware verification • More work needs to be done to apply to OO

28. Sources • [1] Offutt, J. A. & Untch, R. H. (October 2000). Mutation 2000: Uniting the Orthogonal. Retrieved January 18, 2011 from http://cs.gmu.edu/~offutt/rsrch/papers/mut00.pdf • [2] Bakewell, G. (2010). Mutation-Based Testing Technologies Close the “Quality Gap” in Functional Verification for Complex Chip Designs. Retrieved January 18, 2011, from http://electronicdesign.com/article/eda/Mutation-Based-Testing-Technologies-Close-the-Quality-Gap-in-Functional-Verification-for-Complex-Chip-Designs/4.aspx • [3] Offut, J. A. (June 1995). A Practical System for Mutation Testing: Help for the Common Programmer . Retrieved January 18, 2011 from http://cs.gmu.edu/~offutt/rsrch/papers/practical.pdf • [4] Usaola, M. & Mateo, P. (2010). Mutation Testing Cost Reduction Techniques: A Survey. IEEE Software, 27(3), 80-86.  Retrieved January 23, 2011, from ABI/INFORM Global. (Document ID: 2012103051). • [5] Kolawa, Adam (1999). Mutation Testing: A New Approach to Automatic Error-Detection. Retrieved January 18, 2011, from http://www.parasoft.com/jsp/products/article.jsp?articleId=291 • [6] Alexander, R. T.; Bieman, J. M.; Ghosh, S.; & Ji, B (2002). Mutation of Java Objects. Retrieved January 18, 2011 from http://www.cs.colostate.edu/~bieman/Pubs/AlexanderBiemanGhoshJiISSRE02.pdf • [7] Nester - Free Software that Helps to do Effective Unit Testing in C#. http://nester.sourceforge.net/