Automating the Generation of Mutation Tests

1. Automating the Generation of Mutation Tests Mike Papadakis and NicosMalevris DepartmentofInformatics Athens University of Economics and Business

2. Test Data Generation Approaches • Symbolic execution • Select set of paths • Produce a system of algebraic constraints • Solve and generate test cases • Challenges (Symbolic execution) • Infeasible paths • Complex-unhandled expressions • Availability of source code

3. Test Data Generation Approaches • Search based approaches • Definition of program input • Dynamic program execution • Fitness function • (efficiency and effectiveness) • Challenges • Handling of dynamic program inputs • Require a high number of executions • Handling of specific cases (e.g. flag problem) • Effective fitness function

4. Test Data Generation Approaches • Dynamic Symbolic Execution • Simultaneously perform actual and symbolic execution • Simplify the process based on concrete program execution • Produce a system of algebraic constraints • Solve and generate test cases • Challenges • Infeasible paths • Scalability issues

5. Killing Mutants • In order to kill a mutant, tests must Stage 1: Reach the mutant Stage 2: infect the program state Stage 3: propagate the infected state

6. Killing Mutants • Reach the mutant (Reachability condition) • Infect program state (necessity condition) • Program execution must cause a discrepancy in the program state • E.g. Orig: a > b, Mut = a ≥ b → a > b ≠ a ≥ b • a == b where Orig = false, Mut = true • Propagate (sufficiency condition) • Heuristic approaches • explore path space, measure of mutant’s impact • Joint satisfaction (Reach && Infect && Propagate)

7. Killing Mutants (path selection) • Symbolic Execution • Reach • Use paths from program input to mutant • Infect • Use necessity constraints to infect program state • Original Statement ≠ Mutated Statement • Propagate • Explore the path space (from mutant to program output) • Path selection Strategy

8. Killing Mutants(path selection) • Enhanced Test Model (Enhanced Control Flow Graph) • Represent mutants in the model • Guide the selection of paths and the produced constraints based on the Shortest path strategy

9. Killing Mutants(path selection) • Shortest Path Strategy, the k-value • An efficient path selection strategy. (eliminates the effects of infeasible paths) • Select shortest paths from program input (S) to the mutant statement (m) • Heuristic approach to weak mutation • Strong mutation can be tackled incrementally

10. Killing Mutants Dynamic approaches • Dynamic approaches • Gain the required information from actual program execution • Require high number of program execution • Mutation Testing • Need to unify the runtime information of the original and mutant programs • Need to include mutant neccesity and sufficiency conditions into program structure • Need for efficiency (high cost)

11. Mutant Schemata • Mutant schemata-Parameterized mutants • Technique for efficiently producing mutants • Embeds all mutants into one meta-mutant program • Use of global parameter to specify the mutants • Example • original meta-mutant • a > b + c → RelationalGT(a, ArithmeticPlus(M(b), M(c))) • M(x) mutates variable x, e.g. abs(x) • Arithmetic(x, y) mutates arithmetic operator, e.g. x - y • Relational(x, y) mutates relational operator, e.g. x ≥ y

12. Mutant Schemata (approach I) • Weak mutation • Include the mutant evaluation inside schematic functions • Original expression ≠ Mutant expression • Original statement ≠ Mutant statement • After evaluation continue with the original execution (schematic function returns the original expression) • Able to execute all mutants with one execution run • Embeds all mutant’s killable condition into program structure (mutant evaluation → branches)

13. Mutant Schemata • Proposed approach (Mutants to branches) • Mutants M[1]…M[n] on node N. Mutant Evaluation Branches Mutant is killed Mutant is Alive

14. Mutant Schemata (approach II) • Strong mutation • In Same lines as Weak • Include the mutant evaluation inside schematic functions • After evaluation continue with the mutant execution (schematic function returns the mutant expression) • Incremental approach, from weak to strong • Requires multiple execution runs • Record original and mutant’s execution paths • Evaluate strongly killed mutants

15. Killing Mutants (DSE) • Use the shortest path heuristic • Negate the condition that will lead closer to mutant node (reach the mutant) • Weakly kill the mutant • Fulfill the mutant necessity condition • Make the mutant evaluation Branch true • Negate the produced conditions after the mutant • Start from the mutant program • Explores the mutant program path space

16. DSE-Example Scenario Target Mutant

17. DSE-Example Scenario Target Mutant

18. DSE-Example Scenario Target Mutant

19. DSE-Example Scenario Target Mutant

20. DSE-Example Scenario Target Mutant Infect Mutant Mutant is alive

21. DSE-Example Scenario Target Mutant Propagate Mutant is alive

22. DSE-Example Scenario Target Mutant Mutant is alive

23. DSE-Example Scenario Target Mutant Mutant Killed

24. DSE-Necessity condition Example: a + b > k → a + abs(b) > k RelationalGT(ArithmeticPlus(a, Abs(b)), k) Abs(x) → if (x<0) //mutant necessity condition a + b > k ≠ a + abs(b) > k//Statement level condition • Test: a=15, b=2, k=0 • Conditions: b≥0 && a+b>k && … • Negates:b≥0 →b < 0 • Produces: a=15, b=-10, k=0

25. DSE-Necessity condition Example: a + b > k → a + abs(b) > k RelationalGT(ArithmeticPlus(a, Abs(b)), k) Abs(x) → if (x<0) //mutant necessity condition a + b > k ≠ a + abs(b) > k//Statement level condition • Test: a=15, b=-10, k=0 • Conditions: b<0 && a+b>k && a+abs(b)>k && … • Negates:a+abs(b)>k →b<0 && a+b>k && a+abs(b)≤k → Infeasible • Negates:a+b>k →b<0 && a+b≤k • Produces:a=-15, b=-10, k=0

26. DSE-Necessity condition Example: a + b > k → a + abs(b) > k RelationalGT(ArithmeticPlus(a, Abs(b)), k) Abs(x) → if (x<0) //mutant necessity condition a + b > k ≠ a + abs(b) > k//Statement level condition • Test: a=-15, b=-10, k=0 • Conditions:b<0 && a+b≤k && a+abs(b)≤k && … • Negates:a+abs(b)>k →b<0 && a+b≤k && a+abs(b)>k • Produces:a=5, b=-10, k=0 • Mutant infected: Orig: 5-10>0 (false), Mut: 5+10>0 (true)

27. Killing Mutants (Search Based) • Weak-Strong mutation • Measure the closeness of reaching a mutant • Measure branch distance of mutant branches • Closeness of weakly killing the targeted mutant (mutant necessity condition) • Use simplified necessity fitness for improved performance • Sufficiency condition can be approximated by exploring the path space or based on mutants impact

28. Fitness function • Approach level • Closeness of executing a targeted mutant • Calculated by the number of control dependent nodes missed • Branch Distance • Closeness of flipping a specific branch • Mutation Distance • Closeness of weakly killing the targeted mutant

29. Mutant Fitness (example)

30. Path explosion problem Mutants • Original program • 8 program paths • Mutant program • 8 program paths per mutant • Schematic program • Mutant evaluation decisions • 8 * 2number of mutants paths • Introduction of fake paths • Same of them may be useful for higher order mutation Mutants Mutants Mutants

31. Handling of path explosion • Path selection strategy, (symbolic execution) • Use weights on the Enhanced Control Flow Graph • Shortest paths include only the original paths • Dynamic Symbolic Execution • Use the mutant parameters as local values • Dynamic production of constraints eliminates the fake paths. Paths are produced dynamically • Search based approaches • Approach level contains only the control depended nodes • Ignores the path space

32. Case Studies • Dynamic Symbolic Execution (Strong mutation) • Program units: Tritype, Remainder • Siemens Programs: Schedule, Tcas, Replace • ABS, AOR and ROR operators • Search based (Weak mutation results) • Hill climbing approach (AVM) • Maximum 10 attempts per mutant • Program units: Tritype, Triangle, Remainder, Calendar • ABS, AOR, ROR and LCR operators

33. DSE Study-Results

34. DSE Study-Results

35. DSE Study-Results

36. DSE Study-Results

37. DSE Study-Results

38. DSE Study-Results • Strong mutation results • Replace 86.5%, Tcas100%, Schedule 91%(compared with the accompanied test suite), • Remainder 97.5 and Tritype96.8% (all killable mutants)

39. Search Based Study-Results

40. Search Based Study-Results

41. Search Based Study-Results

42. Search Based Study-Results

43. Search Based Study-Results • Weak mutation results

44. Conclusion Mutation based test case generation Lack of attempts First steps using dynamic, state of the art techniques Dynamic approaches can be adopted to perform mutation Search based approaches Mutation distance leads to improved effectiveness Dynamic Symbolic Execution Quite effective

45. Conclusion • Test data generation approaches and tools can be effectively extended to mutation • Practical Technique (schemata) • Future directions • Efficient handling of infeasible paths • Handling of the “flag” problem • Efficient handling of equivalent mutants • Hybrid approaches

46. Thank you for your attention… Questions ? Contact Mike Papadakis mpapad@aueb.gr Nicos Malevris ngm@aueb.gr

47. References • Mike Papadakis and NicosMalevris. "Automatic Mutation Test Case Generation Via Dynamic Symbolic Execution", in 21st International Symposium on Software Reliability Engineering (ISSRE'10), San Jose, California, USA, November 2010. • Mike Papadakis and NicosMalevris. “Metallaxis an Automated Framework for Weak Mutation", Technical Report, http://pages.cs.aueb.gr/~mpapad/TR/MetallaxisTR.pdf. • Mike Papadakis, NicosMalevris and Maria Kallia. "Towards Automating the Generation of Mutation Tests", in Proceedings of the 5th International Workshop on Automation of Software Test (AST'10), Cape Town, South Africa, May 2010, pp. 111-118. • Mike Papadakis and NicosMalevris. “An Effective Path Selection Strategy for Mutation Testing", in Proceedings of the 16th Asia-Pacific Software Engineering Conference (APSEC'09), Penang, Malaysia, December 2009, pp. 422-429.