1 / 59

Mutation Testing Dagstuhl 08/08

Mutation Testing Dagstuhl 08/08. Yue Jia , Mark Harman King’s College London CREST. Agenda. Background Problems Solution Empirical Study Future Work. Background. Problems. Solution. Empirical Study. Future Work.

laksha
Download Presentation

Mutation Testing Dagstuhl 08/08

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Mutation TestingDagstuhl 08/08 YueJia, Mark Harman King’s College London CREST

  2. Agenda Background Problems Solution Empirical Study Future Work Background Problems Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  3. Mutation Testing Background Problems Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  4. Mutation Testing Background Problems Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  5. Mutation Testing Which test case is better? Background Problems Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  6. Mutation Testing Background Problems Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  7. Mutation Testing Background Problems Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  8. Mutation Testing Background Problems Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  9. Mutation Testing Select test cases that would distinguish the program under test from alternative programs that contain hypothetical faults (Pezzand and Young) distinguish Background Problems Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  10. Mutation Testing Select test cases that would distinguish the program under test from alternative programs that contain hypothetical faults (Pezzand and Young) distinguish hypothetical faults • Effectiveness or ability to detect faults Background Problems Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  11. Mutation Testing Step 1 . Generate Mutants ------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------ + + Original Program Background Problems Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  12. Mutation Testing Step 1 . Generate Mutants ------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------ + + + + Original Program First Order Mutant Background Problems Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  13. Mutation Testing Step 1 . Generate Mutants ------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------ + + + + Original Program First Order Mutant Higher Order Mutant Background Problems Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  14. Mutation Testing Step 2. Execute Mutants against Test set T Test set m m m m Killed (Different result) Survived (Same result) Background Problems Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  15. Mutation Testing Step 3.1. Mutation Score Background Problems Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  16. Mutation Testing Step 3.2. Generate test case to kill Test set New Test cases Killed Survived m m m m Background Problems Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  17. Equivalent Mutant An equivalent mutant is syntactically different from the original program, but has the same behavior. e.g. for(inti = 0; i < 10; i++){…} for(inti = 0; i != 10; i++){…} Background Problems Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  18. Problem 1970s Computational cost e.g. Triangle LoC 50 FOM 500 HOM 500500 cost Problems Background Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  19. Solution 1980s Coupling effect hypothesis cost Problems Background Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  20. Solution 1980s Simple Complex cost Problems Background Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  21. Solution 1980s Simple Complex cost If a test set kills simple faults, it also kills the complex faults Problems Background Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  22. Solution 1980s Coupling effect hypothesis cost Problems Background Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  23. Solution 1990s Selective mutation / Mutation sampling cost + - > < a b Problems Background Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  24. Trivial Fault Problems Background Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  25. Subtle fault • 80% of the real faults are complex fault (Dallmeier and Zimmermann) ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Problems Background Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  26. Subtle fault • 80% of the real faults are complex fault (Dallmeier and Zimmermann) ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- HOMs -> Subtle faults Problems Background Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  27. Solution 2000s Higher order mutation testing cost Problems Background Solution Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  28. Solution 2000s Higher order mutation testing cost …Search Solution Background Problems Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  29. Higher Order Mutation Testing Increase the subtlety Reduce the Oracle effort Reduced number of equivalent mutants Solution Background Problems Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  30. Higher Order Mutant Most common case Test set T Tc Ta Tb Tabc Solution Background Problems Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  31. Higher Order Mutant Subsuming higher order mutant Tc Ta Tb Tabc Solution Background Problems Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  32. Higher Order Mutant Strong Subsuming higher order mutant Tc Ta Tb Tabc Solution Background Problems Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  33. Higher Order Mutation Testing Increase the subtlety Reduce the Oracle effort Mutants Test cases Reduced number of equivalent mutants Solution Background Problems Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  34. Higher Order Mutation Testing Increase the subtlety Reduce the Oracle effort Mutants Test cases Reduced number of equivalent mutants Solution Background Problems Empirical Study Future Work Mark Harman and YueJia King’s College London, CREST Centre

  35. Triangle Program Classify triangle by the lengths of the sides Equilateral Isosceles Scalene EmpiricalStudy Background Problems Solution Future Work Mark Harman and YueJia King’s College London, CREST Centre

  36. Triangle Program Sides constraint Scalene Isosceles Equilateral (a,b,c >0) a + b > c a + c > b b + c > a b a a + b > c (a==b) c => a(b) + c > b(a) N/A EmpiricalStudy Background Problems Solution Future Work Mark Harman and YueJia King’s College London, CREST Centre

  37. Triangle Program Variable trian int trian = 0; if ( a = b ) trian = trian + 1; if ( a = c ) trian = trian + 2; if ( b = c ) trian = trian + 3; b a c EmpiricalStudy Background Problems Solution Future Work Mark Harman and YueJia King’s College London, CREST Centre

  38. Triangle Program Variable trian trian = 6 trian = 1,2,3 trian = 0 EmpiricalStudy Background Problems Solution Future Work Mark Harman and YueJia King’s College London, CREST Centre

  39. Triangle Program Original Program if ( trian == 0 ) ... if ( trian == 6 ) ... if ( trian == 1 && a + b > c ) return ISOSCELES if ( trian == 2 && a + c > b ) return ISOSCELES if ( trian == 3 && b + c > a ) return ISOSCELES EmpiricalStudy Background Problems Solution Future Work Mark Harman and YueJia King’s College London, CREST Centre

  40. Valuable HOMs HOMs if ( trian == 1 && a + b > c ) return ISOSCELES if ( trian == 2 && a + c > b ) return ISOSCELES if ( trian == 3 && b + c > a ) return ISOSCELES trian > 1 a + b <= c EmpiricalStudy Background Problems Solution Future Work Mark Harman and YueJia King’s College London, CREST Centre

  41. Subsuming HOM M1 a == b && a + b > c a == c && a + b > c && a + c <= b b == c && a + b > a && b + c <= a trian > 1 if ( trian == 1 && a + b > c ) Ori Mut IOS INV 1 INV IOS 1 1 INV IOS EmpiricalStudy Background Problems Solution Future Work Mark Harman and YueJia King’s College London, CREST Centre

  42. Subsuming HOM M1 a == b && a + b > c a == b && a + b <= c a + b <= c if ( trian == 1 && a + b > c ) Ori Mut IOS INV 2 INV IOS 2 EmpiricalStudy Background Problems Solution Future Work Mark Harman and YueJia King’s College London, CREST Centre

  43. Subsuming HOM M12 a == b && a + b > c a == c && a + b > c && a + c <= b b == c && a + b > c && b + c <= a a == b && a + b <= c a == c && a + b <= c b == c && a + b <= c trian > 1 a + b <= c if ( trian == 1 && a + b > c ) Ori Mut IOS INV 1 2 × INV INV 1 × INV INV 1 × 2 INV INV × 1,2 INV IOS × 1,2 INV IOS EmpiricalStudy Background Problems Solution Future Work Mark Harman and YueJia King’s College London, CREST Centre

  44. Subsuming HOM M1 a == c && a + b > c && a + c <= b b == c && a + b > a && b + c <= a M2 a == b && a + b <= c • M12 • a == b && a + b > c EmpiricalStudy Background Problems Solution Future Work Mark Harman and YueJia King’s College London, CREST Centre

  45. Algorithms Greedy Hill Climbing Genetic Algorithm EmpiricalStudy Background Problems Solution Future Work Mark Harman and YueJia King’s College London, CREST Centre

  46. Representation Vector of integers • Index -> position • Value -> type 1 0 0 3 0 4 0 …M1…… …………… ……M4… …M6…… EmpiricalStudy Background Problems Solution Future Work Mark Harman and YueJia King’s College London, CREST Centre

  47. Fitness EmpiricalStudy Background Problems Solution Future Work Mark Harman and YueJia King’s College London, CREST Centre

  48. Tools (Père David's Deer) MiLu Deer Horse Donkey Strongly Subsuming HOM Cow rare valuable EmpiricalStudy Background Problems Solution Future Work Mark Harman and YueJia King’s College London, CREST Centre

  49. Tools MiLu Higher Order Full C Search Approach Test harness embedding www.dcs.kcl.ac.uk/pg/jiayue/milu/ EmpiricalStudy Background Problems Solution Future Work Mark Harman and YueJia King’s College London, CREST Centre

  50. Results • Mutants LoC FOM sHOM Triangle 50 584 47 Tcas 150 679 98 Schedule2 350 1014 78 Totinfo 500 2570 320 Printtokens 750 866 67 Space 6000 7570 522 EmpiricalStudy Background Problems Solution Future Work Mark Harman and YueJia King’s College London, CREST Centre

More Related