Using a Fault Hierarchy to Improve the Efficiency of DNF Logic Mutation Testing

1. Using a Fault Hierarchy to Improve the Efficiency of DNF Logic Mutation Testing Gary Kaminski and Paul Ammann ICST 2009

2. Logic Mutation Testing Process Mutation testing helps testers find test inputs that detect common faults • yield code coverage • subsume other testing criteria • Create program variations (mutants) each containing a single logic fault if (a && b)  if (a || b) • For each mutant, find an input distinguishing the original and mutant (weakly killing) - a && b and a || b evaluate to different truth values Focus of this paper is step 1)

3. Motivation Typical mutation system: • includes mutation operators in “Introduction to Software Testing” by Ammann and Offutt • generates excess logic mutants • does not detect certain logic faults Goal: - generate fewer logic mutants - detect more logic faults Assumption: - assume minimal Disjunctive Normal Form (DNF) but will relax this in future research

4. Minimal DNF Terms separated by OR, literals by AND ab + a!c vs. a(b + !c) • Make each term true and other terms false ab + ac vs. ab + abc • Impossible to remove a literal without changing the predicate semantics ab vs. abc + ab!c Green – in minimal DNF Red – not in minimal DNF

5. Multiple Identical Mutants Typical Tool may generate same mutant multiple times Original: ab + bc - (true)b + bc (replace literal a with true) - bb + bc (replace literal a with b) Both mutated predicates reduce to the same logic mutant: ab + bc has been mutated to b

6. Equivalent Mutants Typical Tool may generate equivalent logic mutants Original: ab + b!c + !bc Replacing b with c in term ab yields equivalent mutant: ac + b!c + !bc

7. Unnecessary Mutants Typical Tool may generate mutants unnecessarily Original: ab + bc Negating literal a yields: !ab + bc (killed by TTF or FTF) Replacing a with c yields: cb + bc (killed by TTF only) Mutant produced by negating literal a not needed Killing the mutant where literal a is replaced by literal c guarantees killing the mutant where literal a is negated

8. Selective Mutation Operators Missing Typical Tool lacks highly selective mutation operators A mutation operator inserting a literal into a term is lacking Original: ab + cd Mutant not produced: abc + cd (killed by TTFT) Test point of TTFT guarantees killing many other mutants which thus would not need to be generated

9. Minimal DNF Logic Faults Original: ab + bc • Literal Insertion Fault (LIF)* abc + bc • Literal Reference Fault (LRF)* ac + bc • Literal Omission Fault (LOF) a + bc Detecting LIF, LRF, LOF guarantees detecting other faults *LIF and LRF can result in equivalent mutants

10. TRF/LIF New Mutation Operator • Produce a fault where a term is replaced by one or more terms each containing a LIF • The resulting fault can only be detected by the point that is the intersection of the set of points that detects each LIF Original: ab + !acde • abc + !acde detected by TTFXX • abd + !acde detected by TTXFX • abe + !acde detected by TTXXF abc + abd + abe + !acde detected only by TTFFF Generate one TRF/LIF mutant instead of three LIF mutants Similar to Subsuming Higher Order Mutants (Jia/Harman)

11. TIF/LOF New Mutation Operator • Insert a new term that can only be detected by the point that is the intersection of the set of points that detects 1 or more LOFs • Original: ab + cd • LOFs: 1) b + cd detected by any false point where a is F and b is T 2) a + cd detected by any false point where a is T and b is F 3) ab + d detected by any false point where c is F and d is T 4) ab + c detected by any false point where c is T and d is F LOFs 1 and 3 can be detected by FTFT (set intersection) LOFs 2 and 4 can be detected by TFTF (set intersection)

12. TIF/LOF New Mutation Operator (Continued) • Insert a new term that can only be detected by FTFT in one mutant • Insert a new term that can only be detected by TFTF in another mutant • Generate two TIF/LOF mutants instead of four LOF mutants • Original: ab + cd • TIF/LOFs 1) ab + cd + !ab!cd detected only by FTFT 2) ab + cd + a!bc!d detected only by TFTF Similar concept applies for TIF/LRF mutants New mutation approach that produces TRF/LIF, TIF/LRF, TIF/LOF mutants is called TRF-TIF mutation

13. LIF LOF LRF TOF LNF ORF. ORF+ TNF ENF Lau and Yu’s Fault Hierarchy • Arrow indicates any point detecting the source fault will detect the destination fault • Lacks faults produced by typical mutation operators • Lacks faults produced by new highly selective mutation operators • Ignores effect of equivalent faults (LIF  LRF and LRF  LOF) - When an equivalent LIF exists for a term, detecting all non-equivalent LIFs for that term does not guarantee detecting all non-equivalent LRFs for literals in that term (moves LRF up so now it must be detected outright) - When an equivalent LIF exists for a term, detecting all corresponding non-equivalent LRFs guarantees detecting all LOFs for literals in that term (adds new arrow)

14. TRF/LIF TIF/LRF TIF/LOF LRF LOF LIF SVRLOF TOF LSTF1 SVRTOF ORF. LSTF0 ORF.1 ORF.0 LNF ORF+1 ORF+ TNF ESTF1 ORF+0 ESTF0 ENF Extended Fault Hierarchy Red Box – Mutation Operator does not exist Blue Box – Mutation Operator partially exists BlackBox – Mutation Operator exists Green Arrow– Detection may not hold Black Arrow – Detection always holds

15. Mutation Efficiency Ratio Mutation Efficiency Ratio (MER): Faults Detected / Mutants Generated • FD assumes an input is found to kill the mutant • FD includes equivalent mutants detected automatically • FD incorporates all the faults in the extended fault hierarchy (previous slide) Typical Mutation System has at best a MER of 1 • generate mutants that correspond to faults in a 1 to 1 manner • equivalent mutants not detected automatically makes MER < 1 TRF-TIF has higher MER • generates fewer mutants - no equivalent mutants (assuming a complete Boolean space) - never produces multiple identical mutants - never produces a mutant guaranteed to be killed by a test killing some other mutant • detects more faults (LIF and LRF)

16. Case Study • Analyzed 19 predicates (Weyuker, Chen, Lau, and Yu) in avionics software having 5 to 13 unique literals • Built a tool in Java to produce TRF-TIF mutants • Compared MER for typical mutation tool and TRF-TIF tool • Examined how MER varies based on number of unique literals

17. Case Study Results • TRF-TIF tool generated 6% of the mutants a typical mutation tool would • Typical tool guarantees* detecting 76% of faults the TRF-TIF tool does *assumes inputs are found that kill all generated non-equivalent mutants • For the TRF-TIF tool, MER (FD / MG) increased as the number of unique literals increased

18. Mutation Efficiency Ratio (FD/MG) vs. Number of Unique Literals • TRF-TIF tool • Detects faults typical tool does not (LIF and LRF) • Does not generate multiple identical mutants • Does not generate mutants guaranteed to be killed by a test killing • another mutant • Uses highly selective mutation operators not in typical tool

19. Conclusion • Altered a logic fault hierarchy to include: new highly selective and typical mutation operators effects of equivalent faults • Developed a new TRF-TIF logic mutation approach that: eliminates equivalent mutants eliminates identical mutants detects new fault types generates highly selective mutants • Comparison of TRF-TIF approach with typical approach showed improved efficiency with applications for testing software having large predicates