1 / 24

Mutation Testing (ch 5)

Mutation Testing (ch 5). Course Software Testing & Verification 2013/14 Wishnu Prasetya. Negative Test. Recall: to test a program outside its normal behavior range. Idea: mutate inputs.

wardah
Download Presentation

Mutation Testing (ch 5)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Mutation Testing(ch 5) Course Software Testing & Verification 2013/14 WishnuPrasetya

  2. Negative Test • Recall: to test a program outside its normal behavior range. • Idea: mutate inputs. • For inputs described by BNF we can mutate the grammar instead (5.5.2), e.g. replace/delete/ duplicate terminals and non-terminals in a production rule. • Or mutate on-the-fly as we build derivation trees (so that we can control how many times a mutated rule is applied).

  3. Example NLpostcode Street Area NLpostcode Area Space Street Area  FirstDigit Digit Digit Digit Street  Letter Letter FirstDigit 1 | 2 ... Digit  0 | 1 | 2 ... Letter  a | b | c ... | A | B | C ... Space   | “ “ Space Area  FirstDigit Digit Digit Area  FirstDigit Digit DigitDigitDigit

  4. Some coverage concepts for grammar-based mutants generation • (Def 5.46) Mutation operator (e.g. non-terminal deletion); (Def 5.47) mutant • (C 5.33) Mutation Operator Coverage (MOC) : for every mutation operator TR contains one mutant produced via this operator. • (C5.34) Mutation Production Cov: for every production rule and every mutation operator that can be applied to the rule, TR contains one mutant via that rule and mutation.

  5. Using mutation as “coverage” (5.2) • We can “mutate” a program to “fake” a mistake; but the mutant must be a valid program. • (D 5.45) Ground : the original. • (D 5.48) Let P be a ground. A test t of P kills a mutant P’ of P if t’s output of both differ. • We can use mutants to measure the adequacy of our test-set: it is good if it can kill all those mutants. • Obviously, the strength of this scheme depends on your choice of the mutation operators.

  6. Example Fundamental premise : If a program contains a fault, there will usually be a set of mutants that can only be killed by test cases that also detect that fault. (mutations generated by the tool Mothra)

  7. We can also do: mutation-driven testing Fundamental premise : If a program contains a fault, there will usually be a set of mutants that can only be killed by test cases that also detect that fault. not enough mutants percentage killed a test-case fails • Things to note (see next slide) : • reachability of a mutation • infection • propagation This is a simpler variation of the scheme in Fig. 5.2. The one above does not presume automated generation of test-cases.

  8. How to kill a mutant? P(x) { if (x<0) x++ ; return x } P(x) { if (x0) x++ ; return x } • (reachability) you need a test-case whose execution passes the mutation. • You need a test-case that actually observes the effect of the mutation. Note there is a difference in: • (infection) “incorrect” state, right after the mutation. • (propagation) the infection propagates to incorrect P’s output. In principle, a test-case can only kill if there is propagation. test() { r = P(1) ; assert (r==2) } test() { r = P(0) ; assert (r==0) }

  9. Another example • booleanisEven (int X) { • if (X < 0) • 3 X = 0 – X ; • ∆ 3 X = 0 ; • if ((double) (X/2) == ((double) X) / 2.0) • return true ; • else return false ; • } test1() { assert (isEven(-2)==true) }

  10. Issues with killing • No reachability  usually can easily be fixed. • You have propagation, but used oracle is only partial (is not strong enough).  fix the oracle. • You have infection, but no propagation  as in the isEven example  find another test-case. • What if we weaken the definition of “kill” ? We can at least see what that gives us.

  11. Strong and weak kill • Let t be a test on a ground P, and P’ is a mutant. Let l be the location of the mutation in P’. • (D 5.49) t strongly kills P’ if its verdict on P’ and P are different. In other words, if it fails on P’. • ( D 5.50) t weakly kills P’ if the states right after l when executing t on P’ and P are different. P(x) { if (x<0) x++ ; return x } P(x) { if (x0) x++ ; return x } test() { r = P(1) ; assert (r0) } does not even weakly kill test() { r = P(0) ; assert (r0) } does not kill, but still weakly kill

  12. Mutation-based coverage criteria • (C5.32) Given a set M of mutants, the TR is simply M: you have to kill every mutant in M. • C5.35 if you use “strong killing”, C5.36 for “weak killing”. • OA remark that in practice strong vs weak do not make much difference. • The strength depends on the choice of mutation operators. See 5.2.2 and 5.3.2. • Typical problem in mutation-based testing: it generates lots of mutants, roughly O(#dataobjs*#refs)  make testing very computation intensive! • O(#refs) using selective mutation (next slide)

  13. Mutation operators • This depends on the used programming language. • We can start with some set of mutation operators, then see if we can minimize it. • (D5.51) Given a set O of mutation operators; a subset O1 is effective if test-cases designed specifically to kill mutants produced by O1 also kill mutants from O/ O1. • There have been plenty of research in this direction. For mutations over imperative constructs, see 5.2.2. Mutations on OO aspects, see 5.3.2.

  14. Examples of mutops on imperative constructs (5.2.2) • ABS (Absolute Value Insertion)Modify each arithmetic (sub)expression by using functions abs(), negAbs(), and failOnZero(). • AOR (Arithmetic Operator Replacement)Replace each occurrence of arithmetic operators +,－,*,／, % by each other; and in addition, by leftOp, and rightOp. • SVR (Scalar Variable Replacement)Replace each variable reference by another (type compatible and in-scope) variable. • BSR (Bomb Statement Replacement)Replace each statement by Bomb(). • More... see book.

  15. Addressing integration issues (5.3.2) • Faults in component integration are caused by a mismatch of assumptions; e.g. callee thought: • a list was sorted, caller did not do that • all fields were initialized, caller only initialized some • thought values are in kilometers, caller use miles • Integration mutation focuses on mutating the connections between components • Sometimes called “interface mutation” • Both caller and callee methods are considered

  16. Four types of integration mutations • Change a calling method by modifying the call (e.g. deleting the call; replacing the returned obj by a type-compatible obj). • Change a calling method by modifying values that are sent to a called method. • Change a called method by modifying values that enter and leave a method • Includes parameters as well as variables from higher scopes (class level, package, public, etc.)

  17. How about OO?These are typical OO features; also where typical mistakes are made • Encapsulation (hiding details).

  18. Typical OO aspects • Inheritance: method overriding, variables shadowing, constructors overriding works differently • Polymorphism • a required object of type T can be instantiated by a subclass at the runtime • overloading • In addition to object level members we also have class level members (the ‘static’). • 5.3.3 lists 20 mutops targeting the above aspects.

  19. Examples of inheritance mutops • OMD (Overriding Method Deletion)Each entire declaration of an overriding method is deleted. • OMM (Overridden Method Moving)Each call to a super.method is moved to the first and last position, and up and down one statement. override m() { x ++ ; y = y/x ; base.m() ; // super}

  20. Examples of inheritance mutops HVI – Hiding Variable Insertion HVD – Hiding Variable Deletion point int x; int y; point int x; int y; colorpoint 1 int x; 2 int y; colorpoint int x; 1 // int x; int y; 2 // int y;

  21. Examples of inheritance mutops point colorpoint ATC – Actual Type Change DTC – Declared Type Change point p; p = PointFactory();  p = new colorpoint (); point p; colorpoint p; p = PointFactory();

  22. point3D void set (int x, int y, char c); void set (char a, int x, int y); point3D void set (int x, int y, int z); void set (int x, int y); void set (int z); Examples of overloading mutops AOC – Argument Order Change ANC – Argument Number Change point point3D p ; p.set (1, 2, ‘t’) ;  p.set (‘t’, 1, 2) ; point3D p; p.set (1, 2, 3);  p.set (2, 3);  p.set (3);

  23. point int x = 5;  int x; … point point() { … }  // point() { … } … point … void set (int x, int y) { this.x = x; 1 x = x; this.y = y; 2 y = y; } … Examples of “other” mutops VID –Variable Initialization Deletion DCD – Default Constructor Delete TKD – This Keyword Deletion

  24. Ok... but how useful is this !? • Mutation is widely considered the strongest test criterion; but also the most expensive! • This because the TRs of ‘other ‘criteria can be encoded by a mutation (assuming you are completely free to define your mutops), at least with respect to weak killing (other criteria impose local requirements, like weak mutation) • Node coverage, edge coverage, clause coverage • But.. e.g. correlated active clause coverage  cannot be expressed in terms of mutation killing, as the TR is expressed in terms of pairs of tests(unless we also define a concept of killing a pair of mutants)

More Related