Automatic Program Correction

1. Automatic Program Correction Anton Akhi Friday, July 08, 2011

2. Plan • Introduction • Survey of existing tools

3. Why do we need automatic program correction? • Even if bug has been found it is still programmer’s job to think of fix • Allows to fix bug automatically or provide high-quality fix suggestions

4. Isn’t it a magic? • Tools require failing and passing runs • Oracle to check if test passes • Program with bug is not far away from the right one

5. Existing tools • Genetic programming: • Automatic Program Repair with Evolutionary Computation • A Novel Co-evolutionary Approach to Automatic Software Bug Fixing • Machine learning: • BugFix • Contract usage: • Automated Debugging using Path-Based Weakest Preconditions • AutoFix-E • AutoFix-E2

6. Automatic Program Repair with Evolutionary Computation • W. Weimer, S. Forrest, C. Le Goues, T. Nguyen • Usage of GP to fix bugs: • Individuals • Genetic Operators • Fitness function

7. Genetic programming: individuals • Individual is represented as an abstract syntax tree and weighted path • Weighted path is a list of statements visited on negative test case with weight of every statement: • 1 if not visited by positive tests • 0.1 if visited by positive tests

8. Genetic programming: operators • Mutation: • statement on a weighted pass is considered for mutation with probability proportional to its weight • deletion, insertion, swap • Crossover: • exchange of subtrees chosen at random between two individuals

9. Genetic programming: fitness function • Weighted sum of test cases passed • Weight of a negative test is at least as much as positive test

10. Minimizing changes • Trim unnecessary edits • One-minimal subset of changes is a subset such that without any of the changes program will stop passing all the tests • Use delta debugging to compute one-minimal subset of changes

11. Automatic Program Repair with Evolutionary Computation: Conclusions • Needs: • negative and positive test cases • oracle • fault localization • Uses: • genetic programming • Delta debugging • Method has a lot of criticism

12. A Novel Co-evolutionary Approach to Automatic Software Bug Fixing • A. Arcuri and X. Yao • Genetic Programming • Distance Functions • Search Based Software Testing • Co-evolution

13. Genetic Programming • Genetic Program consists of primitives • Program is represented in a tree form • Fitness function to be minimized: • is a number of nodes in a program • is a number of raised exceptions • is a special distance function

14. Distance Functions • Works fine with numbers and boolean expressions • Predicates involving and can be handled only in cases small

15. Search Based Software Testing • Find tests that make evolutionary programs fail • Fitness function for test case to be maximized:

16. Co-evolution • Competitive co-evolution • First generation: copies of buggy program and unit tests • Mutations, crossover, replacement by the original program • Penalty for short programs

17. A Novel Co-evolutionary Approach to Automatic Software Bug Fixing: Conclusions • Needs: • starting set of unit tests • oracle • Uses: • genetic programming • co-evolution • search based software testing • Some bugs are too difficult to solve • Bug that is difficult to be fixed by a human might be very easy for program

18. BugFix: A Learning-Based Tool to Assist Developers in Fixing Bugs • D. Jeffrey, M. Feng, N. Gupta, R. Gupta • Association rule learning • Interesting value mapping pairs (IVMP) • Situation descriptors • Knowledgebase of rules

19. Association rule learning • Association rule learning is a popular method for discovering the relationship between variables in database • Association rule where X, Y are sets of attributes and means that if the items in set X are present then it is probable that the items in set Y are also present • The confidence of the rule is where supp(X) is the fraction of transactions containing X

20. Interesting Value Mapping Pairs • An Interesting Value Mapping Pair (IVMP) is a pair of value mappings (original, alternate) associated with a particular statement instance in a failing run, such that: (1) original is the original value mapping used by the failing run at that instance; and (2) alternate is an alternate (different) value mapping such that if the values in original are replaced by the values in alternateat that instance during re-execution of the failing run, then the incorrect output of the failing run becomes correct

21. Situation descriptors • Statement structure situation descriptors • IVMP pattern situation descriptors • Value pattern situation descriptors

22. Statement structure situation descriptors • Unordered tokens comprising the statement

23. Statement structure situation descriptors: examples

24. IVMP pattern situation descriptors • Consider pattern to occur when corresponding values in the IVMPs compare to each other in the same way across all IVMPs at a statement • Compare in terms less than, greater than, or equal to • Look at the pairs of values: • within original sets of values • within alternate sets of values • between corresponding values in sets

25. IVMP pattern situation descriptors: examples

26. Value pattern situation descriptors • Similar to IVMP patterns

27. Knowledgebase of rules • Database of bug-fix scenarios • Initially created through training data of known debugging situations

28. How it works • Identify rules to consider • rules in which debugging situation is subset of the current debugging situation • Sort rules by confidence values • Report prioritized bug-fix descriptions • Learn from current debugging situation and the corresponding bug fix

29. BugFix: A Learning-Based Tool to Assist Developers in Fixing Bugs: Conclusions • BugFix assists in fixing bugs by producing a list of bug-fix suggestions • Tool learns through new situations • Is not very good with new and logically difficult bugs

30. Automated Debugging using Path‑Based Weakest Preconditions • H. He, N. Gupta • Representation of an error trace • Path-based weakest precondition • Hypothesized program state • Actual program state • Detection of evidence • Location and modification of likely errorneous statement

31. Representation of an error trace • is an instance of an executed statement in an error trace; i is an execution point and j is the line number of statement • Branch predicates: • atomic predicate: (exprrelop const) • compound predicates are in disjunctive normal form: where and is an atomic predicate • Precondition and postcondition for failing run are transformed in DNF: • replace and by disjunction and conjunction

32. Path‑based weakest precondition • pwp(T, R) is the set of all states such that an execution of function F that flows execution trace T begun in any of them is guaranteed to terminate is state satisfying R • where means substituting every occurrence of x in R with a • where B is branch predicate

33. Hypothesized program state • Let where is a trace from point i to the end of trace and R is postcondition • defines the set of hypothesized program states at an execution point i • and

34. Actual program state • Represented by predicates in DNF which are true for given input • Consists of forward program states and backward program states

35. Forward and backward program states • Forward program states are defined as: • positive conjunctions in precondition • and are sets of predicates killed by and derived from statement • Backward program states are defined as: • if is an assignment statement • if is a branch predicate

36. Detection of evidence • A is less restrictive than B if is false • Evidence at point i is situation when is less restrictive than • Two types of evidence: • Explicit • Type I • Type II • Implicit

37. Types of evidence • Explicit • Type I: if at point i in appears negative r in form (0 relop const) then r forms an explicit evidence of Type I • Type II: let an atomic predicate in and a negative atomic predicate in If then q and r form an explicit evidence of Type II • Implicit: negative predicate r in that is not present in an explicit evidence

38. Location and modification of likely erroneous statement • Use transitivity and equality to deduce new predicates. New states are and • Explicit Type I • If = then match r to 0=0. Consider every assignment statement between i and bottom of the trace as a possible candidate for modification • Let from be a corresponding predicate to r, and and • Goal is to make , so • If then problem is solved

39. Location and modification of likely erroneous statement: Explicit Type I • Consider e and c as a set of strings of characters • Let and be difference between e and c and between c and e correspondingly • If appears in rhs than replace it with • If appears in lhs than replace it with only if is a single variable • If none of the above works than select r as e and try every q from as c

40. Location and modification of likely erroneous statement: Explicit Type II • Consider • Either q or r could be in error • Change the form of r to q at i • Same manner as above • Change the form of q to r at i • Change original branch predicate from which q may be derived or some assignment statement • Change of an assignment statement does not change relop in q

41. Location and modification of likely erroneous statement: Implicit • If there is a loop in the trace, which contributes some constraints on , and missed constraints have similarity with constraints added by the loop then try to derive the possible missing iterations in the loop • Try to match negative r from to some q from

42. Automated Debugging using Path‑Based Weakest Preconditions: Conclusions • Uses contracts • Can handle only one error • Cannot handle loops well

43. Automated Fixing of Programs with Contracts • Yi Wei, Yu Pei, C.A. Furia, L.S. Silva, S. Buchholz, B. Meyer, A. Zeller • Assessing Object State • Fault Analysis • Behavioral Models • Generating Candidate Fixes • Linearly Constrained Assertions • Validating and Ranking Fixes

44. Assessing Object State • Argument-less Boolean Queries • Absolutely describe state • Seldom preconditions • Widely used in Eiffel contracts • Complex Predicates • Boolean queries are often correlated • Implication expresses correlation • Mine the contracts • Mutate implications

45. Fault Analysis • Find state invariants: passing and failing states – sets of predicates that hold during all passing and failing runs respectively • Fault profile contains all predicates that hold in the passing run but not in the failing run • Find the strongest predicate that implies the negation of violated assertion

46. Behavioral models • Finite state automaton • States are predicates that hold • Transitions are routines • Automaton is built based on test runs • Determine a sequence of routine calls that change the object state appropriately • A snippet for a set of predicates is any sequence of routines that drive object from a state where none of predicates hold to one where all of them hold

47. Generating Candidate Fixes • Fix Schemas • snippet is a sequence of routine calls • old_stmt – some statements in the original program • fail:

48. Linearly Constrained Assertions • Determine what is variable and what is constant • Assign weights: • Arguments in precondition receive lower weights • In assertion weight is inversely proportional to the number of occurrences • Identifiers that routine can assign receive less weight

49. Linearly Constrained Assertions: Generating fixes • Select a value for variable that satisfies the constraint • Look for extremal values • Plug the value into a fix schema • if notconstraintthennew_stmtelseold_stmtend

50. Validating and Ranking Fixes • Candidate is valid if it passes all the tests • Two metrics for ranking: • Dynamic: estimates the difference in runtime behavior between the fix and the original based on state distance • Static: • OS: 0 for schemas (a) and (b) and number of statements in old_stmt for (c) and (d) • SN: number of statements in snippet • BF: number of branches to reach old_stmt from the point of injection of the instantiated fix schema