1 / 63

Delta Debugging

Delta Debugging. Koushik Sen EECS, UC Berkeley. How do we go from this.

odell
Download Presentation

Delta Debugging

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Delta Debugging Koushik Sen EECS, UC Berkeley

  2. How do we go from this <td align=left valign=top><SELECT NAME="op sys" MULTIPLE SIZE=7><OPTION VALUE="All">All<OPTION VALUE="Windows 3.1">Windows 3.1<OPTION VALUE="Windows 95">Windows 95<OPTION VALUE="Windows 98">Windows 98<OPTION VALUE="Windows ME">Windows ME<OPTION VALUE="Windows 2000">Windows 2000<OPTION VALUE="Windows NT">Windows NT<OPTION VALUE="Mac System 7">Mac System 7<OPTION VALUE="Mac System 7.5">Mac System 7.5<OPTION VALUE="Mac System 7.6.1">Mac System 7.6.1<OPTION VALUE="Mac System 8.0">Mac System 8.0<OPTION VALUE="Mac System 8.5">Mac System 8.5<OPTION VALUE="Mac System 8.6">Mac System 8.6<OPTION VALUE="Mac System 9.x">Mac System 9.x<OPTION VALUE="MacOS X">MacOS X<OPTION VALUE="Linux">Linux<OPTION VALUE="BSDI">BSDI<OPTION VALUE="FreeBSD">FreeBSD<OPTION VALUE="NetBSD">NetBSD<OPTION VALUE="OpenBSD">OpenBSD<OPTION VALUE="AIX">AIX<OPTION VALUE="BeOS">BeOS<OPTION VALUE="HP-UX">HP-UX<OPTION VALUE="IRIX">IRIX<OPTION VALUE="Neutrino">Neutrino<OPTION VALUE="OpenVMS">OpenVMS<OPTION VALUE="OS/2">OS/2<OPTION VALUE="OSF/1">OSF/1<OPTION VALUE="Solaris">Solaris<OPTION VALUE="SunOS">SunOS<OPTION VALUE="other">other</SELECT></td><td align=left valign=top><SELECT NAME="priority" MULTIPLE SIZE=7> <OPTION VALUE="--">--<OPTION VALUE="P1">P1<OPTION VALUE="P2">P2<OPTION VALUE="P3">P3<OPTION VALUE="P4">P4<OPTION VALUE="P5">P5</SELECT></td><td align=left valign=top><SELECT NAME="bug severity" MULTIPLE SIZE=7><OPTION VALUE="blocker">blocker<OPTION VALUE="critical">critical<OPTION VALUE="major">major<OPTION VALUE="normal">normal<OPTION VALUE="minor">minor<OPTION VALUE="trivial">trivial<OPTION VALUE="enhancement">enhancement</SELECT></tr></table> File Print Segmentation Fault

  3. into this <SELECT> File Print Segmentation Fault

  4. Simplification • Once one has reproduced a problem, one must find out what’s relevant: • Does the problem really depend on 10,000 lines of input? • Does the failure really require this exact schedule? • Do we need this sequence of calls?

  5. Broken computer • How do you identify the problem?

  6. Why Simplify? • Ease of communication. A simplified test case is easier to communicate. • Easier debugging. Smaller test cases result in smaller states and shorter executions. • Identify duplicates. Simplified test cases subsume several duplicates.

  7. Motivation • The Mozilla open-source web browser project receives several dozens bug reports a day. • Each bug report has to be simplified • Eliminate all details irrelevant to producing the failure • To facilitate debugging • To make sure it does not replicate a similar bug report • In July 1999, Bugzilla listed more than 370 open bug reports for Mozilla. • These were not even simplified • Mozilla engineers were overwhelmed with work • They created the Mozilla BugAThon: a call for volunteers to process bug reports

  8. Your Solution • How do you solve these problems? • Binary search • Cut the test case in half • Iterate • Brilliant idea: Why not automate this?

  9. Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input.

  10. Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input. ✘

  11. Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input.

  12. Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input. ✘

  13. Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input.

  14. Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input. ✘

  15. Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input.

  16. Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input. ✔

  17. Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input.

  18. Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input. ✘

  19. Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input. ✔

  20. Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input. ✔

  21. Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input. Simplified input

  22. Complex Input <td align=left valign=top><SELECT NAME="op sys" MULTIPLE SIZE=7><OPTION VALUE="All">All<OPTION VALUE="Windows 3.1">Windows 3.1<OPTION VALUE="Windows 95">Windows 95<OPTION VALUE="Windows 98">Windows 98<OPTION VALUE="Windows ME">Windows ME<OPTION VALUE="Windows 2000">Windows 2000<OPTION VALUE="Windows NT">Windows NT<OPTION VALUE="Mac System 7">Mac System 7<OPTION VALUE="Mac System 7.5">Mac System 7.5<OPTION VALUE="Mac System 7.6.1">Mac System 7.6.1<OPTION VALUE="Mac System 8.0">Mac System 8.0<OPTION VALUE="Mac System 8.5">Mac System 8.5<OPTION VALUE="Mac System 8.6">Mac System 8.6<OPTION VALUE="Mac System 9.x">Mac System 9.x<OPTION VALUE="MacOS X">MacOS X<OPTION VALUE="Linux">Linux<OPTION VALUE="BSDI">BSDI<OPTION VALUE="FreeBSD">FreeBSD<OPTION VALUE="NetBSD">NetBSD<OPTION VALUE="OpenBSD">OpenBSD<OPTION VALUE="AIX">AIX<OPTION VALUE="BeOS">BeOS<OPTION VALUE="HP-UX">HP-UX<OPTION VALUE="IRIX">IRIX<OPTION VALUE="Neutrino">Neutrino<OPTION VALUE="OpenVMS">OpenVMS<OPTION VALUE="OS/2">OS/2<OPTION VALUE="OSF/1">OSF/1<OPTION VALUE="Solaris">Solaris<OPTION VALUE="SunOS">SunOS<OPTION VALUE="other">other</SELECT></td><td align=left valign=top><SELECT NAME="priority" MULTIPLE SIZE=7> <OPTION VALUE="--">--<OPTION VALUE="P1">P1<OPTION VALUE="P2">P2<OPTION VALUE="P3">P3<OPTION VALUE="P4">P4<OPTION VALUE="P5">P5</SELECT></td><td align=left valign=top><SELECT NAME="bug severity" MULTIPLE SIZE=7><OPTION VALUE="blocker">blocker<OPTION VALUE="critical">critical<OPTION VALUE="major">major<OPTION VALUE="normal">normal<OPTION VALUE="minor">minor<OPTION VALUE="trivial">trivial<OPTION VALUE="enhancement">enhancement</SELECT></tr></table> File Print Segmentation Fault

  23. Simplified Input • <SELECT NAME="priority" MULTIPLE SIZE=7> • Simplified from 896 lines to one single line • Required 12 tests only

  24. Binary Search • <SELECT NAME="priority" MULTIPLE SIZE=7>

  25. Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7>

  26. Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> ✔

  27. Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> ✔ ✔

  28. Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> What do we do if both halves pass? ✔ ✔

  29. Change Granularity

  30. General DD Algorithm Basic idea: • Start with few & large changes first • If all alternatives pass or are unresolved, apply more & smaller changes. ∆1 ∆2 ∆1 ∆2 ∆1 ∆2 ∆3 ∆4 ∆5 ∆6 ∆7 ∆8 ∆1 ∆2 ∆3 ∆4 ∆5 ∆6 ∆7 ∆8

  31. Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> ✔ ✔

  32. Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> ✔ ✔ ✔

  33. Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> ✔ ✔ ✔ ✘

  34. Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> ✔ ✔ ✔ ✘ ✘

  35. Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> ✔ ✔ ✔ ✘ ✘ ✔

  36. Inputs and Failures • Let E be the set of possible inputs • rP E corresponds to an input that passes • rF E corresponds to an input that fails

  37. Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> Example of rF and rP ? ✔ ✔ ✔ ✘ ✘ ✔

  38. Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> Example of rF and rP ? rP = <SELECT NALE SIZE=7> rF = <SELECT NAME="priori ✔ ✔ ✔ ✘ ✘ ✔

  39. Changes • We can go from one input to another by changes • A change  is a mapping  : E  E which takes one input and changes it to another input r1 = <SELECT NAty" MULTIPLE SIZE=7> ’ = insert ME="priori at appropriate position What is ’(r1)?

  40. Changes • We can go from one input to another by changes • A change  is a mapping  : E  E which takes one input and changes it to another input r1 = <SELECT NAty" MULTIPLE SIZE=7> ’ = insert ME="priori at appropriate position What is ’(r1)? <SELECT NAME="priority" MULTIPLE SIZE=7>

  41. Decomposing Changes • A change  can be decomposed to a number of elementary changes 1, 2, ..., n where =1o 2o ... o n • where (io j)(r) = i(j(r)) • For example, deleting a part of the input file can be decomposed to deleting characters one be one from the input file • another way to say it: by composing deleting of single characters we can get a change that deletes part of the input file • ’ = insert ME="priori • ’ = 1 o 2 o 3 o 4 o 5 o 6 o 7 o 8 o 9 o 10 • 1 = insert M • 2 = insert E

  42. Summary • We have an input with failure: rF • We have an input without failure: rP • We have a set of changes cF = {1, 2, ..., n } such that rF = (1o 2o ... o n )(rP) where rF is a run with failure • Each subset c of cF is a test case

  43. Testing Test Cases • Given a test case c, we would like to know if the input generated by changing rP by the changes in c is an input that causes a failure • We define a function test: Powerset(cF)  {P, F, ?} such that, given c={1, 2, ..., m} cF test(c) = F iff (1o 2o ... o m )(rP) is a failing input

  44. Minimizing Test Cases • Now the question is: Can we find the minimal test case c such that test(c) = F? • A test case c  cF is called the global minimum of cF if for all c’  cF , |c’| < |c|  test(c’)  F • Global minimum is the smallest set of changes which will make the program fail

  45. Minimizing Test Cases • Now the question is: Can we find the minimal test case c such that test(c) = F? • A test case c  cF is called the global minimum of cF if for all c’  cF , |c’| < |c|  test(c’)  F • Global minimum is the smallest set of changes which will make the program fail • Finding the global minimum may require us to perform exponential number of tests

  46. Search for 1-minimal Input • Different problem formulation Find a set of changes that cause the failure, but removing any change causes the failure to go away • This is 1-minimality

  47. Minimizing Test Cases • A test case c  cF is called a local minimum of cF if for all c’  c , test(c’)  F • A test case c  cF is n-minimal if for all c’  c , |c|  |c’|  n  test(c’)  F • A test case is 1-minimal if for all i  c , test(c – {i})  F

  48. Naïve Algorithm • To find a 1-minimal subset of C, simply • Remove one element from C • If c – {} =X, recurse with smaller set • If C – {} ¹X, C is 1-minimal

  49. Analysis • In the worst case, • We remove one element from the set per iteration • After trying every other element • Work is potentially N + (N-1) + (N-2) + … • This is O(N2)

  50. Work Smarter, Not Harder • We can often do better • Silly to start out removing 1 element at a time • Try dividing change set in 2 initially • Increase # of subsets if we can’t make progress • If we get lucky, search will converge quickly

More Related