640 likes | 781 Views
Delta Debugging. Koushik Sen EECS, UC Berkeley. How do we go from this.
E N D
Delta Debugging Koushik Sen EECS, UC Berkeley
How do we go from this <td align=left valign=top><SELECT NAME="op sys" MULTIPLE SIZE=7><OPTION VALUE="All">All<OPTION VALUE="Windows 3.1">Windows 3.1<OPTION VALUE="Windows 95">Windows 95<OPTION VALUE="Windows 98">Windows 98<OPTION VALUE="Windows ME">Windows ME<OPTION VALUE="Windows 2000">Windows 2000<OPTION VALUE="Windows NT">Windows NT<OPTION VALUE="Mac System 7">Mac System 7<OPTION VALUE="Mac System 7.5">Mac System 7.5<OPTION VALUE="Mac System 7.6.1">Mac System 7.6.1<OPTION VALUE="Mac System 8.0">Mac System 8.0<OPTION VALUE="Mac System 8.5">Mac System 8.5<OPTION VALUE="Mac System 8.6">Mac System 8.6<OPTION VALUE="Mac System 9.x">Mac System 9.x<OPTION VALUE="MacOS X">MacOS X<OPTION VALUE="Linux">Linux<OPTION VALUE="BSDI">BSDI<OPTION VALUE="FreeBSD">FreeBSD<OPTION VALUE="NetBSD">NetBSD<OPTION VALUE="OpenBSD">OpenBSD<OPTION VALUE="AIX">AIX<OPTION VALUE="BeOS">BeOS<OPTION VALUE="HP-UX">HP-UX<OPTION VALUE="IRIX">IRIX<OPTION VALUE="Neutrino">Neutrino<OPTION VALUE="OpenVMS">OpenVMS<OPTION VALUE="OS/2">OS/2<OPTION VALUE="OSF/1">OSF/1<OPTION VALUE="Solaris">Solaris<OPTION VALUE="SunOS">SunOS<OPTION VALUE="other">other</SELECT></td><td align=left valign=top><SELECT NAME="priority" MULTIPLE SIZE=7> <OPTION VALUE="--">--<OPTION VALUE="P1">P1<OPTION VALUE="P2">P2<OPTION VALUE="P3">P3<OPTION VALUE="P4">P4<OPTION VALUE="P5">P5</SELECT></td><td align=left valign=top><SELECT NAME="bug severity" MULTIPLE SIZE=7><OPTION VALUE="blocker">blocker<OPTION VALUE="critical">critical<OPTION VALUE="major">major<OPTION VALUE="normal">normal<OPTION VALUE="minor">minor<OPTION VALUE="trivial">trivial<OPTION VALUE="enhancement">enhancement</SELECT></tr></table> File Print Segmentation Fault
into this <SELECT> File Print Segmentation Fault
Simplification • Once one has reproduced a problem, one must find out what’s relevant: • Does the problem really depend on 10,000 lines of input? • Does the failure really require this exact schedule? • Do we need this sequence of calls?
Broken computer • How do you identify the problem?
Why Simplify? • Ease of communication. A simplified test case is easier to communicate. • Easier debugging. Smaller test cases result in smaller states and shorter executions. • Identify duplicates. Simplified test cases subsume several duplicates.
Motivation • The Mozilla open-source web browser project receives several dozens bug reports a day. • Each bug report has to be simplified • Eliminate all details irrelevant to producing the failure • To facilitate debugging • To make sure it does not replicate a similar bug report • In July 1999, Bugzilla listed more than 370 open bug reports for Mozilla. • These were not even simplified • Mozilla engineers were overwhelmed with work • They created the Mozilla BugAThon: a call for volunteers to process bug reports
Your Solution • How do you solve these problems? • Binary search • Cut the test case in half • Iterate • Brilliant idea: Why not automate this?
Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input.
Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input. ✘
Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input.
Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input. ✘
Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input.
Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input. ✘
Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input.
Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input. ✔
Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input.
Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input. ✘
Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input. ✔
Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input. ✔
Binary Search • Proceed by binary search. Throw away half the input and see if the output is still wrong. • If not, go back to the previous state and discard the other half of the input. Simplified input
Complex Input <td align=left valign=top><SELECT NAME="op sys" MULTIPLE SIZE=7><OPTION VALUE="All">All<OPTION VALUE="Windows 3.1">Windows 3.1<OPTION VALUE="Windows 95">Windows 95<OPTION VALUE="Windows 98">Windows 98<OPTION VALUE="Windows ME">Windows ME<OPTION VALUE="Windows 2000">Windows 2000<OPTION VALUE="Windows NT">Windows NT<OPTION VALUE="Mac System 7">Mac System 7<OPTION VALUE="Mac System 7.5">Mac System 7.5<OPTION VALUE="Mac System 7.6.1">Mac System 7.6.1<OPTION VALUE="Mac System 8.0">Mac System 8.0<OPTION VALUE="Mac System 8.5">Mac System 8.5<OPTION VALUE="Mac System 8.6">Mac System 8.6<OPTION VALUE="Mac System 9.x">Mac System 9.x<OPTION VALUE="MacOS X">MacOS X<OPTION VALUE="Linux">Linux<OPTION VALUE="BSDI">BSDI<OPTION VALUE="FreeBSD">FreeBSD<OPTION VALUE="NetBSD">NetBSD<OPTION VALUE="OpenBSD">OpenBSD<OPTION VALUE="AIX">AIX<OPTION VALUE="BeOS">BeOS<OPTION VALUE="HP-UX">HP-UX<OPTION VALUE="IRIX">IRIX<OPTION VALUE="Neutrino">Neutrino<OPTION VALUE="OpenVMS">OpenVMS<OPTION VALUE="OS/2">OS/2<OPTION VALUE="OSF/1">OSF/1<OPTION VALUE="Solaris">Solaris<OPTION VALUE="SunOS">SunOS<OPTION VALUE="other">other</SELECT></td><td align=left valign=top><SELECT NAME="priority" MULTIPLE SIZE=7> <OPTION VALUE="--">--<OPTION VALUE="P1">P1<OPTION VALUE="P2">P2<OPTION VALUE="P3">P3<OPTION VALUE="P4">P4<OPTION VALUE="P5">P5</SELECT></td><td align=left valign=top><SELECT NAME="bug severity" MULTIPLE SIZE=7><OPTION VALUE="blocker">blocker<OPTION VALUE="critical">critical<OPTION VALUE="major">major<OPTION VALUE="normal">normal<OPTION VALUE="minor">minor<OPTION VALUE="trivial">trivial<OPTION VALUE="enhancement">enhancement</SELECT></tr></table> File Print Segmentation Fault
Simplified Input • <SELECT NAME="priority" MULTIPLE SIZE=7> • Simplified from 896 lines to one single line • Required 12 tests only
Binary Search • <SELECT NAME="priority" MULTIPLE SIZE=7>
Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7>
Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> ✔
Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> ✔ ✔
Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> What do we do if both halves pass? ✔ ✔
General DD Algorithm Basic idea: • Start with few & large changes first • If all alternatives pass or are unresolved, apply more & smaller changes. ∆1 ∆2 ∆1 ∆2 ∆1 ∆2 ∆3 ∆4 ∆5 ∆6 ∆7 ∆8 ∆1 ∆2 ∆3 ∆4 ∆5 ∆6 ∆7 ∆8
Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> ✔ ✔
Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> ✔ ✔ ✔
Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> ✔ ✔ ✔ ✘
Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> ✔ ✔ ✔ ✘ ✘
Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> ✔ ✔ ✔ ✘ ✘ ✔
Inputs and Failures • Let E be the set of possible inputs • rP E corresponds to an input that passes • rF E corresponds to an input that fails
Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> Example of rF and rP ? ✔ ✔ ✔ ✘ ✘ ✔
Binary Search ✘ • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> • <SELECT NAME="priority" MULTIPLE SIZE=7> Example of rF and rP ? rP = <SELECT NALE SIZE=7> rF = <SELECT NAME="priori ✔ ✔ ✔ ✘ ✘ ✔
Changes • We can go from one input to another by changes • A change is a mapping : E E which takes one input and changes it to another input r1 = <SELECT NAty" MULTIPLE SIZE=7> ’ = insert ME="priori at appropriate position What is ’(r1)?
Changes • We can go from one input to another by changes • A change is a mapping : E E which takes one input and changes it to another input r1 = <SELECT NAty" MULTIPLE SIZE=7> ’ = insert ME="priori at appropriate position What is ’(r1)? <SELECT NAME="priority" MULTIPLE SIZE=7>
Decomposing Changes • A change can be decomposed to a number of elementary changes 1, 2, ..., n where =1o 2o ... o n • where (io j)(r) = i(j(r)) • For example, deleting a part of the input file can be decomposed to deleting characters one be one from the input file • another way to say it: by composing deleting of single characters we can get a change that deletes part of the input file • ’ = insert ME="priori • ’ = 1 o 2 o 3 o 4 o 5 o 6 o 7 o 8 o 9 o 10 • 1 = insert M • 2 = insert E
Summary • We have an input with failure: rF • We have an input without failure: rP • We have a set of changes cF = {1, 2, ..., n } such that rF = (1o 2o ... o n )(rP) where rF is a run with failure • Each subset c of cF is a test case
Testing Test Cases • Given a test case c, we would like to know if the input generated by changing rP by the changes in c is an input that causes a failure • We define a function test: Powerset(cF) {P, F, ?} such that, given c={1, 2, ..., m} cF test(c) = F iff (1o 2o ... o m )(rP) is a failing input
Minimizing Test Cases • Now the question is: Can we find the minimal test case c such that test(c) = F? • A test case c cF is called the global minimum of cF if for all c’ cF , |c’| < |c| test(c’) F • Global minimum is the smallest set of changes which will make the program fail
Minimizing Test Cases • Now the question is: Can we find the minimal test case c such that test(c) = F? • A test case c cF is called the global minimum of cF if for all c’ cF , |c’| < |c| test(c’) F • Global minimum is the smallest set of changes which will make the program fail • Finding the global minimum may require us to perform exponential number of tests
Search for 1-minimal Input • Different problem formulation Find a set of changes that cause the failure, but removing any change causes the failure to go away • This is 1-minimality
Minimizing Test Cases • A test case c cF is called a local minimum of cF if for all c’ c , test(c’) F • A test case c cF is n-minimal if for all c’ c , |c| |c’| n test(c’) F • A test case is 1-minimal if for all i c , test(c – {i}) F
Naïve Algorithm • To find a 1-minimal subset of C, simply • Remove one element from C • If c – {} =X, recurse with smaller set • If C – {} ¹X, C is 1-minimal
Analysis • In the worst case, • We remove one element from the set per iteration • After trying every other element • Work is potentially N + (N-1) + (N-2) + … • This is O(N2)
Work Smarter, Not Harder • We can often do better • Silly to start out removing 1 element at a time • Try dividing change set in 2 initially • Increase # of subsets if we can’t make progress • If we get lucky, search will converge quickly