1 / 72

Fault Location via State Alteration

Fault Location via State Alteration. CS 206 Fall 2009. Value Replacement: Overview. Aggressive state alteration to locate faulty program statements [Jeffrey et. al., ISSTA 2008]. INPUT:. Faulty program and test suite (1+ failing runs). TASK:.

dahlia
Download Presentation

Fault Location via State Alteration

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Fault Location via State Alteration CS 206 Fall 2009

  2. Value Replacement: Overview • Aggressive state alteration to locate faulty program statements [Jeffrey et. al., ISSTA 2008] INPUT: Faulty program and test suite (1+ failing runs) TASK: (1) Perform value replacements in failing runs (2) Rank program statements according to collected information OUTPUT: Ranked list of program statements

  3. ERROR ERROR REPLACE VALUES Correct? / Incorrect? Incorrect Output Correct Output Alter State by Replacing Values Passing Execution Failing Execution Failing Execution: Altered State

  4. 1 1 1 0 1 1 1 1 1 (F) 2 3 0 5 Example of a Value Replacement PASSING EXECUTION: (output: ?) 1: read (x, y); 2: a := x - y; 3: if (x < y) 4: write (a); else 5: write (a + 1);

  5. 1 1 3 1 1 1 (F) 1 2 1 2 ERROR: plus should be minus ERROR ERROR 3 2 5 Example of a Value Replacement FAILING EXECUTION: (expected output: 1) (actual output: ?) 1: read (x, y); 2: a := x + y; 3: if (x < y) 4: write (a); else 5: write (a + 1);

  6. 1 1 0 1 1 1 1 Original Values Alternate Values 1 2 1 2 ERROR ERROR (T) 0 1 3 REPLACE VALUES 1 4 Interesting Value Mapping Pair (IVMP): Location: statement 2, instance 1 Original: {a = 2, x = 1, y = 1} Alternate: {a = 1, x = 0, y = 1} Example of a Value Replacement STATE ALTERATION: (expected output: 1) (actual output: ?) 1: read (x, y); 2: a := x + y; 3: if (x < y) 4: write (a); else 5: write (a + 1);

  7. Searching for IVMPs in a Failing Run • Step 1: Compute the Value Profile • Set of values used at each statement with respect to all available test case executions • Step 2: Replace values to search for IVMPs • For each statement instance in failing run • For each alternate set of values in value profile • Replace values to see if an IVMP is found • Endfor • Endfor

  8. 1: read (x, y); (0,0) (1,1) (-1,0) VALUE REPLACEMENT RESULTING OUTPUT IVMP? x = 1 y = 1 x = -1 y = 0 x = 0 y = 0 1: \ -1 \ 0 x = 1 y = 1 -1 x = -1 y = 0 a = -1 x = 0 y = 0 a = 0 x = 1 y = 1 a = 2 2: x = 1 y = 1 branch = F x = -1 y = 0 branch = T x = 0 y = 0 branch = F 3: a = -1 output = -1 4: a = 2 output = 3 a = 0 output = 1 5: Searching for IVMPs: Example 1: read (x, y); 2: a := x + y; // + should be – 3: if (x < y) 4: write (a); else 5: write (a + 1); VALUE PROFILE

  9. VALUE PROFILE VALUE REPLACEMENT RESULTING OUTPUT IVMP? x = 1 y = 1 x = -1 y = 0 x = 0 y = 0 1: \ 0 \ 0 x = 1 y = 1 1 x = -1 y = 0 a = -1 x = 0 y = 0 a = 0 x = 1 y = 1 a = 2 2: x = 1 y = 1 branch = F x = -1 y = 0 branch = T x = 0 y = 0 branch = F 3: a = -1 output = -1 4: a = 2 output = 3 a = 0 output = 1 5: Searching for IVMPs: Example 1: read (x, y); 1: read (x, y); 2: a := x + y; // + should be – 3: if (x < y) 4: write (a); else 5: write (a + 1); IVMPs Identified: stmt 1, inst 1:( {x=1, y=1}{x=0, y=0} )

  10. VALUE PROFILE \ -1 \ 0 \ -1 \ 0 \ 0 \ 0 x = 1 y = 1 a = 2 x = 1 y = 1 a = 2 x = 1 y = 1 x = -1 y = 0 x = 0 y = 0 1: x = -1 y = 0 a = -1 x = 0 y = 0 a = 0 x = 1 y = 1 a = 2 2: x = 1 y = 1 branch = F x = -1 y = 0 branch = T x = 0 y = 0 branch = F 3: a = -1 output = -1 4: a = 2 output = 3 a = 0 output = 1 5: Searching for IVMPs: Example 1: read (x, y); 2: a := x + y; // + should be – 3: if (x < y) 4: write (a); else 5: write (a + 1); 2: a := x + y; IVMPs Identified: stmt 1, inst 1:( {x=1, y=1}{x=0, y=0} ) stmt 2, inst 1:( {x=1, y=1, a=2}{x=0, y=0, a=0} )

  11. VALUE PROFILE \ -1 \ 0 \ T \ 0 \ 0 \ F x = 1 y = 1 branch = F x = 1 y = 1 branch = F x = 1 y = 1 x = -1 y = 0 x = 0 y = 0 1: x = -1 y = 0 a = -1 x = 0 y = 0 a = 0 x = 1 y = 1 a = 2 2: x = 1 y = 1 branch = F x = -1 y = 0 branch = T x = 0 y = 0 branch = F 3: a = -1 output = -1 4: a = 2 output = 3 a = 0 output = 1 5: Searching for IVMPs: Example 1: read (x, y); 2: a := x + y; // + should be – 3: if (x < y) 4: write (a); else 5: write (a + 1); 3: if (x < y) IVMPs Identified: stmt 1, inst 1:( {x=1, y=1}{x=0, y=0} ) stmt 2, inst 1:( {x=1, y=1, a=2}{x=0, y=0, a=0} )

  12. VALUE PROFILE \ 0 \ 1 a = 2 output = 3 x = 1 y = 1 x = -1 y = 0 x = 0 y = 0 1: x = -1 y = 0 a = -1 x = 0 y = 0 a = 0 x = 1 y = 1 a = 2 2: x = 1 y = 1 branch = F x = -1 y = 0 branch = T x = 0 y = 0 branch = F 3: a = -1 output = -1 4: a = 2 output = 3 a = 0 output = 1 5: Searching for IVMPs: Example 1: read (x, y); 2: a := x + y; // + should be – 3: if (x < y) 4: write (a); else 5: write (a + 1); 5: write (a + 1); IVMPs Identified: stmt 1, inst 1:( {x=1, y=1}{x=0, y=0} ) stmt 2, inst 1:( {x=1, y=1, a=2}{x=0, y=0, a=0} ) stmt 5, inst 1:( {a=2, output=3}{a=0, output=1} )

  13. VALUE PROFILE x = 1 y = 1 x = -1 y = 0 x = 0 y = 0 1: x = -1 y = 0 a = -1 x = 0 y = 0 a = 0 x = 1 y = 1 a = 2 2: x = 1 y = 1 branch = F x = -1 y = 0 branch = T x = 0 y = 0 branch = F IVMPs Identified: 3: stmt 1, inst 1:( {x=1, y=1}{x=0, y=0} ) a = -1 output = -1 4: stmt 2, inst 1:( {x=1, y=1, a=2}{x=0, y=0, a=0} ) stmt 5, inst 1:( {a=2, output=3}{a=0, output=1} ) a = 2 output = 3 a = 0 output = 1 5: Searching for IVMPs: Example 1: read (x, y); 2: a := x + y; // + should be – 3: if (x < y) 4: write (a); else 5: write (a + 1); DONE

  14. IVMPs at Non-Faulty Statements • Causes of IVMPs at non-faulty statements • Statements in same dependence chain • Coincidence • Consider multiple failing runs • Stmt w/ IVMPs in more runs  more likely to be faulty • Stmt w/ IVMPs in fewer runs  less likely to be faulty

  15. 1: read (x, y); 1: read (x, y); 2: a := x + y; 2: a := x + y; 4: write (a); 5: write (a + 1); Test Case [A] IVMPs: Test Case [B] IVMPs: stmt 1, inst 1:( {x=1, y=1}{x=0, y=1} ) stmt 1, inst 1:( {x=0, y=1}{x=-1, y=0} ) stmt 1, inst 1:( {x=1, y=1}{x=0, y=0} ) stmt 2, inst 1:( {x=0, y=1,a=1}{x=-1, y=0,a=-1} ) stmt 2, inst 1:( {x=1, y=1, a=2}{x=0, y=1, a=1} ) stmt 4, inst 1:( {a=1, output=1}{a=-1, output=-1} ) stmt 2, inst 1:( {x=1, y=1, a=2}{x=0, y=0, a=0} ) stmt 5, inst 1:( {a=2, output=3}{a=0, output=1} ) stmts with IVMPs: {1, 2, 4} stmts with IVMPs: {1, 2, 5} MOST LIKELY TO BE FAULTY {1, 2} {4, 5}{3} LEAST LIKELY TO BE FAULTY Multiple Failing Runs: Example 1: read (x, y); 2: a := x + y; 3: if (x < y) 4: write (a); else 5: write (a + 1); [A] [B] [C] [D]

  16. fraction of failing runs exercising stmt • fraction of passing runs exercising stmt • fraction of failing runs exercising stmt + Ranking Statements using IVMPs • Sort in decreasing order of: • Break ties using Tarantula technique [Jones et. al., ICSE 2002] • The number of failing runs in which the statement is associated with at least one IVMP

  17. Techniques Evaluated • Value Replacement technique • Consider all available failing runs (ValRep-All) • Consider only 2 failing runs (ValRep-2) • Consider only 1 failing run (ValRep-1) • Tarantula technique (Tarantula) • Consider all available test cases • Most effective technique known for our benchmarks • Only rank statements exercised by failing runs

  18. Low Suspiciousness Low Suspiciousness High Suspiciousness High Suspiciousness Higher Score Lower Score Metric for Comparison • Score for each ranked statement list • Represents percentage of statements that need not be examined before error is located • Higher score is better • rank of the faulty stmt size of list x 100% size of list

  19. Benchmark Programs • 129 faulty programs (errors) derived from 7 base programs • Each faulty program is associated with a branch-coverage adequate test suite containing at least 5 failing and 5 passing test cases • Test suite used by Value Replacement, test pool used by Tarantula

  20. Effectiveness Results Value Replacement technique

  21. Effectiveness Results Comparison to Tarantula

  22. Value Replacement: Summary • Highly Effective • Precisely locates 39 / 129 errors (30.2%) • Most effective previously known: 5 / 129 (3.9%) • Limitations • Can require significant computation time to search for IVMPs • Assumes multiple failing runs are caused by the same error

  23. Handling Multiple Errors • Effectively locate multiple simultaneous errors [Jeffrey et. al., ICSM 2009] • Iteratively compute a ranked list of statements to find and fix one error at a time • Three variations of this technique • MIN: minimal computation; use same list each time • FULL: full computation; produce new list each time • PARTIAL: partial computation; revise list each time

  24. Multiple Errors (MIN) Faulty Program and Test Suite Value Replacement Ranked List of Program Statements Developer Find/Fix Error Yes No Failing Run Remains? Done Multiple-Error Techniques Single Error Faulty Program and Test Suite Value Replacement Ranked List of Program Statements Developer Find/Fix Error Done

  25. Multiple Errors (FULL) Multiple Errors (PARTIAL) Faulty Program and Test Suite Faulty Program and Test Suite Value Replacement Partial Value Replacement Ranked List of Program Statements Ranked List of Program Statements Developer Find/Fix Error Developer Find/Fix Error Yes Yes No No Failing Run Remains? Failing Run Remains? Done Done Multiple-Error Techniques

  26. PARTIAL Technique • Step 1: Initialize ranked lists and locate first error • For each statement s, compute a ranked list by considering only failing runs exercising s • Report ranked list with highest suspiciousness value at the front of the list • Step 2: Iteratively revise ranked lists and locate each remaining error • For each remaining failing run that exercises the statement just fixed, recompute IVMPs • Update any affected ranked lists • Report ranked list with the most different elements at the front of the list, compared to previously-selected lists

  27. Computed Ranked Lists: (statementsuspiciousness) 23, 52, 11, 41, 30 1 [based on runs A, B, C] 23, 52, 11, 41, 30 2 [based on runs A, B, C] 22, 11, 51, 30, 40 3 [based on runs A, B] 21, 41, 51, 10, 30 4 [based on run C] 23, 52, 11, 41, 30 5 [based on runs A, B, C] PARTIAL Technique: Example Program (2 faulty statements) 1 2 3 4 5 Report list 1, 2, or 5 (assume 1)  Fix faulty statement 2

  28. Program (1 faulty statement) 1 Computed Ranked Lists: (statementsuspiciousness) 2 22, 11, 41, 51, 30 2 [based on runs A, B, C] (C updated) 22, 11, 51, 30, 40 3 4 3 [based on runs A, B] (no updates) 41, 10, 20, 30, 50 4 [based on run C] (C updated) 5 22, 11, 41, 51, 30 5 [based on runs A, B, C] (C updated) PARTIAL Technique: Example • Report list 4 • Fix faulty statement 4 • Done

  29. Techniques Compared • (MIN) Only compute ranked list once • (FULL) Fully recompute ranked list each time • (PARTIAL) Compute IVMPs for subset of failing runs and revise ranked lists each time • (ISOLATED) Locate each error in isolation

  30. Benchmark Programs Experimental Benchmark Programs Each faulty program contains 5 seeded errors, each in a different stmt Each faulty program is associated with a stmt-coverage adequate test suite such that at least one failing run exercises each error

  31. Effectiveness Results Effectiveness Comparison of Value Replacement Techniques 90 85 80 Isolated 75 Full Avg. Score per Ranked List (%) Partial 70 Min 65 60 55 50 tcas ptok ptok2 sched totinfo replace sched2

  32. Efficiency of Value Replacement • Searching for IVMPs is time-consuming • Lossy techniques • Reduce search space for finding IVMPs • May result in some missed IVMPs • Performed for single-error benchmarks • Lossless techniques • Only affect the efficiency of implementation • Result in no missed IVMPs • Performed for multi-error benchmarks 5 failing runs 15 alt value sets per instance 50,000 stmt instances per run X X 3.75 million value replacement program executions =  Over 10 days if each execution requires a quarter-second

  33. (skip) (skip) orig min < max < min > max > Lossy Techniques • Limit considered statement instances • Find IVMP  skip all subsequent instances of the same statement in the current run • Don’t find IVMP  skip statement in subsequent runs • Limit considered alternate value sets • only use min <, max <, min >, and max >, as compared to original value

  34. Regular Value Replacement Executions (x 6) (x 4) (x 2) • Efficiency Improvements: • Fork child process to do each value replacement in original failing execution • Perform value replacements in parallel Lossless Techniques Original Execution stmt instance 1 stmt instance 2 stmt instance 3 (assume 2 alternate value sets at each stmt instance) (value replacements are independent of each other) (portions of original execution are duplicated multiple times)

  35. With Parallelization (total time required to perform all value replacements is reduced) Lossless Techniques With Redundant Execution Removed (no duplication of any portion of original execution)

  36. Search Reduction by Lossy Techniques Reduction in # of Executions by Lossy Techniques: Single-Error Benchmarks

  37. Search Reduction by Lossy Techniques Reduction in # of Executions by Lossy Techniques: Single-Error Benchmarks On average, total number of executions reduced by a factor of 67

  38. Time Required for Reduced Search Time Required to Search using Lossy Techniques: Single-Error Benchmarks

  39. Time Required for Reduced Search Time Required to Search using Lossy Techniques: Single-Error Benchmarks Only 13% of faulty programs required more than 100 minutes of IVMP search time

  40. Time Required with Lossless Techniques Time to Search in Each Faulty Program using Lossless Techniques: Multi-Error Benchmarks 600 With Lossless techniques, multiple errors in a program can be located in minutes. With Lossy techniques, some single errors require hours to locate. 500 400 Full Avg. Time (seconds) Partial 300 Min 200 100 0 tcas ptok ptok2 sched totinfo replace sched2

  41. Execution Suppression • Efficient location of memory errors through targeted state alteration [Jeffrey et. al., ICSM 2008] • Alter state in a way that will definitely get closer to the goal each time • Goal: identify first point of memory corruption in a failing execution

  42. Memory Errors and Corruption • Memory errors • Buffer overflow • Uninitialized read • Dangling pointer • Double free • Memory leak • Memory corruption • Incorrect memory location is accessed, or • Incorrect value is assigned to a pointer variable

  43. Study of Memory Corruption Traversal of error  First point of memory corruption  Failure

  44. Observations from Study • Total distance from point of error traversal until failure can be large • Different inputs triggering memory corruption may result in different crashes or no crashes • Distance from error traversal to first memory corruption, is considerably lessthan distance from first memory corruption to failure

  45. Execution Suppression: High-Level • Program crash reveals memory corruption • Key: assume memory corruption leads to crash • Component 1: suppression • Iteratively identify first point of memory corruption • Omit the effect of certain statements during execution • Component 2: variable re-ordering • Expose crashes where they may not occur • Helpful since key assumption does not always hold

  46. First point of memory corruption Suppression: How it Works • While a crash occurs • Identify accessed location L directly causing crash • Identify last definition D of location L • Re-execute program and omit execution of D and anything dependent on it • Endwhile • Report the statement associated with the most recent D

  47. Stmt 4: copy-paste error: “x” should be “y” Stmt 8: clobbers def @ stmt 6 Stmts 9 - 11: propagation Stmt 12: potential buffer overflow Stmt 13: potential overflow or NULL dereference Stmt 15: double free Suppression: Example 1: int *p1 = &x[1]; 2: int *p2 = &x[0]; 3: int *q1 = &y[1]; 4: int *q2 = &x[0]; 5: *p1 = readInt(); 6: *p2 = readInt(); 7: *q1 = readInt(); 8: *q2 = readInt(); 9: int a = *p1 + *p2; 10: int b = *q1 + *q2; 11: int c = a + b + 1; 12: intArray[c] = 0; 13: structArray[*p2]->f = 0; 14: free(p2); 15: free(q2);

  48. Suppression: Example 1: int *p1 = &x[1]; 2: int *p2 = &x[0]; 3: int *q1 = &y[1]; 4: int *q2 = &x[0]; 5: *p1 = readInt(); 6: *p2 = readInt(); 7: *q1 = readInt(); 8: *q2 = readInt(); 9: int a = *p1 + *p2; 10: int b = *q1 + *q2; 11: int c = a + b + 1; 12: intArray[c] = 0; 13: structArray[*p2]->f = 0; 14: free(p2); 15: free(q2); Stmt 4: The error as well as the first point of memory corruption (Located in 4 executions)

  49. 5 *p1 1 p1 2 p2 3 q1 6 *p2 7 *q1 4 q2 8 *p2/*q2 9 a 10 b 11 c 12 CRASH Action: Suppress definition of c at stmt 11 and all of its effects Example: Execution 1 of 4 Loc Defined: Stmt: OK? 1: int *p1 = &x[1]; 2: int *p2 = &x[0]; 3: int *q1 = &y[1]; 4: int *q2 = &x[0]; 5: *p1 = readInt(); 6: *p2 = readInt(); 7: *q1 = readInt(); 8: *q2 = readInt(); 9: int a = *p1 + *p2; 10: int b = *q1 + *q2; 11: int c = a + b + 1; 12: intArray[c] = 0; 13: structArray[*p2]->f = 0; 14: free(p2); 15: free(q2);

  50. 6 *p2 1 p1 2 p2 3 q1 5 *p1 7 *q1 4 q2 8 *p2/*q2 9 a 10 b 13 CRASH Action: Suppress def of *p2/*q2 at stmt 8 and all of its effects Example: Execution 2 of 4 Loc Defined: Stmt: OK? 1: int *p1 = &x[1]; 2: int *p2 = &x[0]; 3: int *q1 = &y[1]; 4: int *q2 = &x[0]; 5: *p1 = readInt(); 6: *p2 = readInt(); 7: *q1 = readInt(); 8: *q2 = readInt(); 9: int a = *p1 + *p2; 10: int b = *q1 + *q2; 11: int c = a + b + 1; 12: intArray[c] = 0; 13: structArray[*p2]->f = 0; 14: free(p2); 15: free(q2); 11 12

More Related