Anomaly-Based Bug Prediction, Isolation, and Validation: An Automated Approach for Software Debugging

1. Anomaly-Based Bug Prediction, Isolation, and Validation: An Automated Approach for Software Debugging Martin Dimitrov Huiyang Zhou

2. Background: Terminology • Defect: Erroneous piece of code (a bug) • Infection: The defect is triggered => the program state differs from what the programmer intended. • Failure: An observable error (crash, hang, wrong results) in program behavior. Terminologies are based on the book “Why Programs Fail” by Andreas Zeller. University of Central Florida

3. Background: From Defects to Failures Erroneous code Variable and input values Observer sees failure Figure from the book “Why Programs Fail” by A. Zeller Sane state Infected state Program execution University of Central Florida

4. Motivation • The typical process of software debugging involves: • Examine the point of program failure and reason backwards about the possible causes. • Create a hypothesis of what could be the root cause. • Modify the program to verify the hypothesis. • If the failure is still there, the search resumes. • Software debugging is tedious and time consuming ! • In this work we propose an approach to automate the debugging effort and pinpoint the failure root cause. University of Central Florida

5. Presentation Outline • Motivation • Proposed approach • Detecting anomalies (step 1) • Isolating relevant anomalies (step 2) • Validating anomalies (step 3) • Experimental methodology • Experimental results • Conclusions University of Central Florida

6. Proposed Approach Dynamic Instruction Stream mov ... cmp ... jge ... mov ... mov ... lea ... movl ... inc ... cmp ... jl ... movl ... inc ... cmp ... Current instruction time University of Central Florida

7. Proposed Approach Dynamic Instruction Stream cmp ... jge ... mov ... mov ... lea ... movl ... inc ... cmp ... jl ... movl ... inc ... cmp ... Current instruction jr ... time University of Central Florida

8. Proposed Approach Dynamic Instruction Stream jge ... mov ... mov ... lea ... movl ... inc ... cmp ... jl ... movl ... inc ... cmp ... jr ... Current instruction movl ... time University of Central Florida

9. Proposed Approach Dynamic Instruction Stream mov ... mov ... lea ... movl ... inc ... cmp ... jl ... movl ... inc ... cmp ... jr ... movl ... Current instruction inc ... time University of Central Florida

10. Proposed Approach Dynamic Instruction Stream mov ... lea ... movl ... inc ... cmp ... jl ... movl ... inc ... cmp ... jr ... movl ... inc ... Current instruction cmp ... time University of Central Florida

11. Proposed Approach Dynamic Instruction Stream lea ... movl ... inc ... cmp ... jl ... movl ... inc ... cmp ... jr ... movl ... inc ... cmp ... Current instruction jl ... time University of Central Florida

12. Proposed Approach Dynamic Instruction Stream movl ... inc ... cmp ... jl ... movl ... inc ... cmp ... jr ... movl ... inc ... cmp ... jl ... Current instruction test ... time University of Central Florida

13. Proposed Approach Dynamic Instruction Stream inc ... cmp ... jl ... movl ... inc ... cmp ... jr ... movl ... inc ... cmp ... jl ... test ... Current instruction jne ... time University of Central Florida

14. Proposed Approach Dynamic Instruction Stream cmp ... jl ... movl ... inc ... cmp ... jr ... movl ... inc ... cmp ... jl ... test ... jne ... Current instruction mov ... time University of Central Florida

15. Proposed Approach Dynamic Instruction Stream jl ... movl ... inc ... cmp ... jr ... movl ... inc ... cmp ... jl ... test ... jne ... mov ... Current instruction call ... time University of Central Florida

16. Proposed Approach • A program failure is observed: • Crash • Hang • Incorrect results, etc. • Start the automated debugging process • The output of our approach is a ranked list of instructions (the possible root-cause of failure) Dynamic Instruction Stream movl ... inc ... cmp ... jr ... movl ... inc ... cmp ... jl ... test ... jne ... mov ... call ... Failure point mov ... Failure time University of Central Florida

17. Proposed ApproachStep 1: Detect anomalies in program execution Dynamic Instruction Stream anomaly anomaly anomaly mov ... cmp ... jge ... mov ... mov ... lea ... movl ... inc ... cmp ... jl ... movl ... inc ... cmp ... Current instruction time University of Central Florida

18. Proposed ApproachStep 1: Detect anomalies in program execution Dynamic Instruction Stream anomaly anomaly cmp ... jge ... mov ... mov ... lea ... movl ... inc ... cmp ... jl ... movl ... inc ... cmp ... Current instruction jr ... time University of Central Florida

19. Proposed ApproachStep 1: Detect anomalies in program execution Dynamic Instruction Stream anomaly anomaly jge ... mov ... mov ... lea ... movl ... inc ... cmp ... jl ... movl ... inc ... cmp ... jr ... Current instruction movl ... time University of Central Florida

20. Proposed ApproachStep 1: Detect anomalies in program execution Dynamic Instruction Stream anomaly anomaly anomaly mov ... mov ... lea ... movl ... inc ... cmp ... jl ... movl ... inc ... cmp ... jr ... movl ... Current instruction inc ... time University of Central Florida

21. Proposed ApproachStep 1: Detect anomalies in program execution Dynamic Instruction Stream anomaly anomaly anomaly anomaly mov ... lea ... movl ... inc ... cmp ... jl ... movl ... inc ... cmp ... jr ... movl ... inc ... Current instruction cmp ... time University of Central Florida

22. Proposed ApproachStep 1: Detect anomalies in program execution Dynamic Instruction Stream anomaly anomaly anomaly anomaly lea ... movl ... inc ... cmp ... jl ... movl ... inc ... cmp ... jr ... movl ... inc ... cmp ... Current instruction jl ... time University of Central Florida

23. Proposed ApproachStep 1: Detect anomalies in program execution Dynamic Instruction Stream anomaly anomaly anomaly anomaly movl ... inc ... cmp ... jl ... movl ... inc ... cmp ... jr ... movl ... inc ... cmp ... jl ... Current instruction test ... time University of Central Florida

24. Proposed ApproachStep 1: Detect anomalies in program execution Dynamic Instruction Stream anomaly anomaly anomaly inc ... cmp ... jl ... movl ... inc ... cmp ... jr ... movl ... inc ... cmp ... jl ... test ... Current instruction jne ... time University of Central Florida

25. Proposed ApproachStep 1: Detect anomalies in program execution Dynamic Instruction Stream anomaly anomaly anomaly cmp ... jl ... movl ... inc ... cmp ... jr ... movl ... inc ... cmp ... jl ... test ... jne ... Current instruction mov ... time University of Central Florida

26. Proposed ApproachStep 1: Detect anomalies in program execution Dynamic Instruction Stream anomaly anomaly anomaly anomaly jl ... movl ... inc ... cmp ... jr ... movl ... inc ... cmp ... jl ... test ... jne ... mov ... Current instruction call ... time University of Central Florida

27. Proposed ApproachStep 1: Detect anomalies in program execution • Each anomaly constitutes a hypothesis for the root cause of program failure. Dynamic Instruction Stream anomaly anomaly anomaly anomaly movl ... inc ... cmp ... jr ... movl ... inc ... cmp ... jl ... test ... jne ... mov ... call ... Failure point mov ... Failure time University of Central Florida

28. Proposed ApproachStep 2: Isolate the relevant anomalies Dynamic Instruction Stream anomaly anomaly anomaly anomaly movl ... • Create dynamic forward slices from the anomalies to the failure point. • Discard anomalies which do not lead to the failure point. inc ... cmp ... jr ... movl ... inc ... cmp ... jl ... test ... jne ... mov ... call ... Failure point mov ... Failure time University of Central Florida

29. Proposed Approach Step 3: Validate the isolated anomalies • Automatically “fix” the anomaly and observe if the program still fails. Dynamic Instruction Stream anomaly anomaly movl ... inc ... cmp ... jr ... movl ... inc ... cmp ... jl ... test ... jne ... mov ... call ... Failure point mov ... Failure time University of Central Florida

30. Proposed ApproachStep 3: Validate the isolated anomalies Dynamic Instruction Stream movl ... • If the failure disappears we have a high confidence the root cause have been pinpointed. inc ... cmp ... jr ... movl ... inc ... cmp ... jl ... test ... jne ... mov ... call ... No failure mov ... Success time University of Central Florida

31. Detecting Program Anomalies (Step 1) • When infected by a software bug the program is likely to misbehave: • Out-of-bounds addresses and values • Unusual control paths • Page faults • Redundant computations, etc. • Anomaly detection: Infer program specifications from passing runs and turn them into ‘soft’ assertions. • Learn program invariants during passing runs (e.g. variable “i” is always between 0 and 100) • Flag violated invariants during the failing run (e.g. Report anomaly if variable “i” is 101) University of Central Florida

32. Detecting Program Anomalies • We use several anomaly detectors to monitor a large spectrum of program invariants and catch more bugs. • DIDUCE [S. Hangal et al. , ICSE 2002] • Instructions tent to produce values/addresses within a certain range (e.g. 0 <= i <= 100). Detect violations of these invariants. • AccMon [P. Zhou et al. , MICRO-37 2004] • Only a few static instructions access a given memory location (load/store set locality). Signal an anomaly if memory access does not belong to the load/store set. • LoopCount • Detect abnormal number of loop iterations. University of Central Florida

33. Detecting Program Anomalies void more_arrays () { . . . a_count += STORE_INCR; /* Copy the old arrays. */ for (indx = 1; indx < old_count; indx++) arrays[indx] = old_ary[indx]; /* Initialize the new elements. */ for (; indx < v_count; indx++){/* defect*/ arrays[indx] = NULL; /*infection: */ /* Free the old elements. */ if (old_count != 0){ free (old_ary); /*crash*/ } } Heap Memory data data data data data size a_count  size data data v_count  data data University of Central Florida

34. Detecting Program Anomalies void more_arrays () { . . . a_count += STORE_INCR; /* Copy the old arrays. */ for (indx = 1; indx < old_count; indx++) arrays[indx] = old_ary[indx]; /* Initialize the new elements. */ for (; indx < v_count; indx++){/* defect*/ arrays[indx] = NULL; /*infection: */ /* Free the old elements. */ if (old_count != 0){ free (old_ary); /*crash*/ } } LoopCount: Loop iterates more times than usual. LoopCount: Loop iterates more times than usual. (false – positive) AccMon: store instruction is not in store set of this memory location. AccMon: store instruction is not in store set of this memory location. (false – positive) DIDUCE: Address of store instruction is out of normal range. DIDUCE: Address of store instruction is out of normal range. (false – positive) University of Central Florida

35. Detecting Program Anomalies: Architectural Support • DIDUCE and AccMon capture invariants using limited size caches structures, as proposed in previous work • LoopCount utilizes the existing loop-branch predictor to detect anomalies. • Advantages and disadvantages of hardware support: • Performance efficiency • Portability • Efficient ways to change or invalidate dynamic instructions • Limited hardware resource may become a concern University of Central Florida

36. Isolating Relevant Anomalies (Step 2) • Anomaly detectors alone are NOT effective for debugging: • May signal too many anomalies / false positives • Tradeoff between bug coverage and number of false positives • Our solution: • Allow aggressive anomaly detection for maximum bug coverage • Automatically isolate only the relevant anomalies by constructing dynamic forwards slices from the anomaly to the failure point University of Central Florida

37. Isolating Relevant Anomalies: Architectural Support • Add token(s) to each register and memory word. • Detected anomalies set a token associated with the destination memory word or register. • Tokens propagate based data dependencies. • When the program fails, examine the point of failure for token. University of Central Florida

38. Isolating Relevant Anomalies: Architectural Support void more_arrays () { . . . /* Copy the old arrays. */ for (indx = 1; indx < old_count; indx++) arrays[indx] = old_ary[indx]; /* Initialize the new elements. */ for (; indx < v_count; indx++){/* defect */ arrays[indx] = NULL; /*infection*/ /* Free the old elements. */ if (old_count != 0){ free (old_ary); /*crash */ } } Token Memory . . . Failure mov %ebx,0xc(%edx) University of Central Florida

39. Isolating Relevant Anomalies: Architectural Support • Problem: Many tokens for each memory location/ register • Solution: • We leverage tagged architectures for information flow tracking. • Use only one token (1 bit) (i.e., shared by all anomalies ) • We leverage delta debugging [A. Zeller, FSE 1999] to isolate the relevant anomalies automatically. Number of Initial Anomalies Number of Isolated Anomalies University of Central Florida

40. Delta-Debugging University of Central Florida

41. Validating Isolated Anomalies (Step 3) • Validate the remaining anomalies by applying a “fix” and observing if the program failure disappears. • Our “fix” is to nullify the anomalous instruction (turn it into no-op) • If the program succeeds, we have a high confidence we have found the root cause (or at least broken the infection chain) University of Central Florida

42. Validating Isolated Anomalies void more_arrays () { . . . /* Initialize the new elements. */ for (; indx < v_count; indx++){/* defect */ arrays[indx] = NULL; /*infection*/ /* Free the old elements. */ if (old_count != 0){ free (old_ary); /*crash */ } } Token Memory data data 0x0 0x0 0x0 data data data 0x0 0x0 0x0 size size data Success data data data • The “size” information is not corrupted and the program terminates successfully. University of Central Florida

43. Validating Isolated Anomalies • Four possible outcomes of our validation step: > ... (hang) > Segmentation Fault > Hello World ! > Hello Martin  Failure Success Unknown No crash Program fails the same way as before. Program produces expected output. Program fails in an unexpected manner. Program does not crash, but produces unexpected results. • Rank isolated anomalies based on the outcome: • succeed (highest) , no crash, unknown, failure (lowest) • In our running example the root-cause is ranked #1. University of Central Florida

44. Experimental Methodology • Implemented a working debugging tool using binary instrumentation (PIN). • Evaluated applications from BugBench [S. Lu et al., Bugs 2005] and gcc compiler. University of Central Florida

45. Experimental Results University of Central Florida

46. Case Study: GCC University of Central Florida

47. GCC Defect University of Central Florida

48. GCC Fix University of Central Florida

49. Experimental ResultsCompared to Failure-Inducing Chops University of Central Florida

50. Limitations • No failure, no bug detection • Un-triggered bugs • Bugs are triggered but output is correct • Target at bugs in sequential programs University of Central Florida