1 / 32

AFID: An Automated Fault Identification Tool

AFID aims to automate the collection of real-world software fault data and minimize developer involvement. The tool records fault revealing test cases and source code changes, offering a more efficient approach to software bug research.

charlesboyd
Download Presentation

AFID: An Automated Fault Identification Tool

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AFID: An Automated Fault Identification Tool Alex Edwards Sean Tucker Sébastien Worms Rahul Vaidya Brian Demsky

  2. Motivation • Much research focuses on software bugs • Relatively little emphasis on empirical methods as compared to other fields • Remarkably few software fault data sets are publically available that • Are uniformly structured • Contain faulty source code • Contain fault correction • Contain fault revealing test case • Lack of data sets affects how we approach research

  3. Effects • Guide research based on • General impressions of important bug classes • Are these the important bug classes? • What are we missing? • Often evaluate our research on • Hand selected bugs • Synthetic bugs • Difficult to study dynamic properties of software bugs

  4. Manual Collection • Colleagues tried to get students to manually record their software faults • Asked them to record: • Test case that revealed fault • Copy of the source code with the fault • Copy of the change that removed the fault • Limited success • Tedious • Often forgot

  5. Goal • Automatically record repositories of real software bugs • Minimize developer involvement • Okay to miss software faults

  6. Basic Approach • Obtain fault revealing test cases • Monitor source code for changes • Use fault revealing test cases to detect fault corrections • When correction is detected, record • Fault revealing test case • Faulty version • Fault correcting source code change

  7. Obtaining Fault Revealing Test Cases • Wait for developer to execute program • Record information about program interactions with the operating system • If it crashes, build test case from the recorded information • Record • Command line • Files accessed • Console interactions

  8. Recording Interactions Application System Calls Operating System

  9. Recording Interactions Application Monitor Ptrace System Calls Operating System

  10. Ptrace Monitoring • Open call for read access • Record file name (and copy later if the program crashes) • Open call for write access • Record file name and copy file • Console input • Record user input • Console output • Record program prompting

  11. Extraneous files • Programs read many files that would not be considered input • Java programs read libraries, class files, JVM components • C programs read shared libraries • Such files • are not interesting for the test case • make the test case huge • make the test case less portable

  12. Remove Extraneous Files • Can filter files • Create a test program that does nothing • Record what files are read • Exclude those files • Use patterns to guess other files to exclude • Exclude class files • Exclude library directories • User can use regular expressions to exclude other files (or include files)

  13. Duplicate Test Cases • Developers often rerun test cases • Results in multiple copies of same test case • Use hashing to avoid making multiple copies of test cases • Optimize for performance, ignore possibility of hash collisions

  14. Console Input • Problem: Want to support user interactions • Challenge: Would like to reuse test case in the presence of small modification • Approach: • Record transcript of user interactions • Compute for each user response, the shortest suffix of the output that uniquely identifies when the input occurred • Generate transcript using these suffixes and the user inputs • Provides flexibility to some prompt changes

  15. Monitoring Source Code Changes • Want to detect changes in source files • Need to know which files comprise an application • Goals: • Want to avoid input from developer • Should work with any tool chain

  16. Approach • Use same ptrace-based monitoring infrastructure on the compiler • Detect files when the compiler reads them • Use wildcards to identify source files

  17. Monitoring Source Changes • Build internal SVN repository • Add new files automatically as detected • Check in updates at every compile

  18. Detecting Fault Correcting Changes • Test cases can be used to detect which code changes correct which faults • For each code change, we rerun outstanding test cases to see if they still crash

  19. Replaying Test Cases • Could just copy test case files back to their original locations • Huge downsides • Developer may have written important data in new versions of these files • File system may have different directory structure • Execution could overwrite important data • Need to sandbox execution

  20. Sandboxing • Make a copy of the test case • Replay program in ptrace-based sandbox • Use ptrace to intercept file open commands • Use ptrace to replace open call’s file names with our copies • Intercept console I/O interactions to replay user interactions • Technical details in the paper

  21. Looping • Source code changes can cause formerly crashing test cases to loop • Solution: • Record elapsed time for every execution of application • Estimate upper bound on execution time • Terminate replays once they exceed this bound • Okay to be wrong - just miss recording a fault

  22. Central Repository • When fault correcting change is detected, AFID uploads information to repository server • Information contains: • Buggy source code • SVN repository • Fault correcting change • Fault revealing test case

  23. Overhead Measurements • Jasmin byte code assembler • 11,450 lines of code • I/O intensive benchmark • Inyo ray tracer • 5,843 lines of code • Longer running, compute bound benchmark • Measured on • 2.2 GHz Core 2 Duo, 1GB RAM, Debian 2.6.23 • HotSpot JVM version 1.5.0

  24. Overheads

  25. Case Study • Goal: To determine whether AFID effectively records real software faults • 8 participants • Each participant • Solved a programming contest problem • Used AFID while coding

  26. Fault Breakdown

  27. Fault Counts by Participant

  28. Lessons • Some participants debugged by commenting out code • Cause AFID to detect the wrong fault correcting change • Modified AFID to ask when it detects a fault correcting change • Source code changes can cause applications to loop instead of crash • Execution time estimator

  29. Participant feedback • Found the user experience very good • In general, tool was unnoticeable • Noticed slight delay when compiling

  30. Privacy Concerns • AFID records all source code changes and test inputs that crash the program • Could easily record personal information • Limit use of AFID to projects that are not likely to process personal information • Print out message to remind user that AFID is running

  31. Related Work • Mining CVS Repositories • Software-artifact Infrastructure Repository • iBUGS • Replay systems

  32. Conclusion • Next phase is data collection • Plan to make data available to other researchers • We need participants • Please go to http://demsky.eecs.uci.edu/afid/

More Related