1 / 21

Bug Driven Bug Finding

Bug Driven Bug Finding. Chadd Williams. Motivation. Finding bugs in software is important Statically checking code has been effective finds complex errors no need to run the code Many static checkers available some with specific bug patterns to find

aileen
Download Presentation

Bug Driven Bug Finding

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Bug Driven Bug Finding Chadd Williams

  2. Motivation • Finding bugs in software is important • Statically checking code has been effective • finds complex errors • no need to run the code • Many static checkers available • some with specific bug patterns to find • some allow the user to define the patterns • what kinds of bugs are really out there? • Lots of false positive error reports • can we rank the errors better? • can previous bug history help? • Where to start? • Bug reporting databases • CVS commit messages

  3. Bug Database • Inspect fixed bugs • review bug discussions • tie fixed bug to source code change • classify the type of the bug • look for bugs that can be found statically Users Developers Bug Database

  4. Bug Database: Practical Experience • We inspected the Apache httpd bug database • inspected 200 bug reports marked as fixed • not as helpful as we expected • Only 24% tied directly back to a source code change • bug reports include a discussion of the problem • rarely is a diff or a CVS revision noted • Most are logic errors/feature requests • not the type found by static checkers

  5. Bug Database Bug Types • Most classified bugs are logic errors

  6. Bug Database: Practical Experience • Most bug reports originate from users • 197 out of 200 • does not capture bugs found by developers • Most bug reports came against a release of the software, not a CVS-HEAD • 198 out of 200 • does not capture bugs between releases • What about the bugs that don’t make it into the release? • they may be in the CVS repository…

  7. CVS Repository • Commits may contain useful data • any bug fix must show up in a commit • will commit messages lead us to bug fixes? • Shows bugs fixed between releases • Bugs caught by developers • bugs that could be found by static checking 1.1 1.2 CVS Repository 1.3

  8. CVS Repository: Practical Experience • Inspected commit messages • looked for ‘fix’, ‘bug’ or ‘crash’ • ignored those with bug number listed • looked at mature source files • Commit messages are useful • trivially tied to source code change • less logic errors • Common errors found • NULL pointer check • failing to check the return value of a function before use

  9. CVS Repository Bug Types • NULL pointer bugs and return value bugs can be found by static analysis

  10. Return Value Check Bug • Returning error code and valid data from a function is a common C idiom • the return value should be checked before being used • lint checks for this error • Error types • completely ignored • foo(); • return value used directly as an argument • bar(foo()); • others … int foo(){ … if( error ){ return error_code; } …. return data; } … value = foo(); newPosition + = value; // ???

  11. Return Value Checker • Some functions don’t need their return value checked • no error value returned • could lead to many false positives • Naively flagging all unchecked return values leads to many false positives • over 7,000 errors reported for the Apache httpd-2.0 source • Need to determine which are most likely true errors • use historical data • present this data to the user

  12. Which return values need checked? • Infer from historical data • look for an add of a check of a return value in a CVS commit • implies the programmer thinks it’s important • Infer from current usage • does the return value of a function get checked in the current version of the software • how often? … value = foo(); if( value != Error) { // Check newPosition + = value; } … Commit Bug Fix … value = foo(); newPosition + = value; // ??? …

  13. Our Tool • Static checker that looks for return value check bugs • built on ROSE by Dan Quinlan, et al. • Classify each error by category • ignored return value • return value used as argument, etc. • Produce a ranking of the errors • group errors by called function • rank most promising errors higher • rank functions that most likely need their return value checked higher

  14. Return Value Checker: Ranking • Rank errors in two ways • Split functions into two groups • functions flagged with a CVS bug fix commit • at least one CVS commit adds a check of the function’s return value • functions not flagged with CVS bug fix commit • Within each group: • rank by how often the function’s return value is checked in the current software distribution • checked more often means rank higher

  15. Case Study • Apache httpd-2.0 on Linux • core system • modules • Apache Runtime Library • Checked all the CVS commits for a return value check bug fix • 6100 commits checked • 2600 commits failed to go through our tool • wrong (too new) version of autoconf • parser problems • compile bugs in the CVS commits

  16. Case Study: Results • Our checker marked over 7,000 errors • individual call site for non-void function where the return value is not checked • Too many too look at! • expect many are false positives • Rank errors • inspect CVS bug fix commit flagged functions • inspect functions with return value checked more than 50% of the time in the current source tree value = foo(); // ERROR newPosition + = value; … result = foo(); // ERROR zoo(result);

  17. Case Study: Error Breakdown • Inspected 453 errors (of 7,000) • found 98 that may be bugs! • 231 errors associated with a CVS bug fix flagged function • 61 of the 98 bugs found here • false positive rate of 74% • 222 errors associated with a function that has its return value checked > 50% of the time • 37 of the 98 bugs found here • false positive rate of 83%

  18. Case Study: A Bug • We investigated an error and found it did crash httpd • error reported near the top of the ranking • The called function builds a filename • arguments represent file and pathname • a char array is returned and directly used as an argument to strcmp() • strcmp(foo()) • NULL return value will cause a seg fault • return value is NULL if the path is too long!

  19. Analysis • False positive rate too high! • overall false positive rate: 78% (1-(98/453)) • A false positive rate closer to 50% would be acceptable • the user is likely as not to find a true error • cluster them near the top of the ranking • We did cull 7,000 errors down to 453 • lint would have flagged only the ‘ignored’ errors and not ranked them

  20. Conclusion • Bug databases are not useful in understanding much about low-level bugs • good for logic errors • good for misunderstood specifications • CVS commit messages give a better picture of low-level bugs • especially bugs that don’t enter a release • CVS commits can give useful data to help classify error reports

  21. Future Work • What other types of bugs are common? • What other checkers can benefit from CVS data? • How can we cut the false positive rate? • Can we dynamically gather data on functions called via function pointers? • many of the error messages involved calls through function pointers • Dyninst will allow us to instrument function pointer call sites and gather data

More Related