1 / 39

Milk or Wine: Does Software Security Improve with Age?

Milk or Wine: Does Software Security Improve with Age?. Andy Ozment & Stuart Schechter Group 62 Internal Presentation 14 February 2006. Outline. Motivation Methodology Results characterizing vulnerabilities Vulnerability date of introduction Code base evolution Vulnerability density

pembroke
Download Presentation

Milk or Wine: Does Software Security Improve with Age?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Milk or Wine:Does Software Security Improve with Age? Andy Ozment & Stuart Schechter Group 62 Internal Presentation 14 February 2006

  2. Outline • Motivation • Methodology • Results characterizing vulnerabilities • Vulnerability date of introduction • Code base evolution • Vulnerability density • Vulnerability half-life • Rate of vulnerability detection • Conclusion

  3. Motivation Eric Rescorla. “Is Finding Security Holes a Good Idea?” Workshop on Economics and Information Security.May 2004: Minneapolis, MN, USA. • Rate of vulnerability detection is not decreasing • 4 operating systems • All vulns found 1997-2000 • Used ICAT database (now NVD) Vuln finding effort not producing return • Pool of undiscovered vulns is not diminishing Or… • Result is artifact of shortcomings inherent to ICAT db

  4. Motivation:Know Your Enemy Key Question: Is the rate of vuln detection decreasing? General Concern: Software reliability field has knowledge about defects what do we know about vulnerabilities? • Are LOC numbers correlated to vuln numbers? • Is newer code more secure than older code? • What is the half-life of a vuln? • How important is old code to security?

  5. Outline • Motivation • Methodology • Results characterizing vulnerabilities • Vulnerability date of introduction • Code base evolution • Vulnerability density • Vulnerability half-life • Rate of vulnerability detection • Conclusion

  6. Methodology • Examined 7.5 years of OpenBSD vulnerabilities • May 1998 to November 2005 • 15 versions • 2.3 to 3.5, inclusive • Why OpenBSD? • Prioritizes security • Source code available via CVS

  7. Methodology • All vulns listed on OpenBSD security page • Additional sources: ICAT/NVD, OSVDB, Bugtraq, X-Force • Vulnerability ‘birth’ and ‘death’ dates from CVS • ‘Foundation’ version OpenBSD 2.3 • First source stable release in which vulns consistently documented

  8. Caveat:Current Vuln Hunting Environment Data set represents current vuln hunting environment • Cannot encompass changes in: • Number of vuln hunters • Effort expended by vuln hunters • Vuln hunting fads • Image file processing • Microsoft • Ideal: a precise, universal measure of time • E.g. execution time • Our data set is a reflection of the current environment

  9. Outline • Motivation • Methodology • Results characterizing vulnerabilities • Vulnerability date of introduction • Code base evolution • Vulnerability density • Vulnerability half-life • Rate of vulnerability detection • Conclusion

  10. Chart:Version of Introduction • OpenBSD 2.3 contributed 62% of vulns • These vulns introduced prior to start of 7.5 year study

  11. Chart:Birth & Death Versions

  12. Dominance of Old Vulnerabilities • Majority of vulns discovered were in ‘foundation’ version • 62% (87 of 140) of vulns discovered in 7.5 years • Why does foundational code dominate vuln discovery? • Is it badly written? • Does it dominate security functionality? • Is it more interesting/easy to vuln hunters? • Does it dominate the code base?

  13. Motivation & Methodology:Source Code Evolution Does foundation version dominate the code base? • Methodology • Compare base versions (2.3 vs. 2.4) • Strip comments • Examine .c and .h files • Use diff tool • Ignore whitespace and end-of-lines

  14. Chart:Source Code Evolution 3.5 2.9 2.6

  15. Source Code Evolution • After 7.5 years and 15 releases, the foundation version • Contributed 62% of vulnerabilities • Constituted 61% of unaltered lines of code • So... • Are lines of code correllated with number of vulns?

  16. Scatterplot:Lines of Code vs Number Vulnerabilities No corellation.

  17. Vulnerability Densities • Wide range of vuln densities • Inflated LOC counts • Version 2.4 • Internet Key Exchange (2) • OpenSSL (3) • Compare vuln densities to conventional wisdom on defect densities • Vulns: < 0.033 / 1K LOC • Defects: 8-12 / 1K LOC

  18. Vulnerability Half-life Different methods: • Lower bound: assume all vulns have been found • Constant time: equal length of time after different releases • Model discovery process: fit a curve and project outwards Constant time: 6 years after release (days)

  19. Chart:Vulnerability Half-life Lower Bound

  20. Outline • Motivation • Methodology • Results characterizing vulnerabilities • Vulnerability date of introduction • Code base evolution • Vulnerability density • Vulnerability half-life • Rate of vulnerability detection • Conclusion

  21. Rate of Vulnerability Discovery • Remember original motivation: • Rate of vuln detection not decreasing • Costs of vuln hunting therefore overwhelm benefits • (Rescorla 2004) • Our data set is more precise • Caveat: reflection of current vuln hunting environment • Is the rate of vuln detection in OpenBSD 2.3 decreasing?

  22. Chart:Raw Data Mean: 29.1 Median: 18  29.1 Range: 1 - 126 Days Since Study Began

  23. Chart:Eight Slices of Raw Data Days Since Study Began

  24. Chart:Number of Vulns per Study Eighth Confidence intervals derived from a normal approximation of a homogenous Poisson process

  25. Chart:Two Slices of Raw Data Days Since Study Began

  26. Chart:Number of Vulns per Study Half Confidence intervals derived from a normal approximation of a homogenous Poisson process

  27. Chart:FIve Cuts of Raw Data Days Since Study Began

  28. Chart:Time-Between-Failures

  29. Reliability Growth Modeling • Rescorla argued vuln discovery rate was not decreasing • Used Laplace test • Attempted to fit reliability growth models • Laplace test • Assumes vuln discovery is non-homogenous Poisson process H0: No trend H1: Trends towards increasing rate of vuln discovery H2: Trends towards decreasing rate of vuln discovery

  30. Chart:Laplace Test for Trend

  31. Reliability Growth Models • Tested 8 models • Musa’s Logarithmic model fit • Acceptable one-step-ahead predictive accuracy • Intensity Start: 0.051 vulns/day End: 0.024 vulns/day • Proportion of total vulns that have been discovered: 67.6% • Current MTTF: 57.7 days

  32. Conclusion • Foundation version dominates • 62% of vulnerabilities • Half-life lower bound of 2.6 years • 61% of code base at the end of study • Rate of detection is decreasing • Visual assessment of assumed homogenous Poisson • Laplace test • Musa’s Log. model estimates 67.6% of vulns have been found

  33. Questions Questions?

  34. Bibliography • Andy Ozment and Stuart Schechter. “Milk or Wine: Does Software Security Improve with Age?” Under consideration. • Eric Rescorla. “Is Finding Security Holes a Good Idea?” In Workshop on Economics and Information Security. May 2004: Minneapolis, MN, USA.

  35. Motivation: Better to be found by the good guys Better to be found by good guys than by bad guys Implication • There exists a pool of vulns in every product • As that pool is diminished, finding vulns becomes more difficult • Bad guys and good guys are both less likely to find vulns • Rate of detection decreases (all else equal) Vulns repaired, diminishing remaining pool Finding vulns becomes more difficult Rate of vuln detection decreases • Eric Rescorla. “Is Finding Security Holes a Good Idea?” WEIS 2004

  36. Motivation Does the rate of vulnerability detection in softwaredecrease as the software ages? Questions • Does the rate of discovery diminish significantlyduring the product’s lifespan? • Finding vulns imposes costs on users

  37. Motivation:Better to be found by the good guys Better to be found by good guys than by bad guys • There exists a pool of vulns in every product • As pool is diminished, finding vulns becomes more difficult • Rate of detection decreases (all else equal) But… • Rate of detection is not decreasing (Rescorla 2004) • Pool is too large to be diminished during product life cycle or • All else is not equal Or… • Shortcomings inherent to ICAT db used by Rescorla

  38. Motivation:Better to be found by the good guys if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;until;return; Better to be found by good guys than by bad guys • There exists a pool of vulns in every product • As pool is diminished, finding vulns becomes more difficult • Rate of detection decreases (all else equal) if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;until;return;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;until;return;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;until;return;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;return; do; calloc; else; do;

  39. Motivation (Rescorla 2004) • There exists a ‘White Hat’ vulnerability-hunting community • Not malicious • Not working for developer of product • Work of the White Hat community is expensive • Cost to vuln hunter of finding vuln • Cost to vendor of creating and testing patch • Patches inform bad guys, while not installed by all good guys • Asserted value of White Hat efforts • Encourages better coding • Provides (limited) insight into software quality • Better to be found by good guys than by bad guys

More Related