Milk or Wine: Does Software Security Improve with Age?

1. Milk or Wine:Does Software Security Improve with Age? Andy Ozment & Stuart Schechter Group 62 Internal Presentation 14 February 2006

2. Outline • Motivation • Methodology • Results characterizing vulnerabilities • Vulnerability date of introduction • Code base evolution • Vulnerability density • Vulnerability half-life • Rate of vulnerability detection • Conclusion

3. Motivation Eric Rescorla. “Is Finding Security Holes a Good Idea?” Workshop on Economics and Information Security.May 2004: Minneapolis, MN, USA. • Rate of vulnerability detection is not decreasing • 4 operating systems • All vulns found 1997-2000 • Used ICAT database (now NVD) Vuln finding effort not producing return • Pool of undiscovered vulns is not diminishing Or… • Result is artifact of shortcomings inherent to ICAT db

4. Motivation:Know Your Enemy Key Question: Is the rate of vuln detection decreasing? General Concern: Software reliability field has knowledge about defects what do we know about vulnerabilities? • Are LOC numbers correlated to vuln numbers? • Is newer code more secure than older code? • What is the half-life of a vuln? • How important is old code to security?

5. Outline • Motivation • Methodology • Results characterizing vulnerabilities • Vulnerability date of introduction • Code base evolution • Vulnerability density • Vulnerability half-life • Rate of vulnerability detection • Conclusion

6. Methodology • Examined 7.5 years of OpenBSD vulnerabilities • May 1998 to November 2005 • 15 versions • 2.3 to 3.5, inclusive • Why OpenBSD? • Prioritizes security • Source code available via CVS

7. Methodology • All vulns listed on OpenBSD security page • Additional sources: ICAT/NVD, OSVDB, Bugtraq, X-Force • Vulnerability ‘birth’ and ‘death’ dates from CVS • ‘Foundation’ version OpenBSD 2.3 • First source stable release in which vulns consistently documented

8. Caveat:Current Vuln Hunting Environment Data set represents current vuln hunting environment • Cannot encompass changes in: • Number of vuln hunters • Effort expended by vuln hunters • Vuln hunting fads • Image file processing • Microsoft • Ideal: a precise, universal measure of time • E.g. execution time • Our data set is a reflection of the current environment

9. Outline • Motivation • Methodology • Results characterizing vulnerabilities • Vulnerability date of introduction • Code base evolution • Vulnerability density • Vulnerability half-life • Rate of vulnerability detection • Conclusion

10. Chart:Version of Introduction • OpenBSD 2.3 contributed 62% of vulns • These vulns introduced prior to start of 7.5 year study

11. Chart:Birth & Death Versions

12. Dominance of Old Vulnerabilities • Majority of vulns discovered were in ‘foundation’ version • 62% (87 of 140) of vulns discovered in 7.5 years • Why does foundational code dominate vuln discovery? • Is it badly written? • Does it dominate security functionality? • Is it more interesting/easy to vuln hunters? • Does it dominate the code base?

13. Motivation & Methodology:Source Code Evolution Does foundation version dominate the code base? • Methodology • Compare base versions (2.3 vs. 2.4) • Strip comments • Examine .c and .h files • Use diff tool • Ignore whitespace and end-of-lines

14. Chart:Source Code Evolution 3.5 2.9 2.6

15. Source Code Evolution • After 7.5 years and 15 releases, the foundation version • Contributed 62% of vulnerabilities • Constituted 61% of unaltered lines of code • So... • Are lines of code correllated with number of vulns?

16. Scatterplot:Lines of Code vs Number Vulnerabilities No corellation.

17. Vulnerability Densities • Wide range of vuln densities • Inflated LOC counts • Version 2.4 • Internet Key Exchange (2) • OpenSSL (3) • Compare vuln densities to conventional wisdom on defect densities • Vulns: < 0.033 / 1K LOC • Defects: 8-12 / 1K LOC

18. Vulnerability Half-life Different methods: • Lower bound: assume all vulns have been found • Constant time: equal length of time after different releases • Model discovery process: fit a curve and project outwards Constant time: 6 years after release (days)

19. Chart:Vulnerability Half-life Lower Bound

20. Outline • Motivation • Methodology • Results characterizing vulnerabilities • Vulnerability date of introduction • Code base evolution • Vulnerability density • Vulnerability half-life • Rate of vulnerability detection • Conclusion

21. Rate of Vulnerability Discovery • Remember original motivation: • Rate of vuln detection not decreasing • Costs of vuln hunting therefore overwhelm benefits • (Rescorla 2004) • Our data set is more precise • Caveat: reflection of current vuln hunting environment • Is the rate of vuln detection in OpenBSD 2.3 decreasing?

22. Chart:Raw Data Mean: 29.1 Median: 18  29.1 Range: 1 - 126 Days Since Study Began

23. Chart:Eight Slices of Raw Data Days Since Study Began

24. Chart:Number of Vulns per Study Eighth Confidence intervals derived from a normal approximation of a homogenous Poisson process

25. Chart:Two Slices of Raw Data Days Since Study Began

26. Chart:Number of Vulns per Study Half Confidence intervals derived from a normal approximation of a homogenous Poisson process

27. Chart:FIve Cuts of Raw Data Days Since Study Began

28. Chart:Time-Between-Failures

29. Reliability Growth Modeling • Rescorla argued vuln discovery rate was not decreasing • Used Laplace test • Attempted to fit reliability growth models • Laplace test • Assumes vuln discovery is non-homogenous Poisson process H0: No trend H1: Trends towards increasing rate of vuln discovery H2: Trends towards decreasing rate of vuln discovery

30. Chart:Laplace Test for Trend

31. Reliability Growth Models • Tested 8 models • Musa’s Logarithmic model fit • Acceptable one-step-ahead predictive accuracy • Intensity Start: 0.051 vulns/day End: 0.024 vulns/day • Proportion of total vulns that have been discovered: 67.6% • Current MTTF: 57.7 days

32. Conclusion • Foundation version dominates • 62% of vulnerabilities • Half-life lower bound of 2.6 years • 61% of code base at the end of study • Rate of detection is decreasing • Visual assessment of assumed homogenous Poisson • Laplace test • Musa’s Log. model estimates 67.6% of vulns have been found

33. Questions Questions?

34. Bibliography • Andy Ozment and Stuart Schechter. “Milk or Wine: Does Software Security Improve with Age?” Under consideration. • Eric Rescorla. “Is Finding Security Holes a Good Idea?” In Workshop on Economics and Information Security. May 2004: Minneapolis, MN, USA.

35. Motivation: Better to be found by the good guys Better to be found by good guys than by bad guys Implication • There exists a pool of vulns in every product • As that pool is diminished, finding vulns becomes more difficult • Bad guys and good guys are both less likely to find vulns • Rate of detection decreases (all else equal) Vulns repaired, diminishing remaining pool Finding vulns becomes more difficult Rate of vuln detection decreases • Eric Rescorla. “Is Finding Security Holes a Good Idea?” WEIS 2004

36. Motivation Does the rate of vulnerability detection in softwaredecrease as the software ages? Questions • Does the rate of discovery diminish significantlyduring the product’s lifespan? • Finding vulns imposes costs on users

37. Motivation:Better to be found by the good guys Better to be found by good guys than by bad guys • There exists a pool of vulns in every product • As pool is diminished, finding vulns becomes more difficult • Rate of detection decreases (all else equal) But… • Rate of detection is not decreasing (Rescorla 2004) • Pool is too large to be diminished during product life cycle or • All else is not equal Or… • Shortcomings inherent to ICAT db used by Rescorla

38. Motivation:Better to be found by the good guys if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;until;return; Better to be found by good guys than by bad guys • There exists a pool of vulns in every product • As pool is diminished, finding vulns becomes more difficult • Rate of detection decreases (all else equal) if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;until;return;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;until;return;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;until;return;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;free;for;while;do;i++;elsif;return;next;free;until;malloc;if;then;else;malloc;return; do; calloc; else; do;

39. Motivation (Rescorla 2004) • There exists a ‘White Hat’ vulnerability-hunting community • Not malicious • Not working for developer of product • Work of the White Hat community is expensive • Cost to vuln hunter of finding vuln • Cost to vendor of creating and testing patch • Patches inform bad guys, while not installed by all good guys • Asserted value of White Hat efforts • Encourages better coding • Provides (limited) insight into software quality • Better to be found by good guys than by bad guys