Analysis is necessary – but far from sufficient

1. Analysis is necessary – but far from sufficient Jon Pincus Reliability Group (PPRC) Microsoft Research

2. Why are so few successful real-world development and testing tools influenced by program analysis research? Jon Pincus (Microsoft Research)

3. Outline • Provocation • Successful tools • Analysis – in context • Implications for analysis • Conclusion Jon Pincus (Microsoft Research)

4. Success: a simple view • A tool is successful if people use it • Not if people think it’s interesting – but don’t try it • Not if people try it but don’t use it • Not if people buy it but don’t use it (“Shelfware”) Jon Pincus (Microsoft Research)

5. Some examples of success • Purify • BoundsChecker • PREfix (2.X and later) • Especially interesting because 1.0 was unsuccessful Jon Pincus (Microsoft Research)

6. Why do people use a tool? If • it helps them get their work done … • … more efficiently than they would otherwise • … without making them look (or feel) bad. Aside: look at organizational and personal goals. See Alan Cooper’s books, e.g. About Face Jon Pincus (Microsoft Research)

7. Value vs. Cost • Value: the quantified benefit from the tool • Cost: primarily time investment • Licensing cost is typically much smaller • (Value – Cost) must be • Positive • Positive fairly quickly • More positive than any alternatives • Value and cost are difficult to estimate … … and others’ estimates are often questionable Jon Pincus (Microsoft Research)

8. An example • Purify 1.0: • Virtually zero initial cost on most code bases • “trial” license • easy to integrate • Immediate value • Companies then invested to increase the value • E.g., changing memory allocators to better match Purify’s • (and buying lots of licenses) Jon Pincus (Microsoft Research)

9. Characteristics of successful tools • Successful tools almost always • address significant problems, • on real code bases, • give something for (almost) nothing, • and are easy to use. Jon Pincus (Microsoft Research)

10. Significant problems • Nobody fixes all the bugs. • What are the key ones? • Often based on most recent scars • Often based on development or business goals • Examples: • Purify: memory leaks • BoundsChecker: bounds violations • Lint (back in K&R days): portability issues Jon Pincus (Microsoft Research)

11. Real code bases • Large code bases in nasty languages (e.g., C/C++) • 1M+ LOC is medium-sized; 10M+ LOC is large • Or, smaller code bases in different nasty languages • Perl, JScript, VBScript, HTML/DHTML, TCL/Tk, SQL • 5000+ LOC is medium; 50K+ is large Jon Pincus (Microsoft Research)

12. More reality … • Most code bases involve multiple languages • Extensions and incompatibilities, e.g. • GCC/G++, MS C++, Sun C++ • ECMAScript/JScript/JavaScript • HTML versions • People use all those nasty language features (e.g., casts between pointers and ints, unions, bit fields, gotos, …) Jon Pincus (Microsoft Research)

13. Something for (almost) nothing • Engineering time is precious • Engineers are skeptical … so are unwilling to commit their valuable time • Don’t even think about requiring significant up-front investment • code modifications • process changes Jon Pincus (Microsoft Research)

14. Examples: something for (almost) nothing • Purify for UNIX: just relink! • BoundsChecker: you don’t even need to relink!! • PREfix 2.X: point your web browser to a URL!!! • A non-technology solution: “we’ll do it for you” • Commercial variant: an initial benchmark for $X • Preferably: money back if it isn’t useful • In many cases, money is cheaper than engineering time … Jon Pincus (Microsoft Research)

15. “Revolutionary tools” • People may be willing to do up-front work to • Enable something previously impossible • Or provide order-of-magnitude improvements • BUT! • Still must be significant problem, real code base • Need compelling evidence of chance for success • Any examples? Jon Pincus (Microsoft Research)

16. Outline • What makes a tool successful? • Successful tools • Analysis – in context • Implications for analysis • Conclusion Jon Pincus (Microsoft Research)

17. PREfix • Analyzes C/C++ source code • Identifies defects • GUI to aid understanding and prioritization • Viewing individual defects • Sorting/filtering sets of defects • Integrates smoothly into existing builds • Stores results in database Jon Pincus (Microsoft Research)

18. #include <std.h> int PwrOf2(int a) { if (a & (a - 1)) return 0; else return 1; } Simulator (mod “PwrOf2” (c “a” init) (t t1 (& a (-a 1))) (g t1<0:4> !0 (r 0 success)) (g t1<0:4> 0) (r 1 success))) Execution Control Auto Modeler Error Analysis Virtual Machine PREfix 2.X Architecture Web Browser C/C++ Parser Source Code Defect Database Model Database Jon Pincus (Microsoft Research)

19. Counterintuitively … Actual analysis is only a small part of any “program analysis tool”. In PREfix, < 10% of the “code mass” Jon Pincus (Microsoft Research)

20. 3 key non-analysis issues • Parsing • Integration • Build process • Defect tracking system • SCM system • User interaction • Information presentation • Navigation • Control Jon Pincus (Microsoft Research)

21. Parsing • You can’t parse better than anybody else … … but you can parse worse • Complexities: • Incompatibilities and extensions • Full language complexity • Language evolution • Solution: don’t • Alternatives: GCC, EDG, … Jon Pincus (Microsoft Research)

22. Integration • A tool is useless if people can’t use it • Implied: “use it in their existing environment” • “Environment” includes • Configuration management (SCM) • A build process (makefiles, scripts, …) • Policies • A defect tracking system • People have invested hugely in their environment • They probably won’t change it just for one tool Jon Pincus (Microsoft Research)

23. User interaction • Engineers must be able to • Use the analysis results • Understanding individual defects • Prioritizing, sorting, and filtering sets of defects • Interact with other engineers • Influence the analysis • Current tools are at best “okay” here • Improvement is highly leveraged Jon Pincus (Microsoft Research)

24. Example: Noise • Noise = “messages people don’t care about” • Noise can result from • Incorrect tool requirements • Integration issues • Usability issues (e.g., unclear messages) • Analysis inaccuracies • … Jon Pincus (Microsoft Research)

25. Dealing with noise • Improving analysis is usually not sufficient • May be vital; may not be required • Successful user interaction techniques: • Filtering • History • Prioritization • Improving presentation, navigation • Providing more detail Jon Pincus (Microsoft Research)

26. Outline • What makes a tool successful? • Characteristics of successful tools • Analysis – in context • Implications for analysis • Conclusion Jon Pincus (Microsoft Research)

27. Characteristics of useful analyses • Scalable to “large enough” system • Typically implies incomplete, unsound, decomposable, and/or very simple • “Accurate enough” for the task at hand • Produce information usable by typical engineer • E.g., if there’s a defect, where? How? Why? • Remember: half the engineers are below average • Handle full language complexity • (or degrades gracefully for unhandled constructs) • Handle partial programs Jon Pincus (Microsoft Research)

28. Analyses are not useful if … • They don’t apply to the tool’s “reality” • “For a subset of C, excluding pointers and structs …” • “We have tested on our approach on programs up to several thousand lines of Scheme …” • They assume up-front work for the end user • “Once the programmer modifies the code to include calls to the appropriate functions …” • “The programmer simply inserts the annotations to be checked as conventional comments …” Jon Pincus (Microsoft Research)

29. Different tradeoffs from compilers • Focus on information, not just results • Compilers don’t have to explain what they did and why • Unsoundness is death for optimization – but may be okay for other purposes • Intra-procedural analysis often not enough Jon Pincus (Microsoft Research)

30. Types of analyses • FCIA: Flow- and context-insensitive • FSA: Flow-sensitive • CSA: Context-sensitive • FCSA: Flow and context sensitive • PSA: Path-sensitive Jon Pincus (Microsoft Research)

31. Performance vs. “Accuracy”

32. Don’t forget “information”!

33. Example analysis tradeoffs • PREfix: scalable, usable analysis results • Path-sensitive • Incomplete (limit # of paths traversed) • Unsound (many approximations) • Major emphasis on summarization (“models”) • PREfast: fast, usable analysis results • Local analyses, using PREfix models • Flow-insensitive and flow-sensitive analyses • Far less complete than PREfix Jon Pincus (Microsoft Research)

34. Aside: Techniques for scalability • Decompose the problem • Use the existing structure (function, class, etc.) • Summarization, memoization • Caveat: make sure you don’t lose key info! • Give up completeness and soundness • Use three-valued logic with “don’t know” state • Track approximations to limit the damage • Examine and re-examine tradeoffs!!!! • Optimize for significant special cases Jon Pincus (Microsoft Research)

35. Outline • What makes a tool successful? • Characteristics of successful tools • Analysis – in context • Implications for analysis • Conclusion Jon Pincus (Microsoft Research)

36. Recap: successful tools • People use tools to accomplish their tasks • Successful tools must • address real problems, • on real code bases, • give something for (almost) nothing, • and be easy to use • Analysis is only one piece of a tool • Information is useless if it’s not presented well Jon Pincus (Microsoft Research)

37. One person’s opinion Why are so few successful real-world development and testing tools influenced by program analysis research? Several key areas are outside the traditional scope of program analysis research User interaction Visualization (of programs and analysis results) Integration Jon Pincus (Microsoft Research)

38. One person’s opinion (cont.) Why are there so few successful real-world programming and testing tools based on academic research? Program analysis research in general: Not directly focused on “key problems” Not applicable to “real world” code bases Makes unrealistic assumptions about up-front work Jon Pincus (Microsoft Research)

39. One tool developer’s mindset • We have plenty of ideas already. • We can’t even implement all our pet projects! • We are interested in new ideas – but skeptical • The burden is on you to show relevance • Remember, analysis is only part of our problem • If we can’t figure out how to present it – forget it Jon Pincus (Microsoft Research)

40. Making analysis influential • Show how the analysis addresses a significant problem • Synchronization, security, … • Convince us that it will work in our reality • Avoid the obvious problems discussed above • Demonstrate in our reality … (perhaps by using real-world code bases) • … or persuade us that it will work Jon Pincus (Microsoft Research)

41. Some interesting questions … • Which analyses are right for which problems? • How to get difficult analyses to scale well? • Are there soundness/completeness tradeoffs? • Are there opportunities to combine analyses? • Can we use a cheap flow-insensitive algorithm to focus a more expensive algorithm on juicy places? • Can we use expensive local path-sensitive algorithms to improve global flow-insensitive algorithms? Jon Pincus (Microsoft Research)

42. Beyond analysis • Can visualization and user interaction for analysis tools become an interesting research area? • How can analysis be used to refine visualization and user interaction? Jon Pincus (Microsoft Research)

43. Questions? Jon Pincus (Microsoft Research)

44. Analysis is necessary – but far from sufficient Jon Pincus Reliability Group (PPRC) Microsoft Research