Inferring and checking system rules by static analysis

1. Inferring and checking system rules by static analysis William R Wright

2. Material reviewed… • Checking System Rules Using System-Specific, Programmer-Written Compiler Extensions. Dawson Engler, et al. OSDI 2000. • RacerX: Effective, Static Detection of Race Conditions and Deadlocks. Dawson Engler, et al. SOSP 2003. • Bugs as Deviant Behavior: A General Approach to Inferring Errors in Systems Code. OSDI 2000. CS 297: Security and Programming Languages

3. System Rules • Systems follow their own unique set of design based “correctness” rules • Such rules go beyond “no dereferencing of NULL pointers CS 297: Security and Programming Languages

4. Example – System rule • Temporal ordering: b must always follow a CS 297: Security and Programming Languages

5. Checking of such rules • These rules are often unchecked. • For example, suppose I am required to issue conn.close() but forget to do so. • Code compiles even though I broke a system “rule”. CS 297: Security and Programming Languages

6. How to check system wide rules? • Although compilers observe general language semantics, they are ignorant of the rules unique to systems. • Static analysis can apply systems rules. CS 297: Security and Programming Languages

7. Checking systems Rules • ESC enables some checking via annotations (even automating annotations via Houdini). • Vault – very manual. • SLAM – labeled “promising” - next presentation! CS 297: Security and Programming Languages

8. Proposed method • The proposed method complements those efforts. • Goal is to extract ad hoc rules from source code, requiring minimal effort. • Also, to provide an extensible framework to define and check systems rules. CS 297: Security and Programming Languages

9. Meta-Level compilation (MC) • One may define a systems rule via metal, a “high level, state-machine” language. CS 297: Security and Programming Languages

10. Details about metal • A rule defined in metal is called a State Machine (SM). • Once so defined, we compile the rule(s) with mcc, a metal compiler, and dynamically link the result into xgcc (based on GNU gcc). CS 297: Security and Programming Languages

11. Details about metal (cont’d) • When compiling the code be analyzed, xgcc outputs errors based on deviations from the metal rules. • Notice that modifications to source are unnecessary. If one there is a bug fix, one can easily recompile with the compiler of choice. CS 297: Security and Programming Languages

12. Sample Metal rule templates • With metal one may define systems rules such as: • “Never/always do X” • “Always do X before/after Y” • “In situation X, do (not do) Y” • “In situation X, do Y rather than Z” CS 297: Security and Programming Languages

13. Example – metal rule(from Deviant paper) CS 297: Security and Programming Languages

14. Example: …more problem code (Extensions paper) CS 297: Security and Programming Languages

15. Example (cont’d) • metal rule can find the bug - 6 lines of code • (Extensions, Fig. 1, Section 3.1) • Finds deviations by looking fo • Functions to look for • Disable interrupts: cli() • Re-enable interrupts: sti() or restore_flags(flags) [restores to original interrupt status when paired with save_flags(flags)] CS 297: Security and Programming Languages

16. Inferring deviant behavior • Suppose you were just hired to get uptime from 98% to 99.999% on an airline reservation system with 5,000,000 lines of code. • You know little about the system. Those who do ask you to help with the debugging. • The newspaper reports that “Software problems” caused your employer’s latest disaster. CS 297: Security and Programming Languages

17. Inferring deviant behavior (cont’d) • The programming team has already used their knowledge of the system and as much static analysis as they could come up with. • What if you could automate an examination of the source that results in a set of Metal rules that reflect the ad hoc (undocumented) behavior that the system should follow? CS 297: Security and Programming Languages

18. Inference Method • Assume that action taken by code is there to accomplish something. • Divide actions into inferences about programmer “beliefs”. CS 297: Security and Programming Languages

19. MUST beliefs • Beliefs then go in two categories: • MUST beliefs: if (p==null){ System.out.println( “The pointer is” + p + “.”); } Programmer expresses belief that p is null inside the block, then contradicts the belief. CS 297: Security and Programming Languages

20. MAY beliefs • MAY beliefs: back to original example • Since we see stmt.close after stmt.executeQuery, maybe this is a system rule. CS 297: Security and Programming Languages

21. Beliefs – discovering • One may derive possible MAY beliefs by: • Traversing the program, observing all actions that happen in tandem. • Assuming that they MUST happen in that way. • In a second pass, applying those assumptions. • Initiate a statistical analysis of the results (errors). CS 297: Security and Programming Languages

22. Statistical analysis • One may quickly rule out many MAY beliefs when finding that they are rarely or infrequently followed. These are “coincidences”, not MUST beliefs. CS 297: Security and Programming Languages

23. Statistical analysis - detail • Sort the errors by the z statistic: • Which essentially measures the degree to which a MAY belief is supported by its incidence (contradictions accounted for) in the code sample. • One examines errors from the most likely to the least, stopping when the effort becomes counterproductive. CS 297: Security and Programming Languages

24. Results • Coverity, commercialized these methods • Look at http://scan.coverity.com/ • “Number of defects Fixed (since 3/6/2006): 6131. • Targeted gcc, Samba, Linux-2.6, Perl, PHP, OpenSSL, and apparently plenty of “private” code. CS 297: Security and Programming Languages