Software Reliability Issues: Examination and Analysis

1. Final exam week • Three things on finals week: • final exam • final project presentations • final project report

2. Program analysis and Software Reliability

3. Software reliability: issues • What are the issues?

4. Software reliability: issues • What is software reliability? How to measure it? • Bug counts ? Will we ever have bug-free software? • How many 9’s ? • Service Level Agreements ? • What is a bug? • Adherence to specifications • But what is a specification… • User unhappy: is that a bug? • Different levels of severity

5. Software reliability: issues • Cost of the methods for achieving reliability • Independently develop 5 versions of the software, run them all in parallel ) less likely that they fail at the same time in the same way. But… cost… is… high • For tools, cost of development of the tools • Burden on the programmer • fully automated vs. semi-automated methods • allow progressive adoption • Scalability vs. precision • start with scalability and get precision later? • Or the other way around?

6. Software reliability: issues • Level of guarantee provided by the method • Hard guarantees, statistical guarantees, no formal guarantee • What if tool is broken: trusted computing base • When is the method used? • compile-time, link-time, load-time, run-time • What does the tool see? • source code, assembly, the whole program or part of the program

7. if (…) { x := …; } else { y := …; } …; One way of dividing the spectrum Compiler 010010110100101011011

8. Compiler 010010110100101011011 if (…) { x := …; } else { y := …; } …; if (…) { x := …; } else { y := …; } …; One way of dividing the spectrum Static techniques Testing techniques Run-time techniques Compiler 010010110100101011011

9. Static techniques Testing techniques Run-time techniques if (…) { x := …; } else { y := …; } …; 010010110100101011011 One way of dividing the spectrum Static techniques Testing techniques Run-time techniques Compiler 010010110100101011011

10. if (…) { x := …; } else { y := …; } …; Static Techniques • Spec: says what code should and should not do • Complete spec: specifies all behaviors (hard to formalize) • Incomplete spec: only defines some behaviors • e.g. “no null derefs”, “requests received are eventually processed” • Many formalisms exist for specs (Pre/Post conditions, FSMs, Temporal Logic, Abstract State Machines etc.) Spec «¬  $  \ r t  l Code satisfies spec?

11. CleanL TSys DSL DFA WP/SP MC ATP Language Design Clean language design Type Systems Domain-specific languages … Spec «¬  $  \ r t  l Code satisfies spec? if (…) { x := …; } else { y := …; } …; Static Techniques • Program Analysis • Dataflow analysis • WP/SP • Model checking • Automated Theorem Proving • … Interaction between the two

12. CleanL TSys DSL DFA WP/SP MC ATP ESC/Java [Leino et al PLDI 2002] • Programmer annotates code with pre- and post-conditions, tool verifies that these hold Automated Theorem Prover object Foo { //@ PRE (FORMULA) method bar(...) { ... } //@ POST (FORMULA) } ) WP(POST, bar) = weakest condition Q such that Q at entry to bar establishes POST at exist Compute Weakest Precondition

13. How does Program Analysis fit in? • First, WP computation is a kind of program analysis • in fact, can be seen as a dataflow analysis • predicates as sets of states • WP operator acts as flow function • WP-based concrete semantics • Second, can use global points-to analysis to help WP computation • do better job with pointer stores *x := ... • or with field assignments o.f := ...

14. CleanL TSys DSL DFA WP/SP MC ATP Checker Checker Checker Rhodium [Lerner et al POPL 2005] Compiler DSL Opt DSL Opt DSL Opt DSL Opt DSL Opt DSL Opt Parser Code Gen

15. CleanL TSys DSL DFA WP/SP MC ATP Rhodium [Lerner et al POPL 2005] Compiler DSL Opt DSL Opt DSL Opt DSL Opt DSL Opt DSL Opt Parser Code Gen Checker Checker Checker

16. CleanL TSys DSL DFA WP/SP MC ATP Lemma For any Rhodium opt: If Local VC is true Then opt is OK Proof «¬  $  \ r t  l VCGen LocalVC Automatic Theorem Prover Rhodium [Lerner et al POPL 2005] Rdm Opt Opt-independent Checker Opt-dependent

17. How does Program Analysis fit in? • First, Rhodium is a system for expressing and checking the correctness of DFAs • Second, the Rhodium system is an example of program analysis of a domain-specific language

18. CleanL TSys DSL DFA WP/SP MC ATP ESP [Das et al PLDI 2002] Interface usage rules in documentation • Order of operations, data access • Resource management • Incomplete, wordy, not checked Violated rules ) crashes • Failed runtime checks • Unreliable software

19. CleanL TSys DSL DFA WP/SP MC ATP ESP [Das et al PLDI 2002] C Program Rules ESP Safe Not Safe

20. CleanL TSys DSL DFA WP/SP MC ATP ESP [Das et al PLDI 2002] • ESP is a program analysis that keeps track of object state at each program point • e.g.: is file handle open or closed? • Challenge: scale to large programs • One of scalability issues: merge nodes • Always analyze both sides of merge node ) exponential (or non-terminating) program analyses • ESP has a heuristic for handling merges that • avoids exponential blow-up and runs fast in practice • maintains enough precision to verify programs

21. How does Program Analysis fit in? • The core of ESP is a dataflow analysis that keeps track of the state of objects • we’ve seen examples of this before in class • More on ESP later today

22. CleanL TSys DSL DFA WP/SP MC ATP BLAST [Henzinger et al POPL 2000] Interface usage rules in documentation • Order of operations, data access • Resource management • Incomplete, wordy, not checked Violated rules ) crashes • Failed runtime checks • Unreliable software

23. CleanL TSys DSL DFA WP/SP MC ATP BLAST [Henzinger et al POPL 2000] C Program Rules BLAST Safe Error Trace

24. CleanL TSys DSL DFA WP/SP MC ATP BLAST [Henzinger et al POPL 2000] C Program Rules BLAST Safe Error Trace

25. CleanL TSys DSL DFA WP/SP MC ATP BLAST [Henzinger et al POPL 2000] BLAST start with a set of predicates augmented set of predicates Perform “Predicate Abstraction” Refine set of predicates Rules C Program Trace infeasible No errors found Analyze trace error trace found Trace feasible Error Trace Safe

26. CleanL TSys DSL DFA WP/SP MC ATP BLAST [Henzinger et al POPL 2000] BLAST start with a set of predicates augmented set of predicates Perform “Predicate Abstraction” Refine set of predicates Rules C Program Trace infeasible No errors found Analyze trace error trace found Trace feasible Error Trace Safe

27. How does Program Analysis fit in? • Predicate abstraction can be seen as a dataflow analysis that keeps track of certain predicates • Flow functions are computed using a theorem prover • More on predicate abstraction later today

28. Two examples in more detail • Property simulation • algorithm behind ESP • Predicate abstraction • one of the (many) algorithms used in BLAST and SLAM

29. Precision vs. scalability Precision Scalability

30. Prop Sim example: stdio usage in gcc void main () { if (dump) fil = fopen(dumpFile,”w”); if (p) x = 0; else x = 1; if (dump) fclose(fil); }

31. Prop Sim example: stdio usage in gcc void main () { if (dump) Open; if (p) x = 0; else x = 1; if (dump) Close; }

32. $uninit Print/Close * $error Open Close Open Opened Print Prop Sim example: stdio usage in gcc void main () { if (dump) Open; if (p) x = 0; else x = 1; if (dump) Close; }

33. Property Simulation • Goal: compute set of states an object can be in • in this case, what states files are in • The backdrop: context-sensitive RHS-style algorithm that propagates info about set of states objects can be in • Key novelty in Prop Sim lies in path sensitivity technique

34. Dataflow property analysis • Track only FSM state • Ignore non-state-changing code • At control flow join points: • Accumulate FSM states

35. entry dump T F Open p F T x = 0 x = 1 dump T F Close exit Example

36. entry dump T F Open p F T x = 0 x = 1 dump T F Close exit Example {$uninit} {$uninit,Opened} {$uninit,Opened} {$error,$uninit,Opened}

37. Path-sensitive property analysis • Symbolically evaluate the program • Track FSM state and execution state • At branch points: • Execution state implies branch direction? • Yes: process appropriate branch • No: split state and process both branches

38. entry dump T F Open p F T x = 0 x = 1 dump T F Close exit Example

39. entry dump T F Open p F T x = 0 x = 1 dump T F Close exit Example [$uninit] [$uninit|dump=T] [Opened|dump=T] [Opened|dump=T,p=T] [Opened|dump=T,p=T,x=0] [Opened|dump=T,p=T,x=0] [$uninit|dump=T,p=T,x=0]

40. $uninit Print/Close * $error Open Close Open Opened Print What happens in correct code? void main () { if (dump) Open; if (p) x = 0; else x = 1; if (dump) Close; } void main () { if (dump) Open; if (p) x = 0; else x = 1; if (dump) Close; }

41. When is a branch relevant? • Precise answer • When the value of the branch condition determines the property FSM state • Heuristic answer • When the property FSM is driven to different states along the arms of the branch statement

42. Property simulation • Same as path-sensitive analysis, except • At control flow join points: • States agree on property state? • Yes: merge states • No: process states separately

43. entry dump T F Open p F T x = 0 x = 1 dump T Close F exit Example

44. entry dump T F Open p F T x = 0 x = 1 dump T [Opened|dump=T,p=T,x=0] [Opened|dump=T,p=F,x=1] Close F exit Example [$uninit] [Opened|dump=T] [$uninit|dump=F] [Opened|dump=T] [$uninit|dump=F] [$uninit|dump=T][$uninit|dump=F] [$uninit|dump=T] [$uninit]

45. void main () { if (dump1) fil1 = fopen(dumpFile1,”w”); if (dump2) fil2 = fopen(dumpFile2,”w”); if (dump1) fclose(fil1); if (dump2) fclose(fil2); } void main () { if (dump1) Open(fil1); if (dump2) Open(fil2); if (dump1) Close(fil1); if (dump2) Close(fil2); } $uninit Print/Close * $error Source code pattern Source code pattern Transition Transition Open Close Open e = fopen(_) e = fopen(_) Open Open Opened Print fclose(e) fclose(e) Close Close Reality bites void main () { if (dump1) fil1 = fopen(dumpFile1,”w”); if (dump2) fil2 = fopen(dumpFile2,”w”); if (dump1) fclose(fil1); if (dump2) fclose(fil2); }

46. Property simulation, bit by bit void main () { if (dump1) Open; if (dump2) ID; if (dump1) Close; if (dump2) ID; } void main () { if (dump1) ID; if (dump2) Open; if (dump1) ID; if (dump2) Close; }

47. Property simulation, bit by bit • One FSM at a time + Avoids exponential property state + Fewer branches are relevant + Lifetimes are often short + Embarassingly parallel − Cannot correlate FSMs

48. Reality bites, again void main () { if (dump1) fil1 = fopen(dumpFile1,”w”); if (dump2) fil2 = fopen(dumpFile2,”w”); fil3 = fil1; if (dump1) fclose( fil3 ); if (dump2) fclose( fil2 ); } • Run-time values have state • Syntactic names hold run-time values • Values come from creation patterns

49. Value flow analysis • When does a pattern on name e cause a state transition for value v? • When v may flow to (be held by) e • ESP uses an alias analysis to figure out if two one value flows to another • intuition: a flows to b if in the points to graph *a and *b are represented using the same node (ie: *a and *b alias) • ESP uses Generalized One Level Flow (GOLF)

50. x y x y x y x = y Recall: Steensgaard Steensgaard