Software Reliability Methods

1. Software Reliability Methods Sorin Lerner

2. Software reliability methods: issues • What are the issues?

3. Software reliability methods: issues • What is software reliability? How to measure it? • Bug counts ? Will we ever have bug-free software? • How many 9’s ? • Service Level Agreements ? • What is a bug? • Adherence to specifications • But what is a specification… • User unhappy: is that a bug? • Different levels of severity

4. Software reliability methods: issues • Cost of the methods for achieving reliability • Independently develop 5 versions of the software, run them all in parallel ) less likely that they fail at the same time in the same way. But… cost… is… high • For tools, cost of development of the tools • Burden on the programmer • fully automated vs. semi-automated methods • allow progressive adoption

5. Software reliability methods: issues • Level of guarantee provided by the method • Hard guarantees, statistical guarantees, no formal guarantee • What if tool is broken: trusted computing base • When is the method used? • compile-time, link-time, load-time, run-time • What does the tool see? • source code, assembly, the whole program or part of the program

6. if (…) { x := …; } else { y := …; } …; One way of dividing the spectrum Compiler 010010110100101011011

7. Compiler 010010110100101011011 if (…) { x := …; } else { y := …; } …; if (…) { x := …; } else { y := …; } …; One way of dividing the spectrum Static techniques Testing techniques Run-time techniques Compiler 010010110100101011011

8. Static techniques Testing techniques Run-time techniques if (…) { x := …; } else { y := …; } …; 010010110100101011011 One way of dividing the spectrum Static techniques Testing techniques Run-time techniques Compiler 010010110100101011011

9. if (…) { x := …; } else { y := …; } …; Static Techniques • Spec: says what code should and should not do • Complete spec: specifies all behaviors (hard to formalize) • Incomplete spec: only defines some behaviors • e.g. “no null derefs”, “requests received are eventually processed” • Many formalisms exist for specs (Pre/Post conditions, FSMs, Temporal Logic, Abstract State Machines etc.) Spec «¬  $  \ r t  l Code satisfies spec?

10. CleanL TSys DSL DFA WP/SP MC ATP Language Design Clean language design Type Systems Domain-specific languages … Spec «¬  $  \ r t  l Code satisfies spec? if (…) { x := …; } else { y := …; } …; Static Techniques • Program Analysis • Dataflow analysis • WP/SP • Model checking • Automated Theorem Proving • … Interaction between the two

11. CleanL TSys DSL DFA WP/SP MC ATP ESC/Java [Leino et al PLDI 2002] • Programmer annotates code with pre- and post-conditions, tool verifies that these hold Automated Theorem Prover object Foo { //@ PRE (FORMULA) method bar(...) { ... } //@ POST (FORMULA) } ) WP(POST, bar) = weakest condition Q such that Q at entry to bar establishes POST at exist Compute Weakest Precondition

12. CleanL TSys DSL DFA WP/SP MC ATP Checker Checker Checker Rhodium [Lerner et al POPL 2005] Compiler DSL Opt DSL Opt DSL Opt DSL Opt DSL Opt DSL Opt Parser Code Gen

13. CleanL TSys DSL DFA WP/SP MC ATP Rhodium [Lerner et al POPL 2005] Compiler DSL Opt DSL Opt DSL Opt DSL Opt DSL Opt DSL Opt Parser Code Gen Checker Checker Checker

14. CleanL TSys DSL DFA WP/SP MC ATP Lemma For any Rhodium opt: If Local VC is true Then opt is OK Proof «¬  $  \ r t  l VCGen LocalVC Automatic Theorem Prover Rhodium [Lerner et al POPL 2005] Rdm Opt Opt-independent Checker Opt-dependent

15. CleanL TSys DSL DFA WP/SP MC ATP ESP [Das et al PLDI 2002] Interface usage rules in documentation • Order of operations, data access • Resource management • Incomplete, wordy, not checked Violated rules ) crashes • Failed runtime checks • Unreliable software

16. CleanL TSys DSL DFA WP/SP MC ATP ESP [Das et al PLDI 2002] C Program Rules ESP Safe Not Safe

17. CleanL TSys DSL DFA WP/SP MC ATP ESP [Das et al PLDI 2002] • ESP is a program analysis that keeps track of object state at each program point • e.g.: is file handle open or closed? • Challenge: scale to large programs • One of scalability issues: merge nodes • Always analyze both sides of merge node ) exponential (or non-terminating) program analyses • ESP has a heuristic for handling merges that • avoids exponential blow-up and runs fast in practice • maintains enough precision to verify programs

18. BLAST [Henzinger et al POPL 2000] Interface usage rules in documentation • Order of operations, data access • Resource management • Incomplete, wordy, not checked Violated rules ) crashes • Failed runtime checks • Unreliable software

19. BLAST [Henzinger et al POPL 2000] C Program Rules BLAST Safe Error Trace

20. BLAST [Henzinger et al POPL 2000] C Program Rules BLAST Safe Error Trace

21. BLAST [Henzinger et al POPL 2000] BLAST start with a set of predicates augmented set of predicates Perform “Predicate Abstraction” Refine set of predicates Rules C Program Trace infeasible No errors found Analyze trace error trace found Trace feasible Error Trace Safe

22. CleanL TSys DSL DFA WP/SP MC ATP BLAST [Henzinger et al POPL 2000] BLAST start with a set of predicates augmented set of predicates Perform “Predicate Abstraction” Refine set of predicates Rules C Program Trace infeasible No errors found Analyze trace error trace found Trace feasible Error Trace Safe

23. CleanL TSys DSL DFA WP/SP MC ATP Type Systems • What is a type system? • A discipline for writing code that can be mechanically checked, and can prevent certain kinds of run-time errors • For example, java type system prevents calling methods that don’t exists, or calling methods with parameters of the wrong type

24. CleanL TSys DSL DFA WP/SP MC ATP Type Systems • Type systems can track and provide guarantees about many other aspects of computation: • Safe explicit memory management (Crary, Walker and Morrisett, POPL 99) • Execution time bounds (Crary and Weirich, POPL 00) • Information flow (Myers, POPL 00) • Security automata (Walker, POPL 00)

25. CleanL TSys DSL DFA WP/SP MC ATP Type Systems • MultiJava [Clifton et al 2000] adds to Java: • multi-methods: methods that dispatch symetrically on the type of all params, not just the first • open classes: classes • Adding these features makes modular type checking harder, and required innovations on the type system side • Interplay between language design and type systems