Today

1. Today • Executions, paths, and states • Testing via model checking

2. Executions, Paths, and States Control flow graph (CFG):structure of a program’s code An execution is simply arun of the program – the resultof some set of commands Some executions, with different operations: A path is a particular waythrough the CFG – ignoringthe commands – where theprogram “goes” A state is the program’s location+ memory contents at a point intime Same state, different executions,different paths (in the executions, color only showsthe location in the CFG, not thecontents of memory, which may bedifferent even if colors are the same) Same path, different executions Same paths, different states

3. A Question • Testing can focus on exploring lots of • Executions • Random testing • Paths • Concolic testing • States? • Model checking • But first: does this gain us anything?

4. A Question • First question: does it make the problem more manageable • That is, are there cases where exploring all states is more feasible than exploring all paths?

5. Paths vs. States int main () {foo(x, y);bar(x, y);if (x + y == 40) G++; } int G = 0; int foo (int x, int y) {if (x < y) G++;if (x == 10) G++; } int bar (int x, int y) {if (x < 50) G++;if (y > 8) G++; } 2 32 4 Once more, how many paths through thisprogram? 8 16

6. Paths vs. States int main () {foo(x, y);bar(x, y);if (x + y == 40) G++; } int G = 0; int foo (int x, int y) {if (x < y) G++;if (x == 10) G++; } int bar (int x, int y) {if (x < 50) G++;if (y > 8) G++; } What if we consider the number of states? If we include x and y’s values in the state,the number is much larger than the numberof paths… so it’s certainly not always animprovement!

7. Paths vs. States Less clear which is better in thiscase: lots of states, but also lotsof paths pick(chunk, 0, 10); pick(nbytes, 0, 10); write(chunk, wbuf, nbytes); pick(chunk, 0, 10); pick(nbytes, 0, 10); read(chunk, rbuf, nbytes); pick(chunk, 0, 10); pick(nbytes, 0, 10); write(chunk, wbuf, nbytes); Paths lets us consider inputs thatproduce the same sequence of writesas “the same” States means we don’t have to worryabout error returns in the past, if thaterror return doesn’t change the systemstate

8. Paths vs. States • So, considering states rather than paths seems to be a trade-off without a clear winner – often at least as un-manageable as complete path exploration • Which is better for error detection? • Hard to say, but for a class of errors (reachability), we can say that complete state coverage would definitely expose them

9. Model Checking • A model checker is a tool for exploring a state space • Basic idea: generate every reachable state of a transition system • Think of states as nodes in a graph • Directed edges mean “from this state, this is a possible next state of the program” • Multiple outgoing edges where there is input/thread scheduling/other nondeterminism

10. Model Checking • Model checking is a huge research field • Clarke, Emerson, and Sifakis received this year’s Turing Award for coming up with the basic idea • Some major subfields: • Hardware and software • Symbolic vs. explicit-state

11. Model Checking • Symbolic model checkers “explore” a state space by manipulating logical expressions representing many states • In explicit-state model checking (part of) the graph is actually enumerated • Especially useful for software, where complex data structures are hard to handle symbolically • Closer to testing Gerard’s SPIN is the most popularexplicit-state model checker

12. Model Checking • We will be looking at one particular kind of explicit-state model checking • Using SPIN • To explore the state spaces of C programs • Other model checkers used for this approach to testing • Java PathFinder 2 (NASA Ames) • Bogor (U Kansas/Nebraska)

13. Model Checking: State-Based Testing • Model-checking by executing the program • Backtracking search for all states Will explore, as a side-effect,many executions and many paths,but the goal is to explore states CFG State already visited!Backtrack and try adifferent operation Done with test!Backtrack and try adifferent operation State already visited!Backtrack and try adifferent operation

14. Many Software Model Checkers BLAST SPIN CRunner CMC CBMC JPF2 SLAM MAGIC Bogor VeriSoft

15. Two Approaches Execution of actual code (dynamic: like testing) BLAST SPIN CRunner CRunner CMC CBMC JPF2 Our focus in this class SLAM MAGIC Bogor VeriSoft Analysis of derived transition system (static: like a complete analysis)

16. SPIN and Model-Driven Verification • SPIN 4.0 introduced model-driven verification • Can embed C code in a PROMELA model • Code is executed as part of a transition during depth-first search • SPIN compiles a PROMELA model into a C program: it’s a model checker generator • Embed C code in transitions by executing the compiled C code – at native speed, with real compiled behavior (including optimizations) • Take advantage of all SPIN features – hashing, multicore exploration, etc.

17. SPIN and Model-Driven Verification • Used to “model check” significant pieces of implementation code at JPL • Very useful for checking rich properties that automated abstraction engines often fail to handle • Functional correctness (differential checking with a reference implementation) • Invariants of rich data structures • Using SPIN for testing C programs

18. SPIN and Model-Driven Verification Execute C codeuntil controlreturns to SPIN • When SPIN backtracks, it uses information on how to restore the state of the C program: • Tracked memory is restored on backtrack • Matched memoryis also used to determine if a state has been visited before • All other memory is not restored on backtrack or stored in state table Push tracked & matchedstate on stack Has state beenvisited before? N Y Backtrack:pop stack &restore tracked &matched state Store matchedstate in statetable

19. SPIN and Model-Driven Verification • Use state that is tracked but not matched to introduce abstractions • E.g., symmetry reduction: a property that does not depend on the order of items in a linked list • Track the original version of the list • Copy and sortthe list after each transition • Match on the sorted list only • SPIN will backtrack when it visits a state with same list contents, even in a different order

20. Testing via Model Checking • We do not expect to cover all states • There are far too many (trillions, at least) • Use abstraction (“consider this state and that state to be the same”) to limit the exploration – may miss errors • Use techniques that have a small probability of missing some states (hashing schemes)

21. Testing via Model Checking • Unsound abstractions: • For example, in the flight flash file systems we’ve checked, the state of the flash device Used page Free page Dirty page Bad block live pages? (0-1) x dirty pages (0-1) x block state (bad, free, current)

22. Testing via Model Checking • Does not guarantee that states (or errors) won’t be missed, but provides a better understanding of which states have been explored, based on the program’s purpose • Manages state space size Used page Free page Dirty page Bad block live pages? (0-1) x dirty pages (0-1) x block state (bad, free, current)

23. Unsound Abstractions • “Abstraction” can mean a lot of things • What I’ll be talking about in this class: • One operation: from a concrete state, c, we can compute an abstract state A(c) • Backtrack according to abstract state matching • Goal: we would like to explore every reachable abstract state of a C program

24. Unsound Abstractions • For some abstractions, the goal is reached if we simply match on abstract state – if  c1, c2 . A(c1) = A(c2) c3 . T(c1, c3)c4 . T(c2, c4) A(c3) = A(c4)

25. Unsound Abstractions • Otherwise: These states abstract the same These, however, do not abstract equivalently,and can only be reached through their parents(and are the only concrete representatives oftheir respective abstract states). Whichever of these concrete states we explorefirst, we will miss one of the children. So do these And these

26. Unsound Abstractions • We usually don’t have soundness for any abstractions that reduce the state-space to a manageable size • If not, unsound in our context (underapproximating abstract successors by abstracting concrete states) • That’s ok • It’s just testing, we don’t expect to be complete

27. Testing via Model Checking • Requires the ability to restore a running program to an earlier execution state • Difficult engineering problem, handled by applying automatic code instrumentation • Rewrite the program to check that the backtracking works properly • As a side-benefit, can simulate flight software resets and hardware faults

28. Simple PROMELA Code int x; int y; active proctype main () { if:: x = 1:: x = 2fi; assert (x == y); } Start simple This model has 7 states 1 2 What are they? State = (PC, x, y) 3 5 7 SPIN’s nondeterministic choice construct Picks any one of the choices that is enabled Not mutuallyexclusive! if:: (x < 10) -> y = 1:: (x < 5) -> y = 3:: (x > 1) -> y = 4fi; How do we guard a choice?

29. Simple PROMELA Code int x; int y; active proctype main () { if:: x = 1:: x = 2fi; if:: y = 1:: y = 2fi; if:: x > y -> x = y:: y > x -> y = x:: else -> skipfi; assert (x == y); } 1 This model has 17 states 2 3 5 What are they? State = (PC, x, y) 7 9 13 Er… 14 15 17 Don’t worry about state-counting toomuch – SPIN has various automaticreductions and atomicity choices thatcan make that difficult

30. Simple PROMELA Code int x; active proctype main () { do :: (x < 10) -> x++ :: break od Only a couple more PROMELAconstructs to learn for building testharnesses: the do loop Like if, except it introducesa loop to the top – break choicecan exit the loop This nondeterministically assigns xa value in the range 0…9

31. Simple PROMELA Code int x; inline pick (var, MAX) var = 0; do :: (var < MAX) -> var++ :: break od inline gives us a macro facility As you can imagine, this is auseful macro for building a test harness!

32. Testing via Model Checking • Basic idea: • We’ll write a test harness in PROMELA • Use SPIN to backtrack and explore all input choices • Call C code we’re testing • Use abstraction to limit the number of states we consider • We can even “trick” SPIN into doing pure random testing!

33. Testing via Model Checking • Rest of class: • We’ll look at the idea of “model-driven verification” that drives all this • I’ll show how to write a simple PROMELA harness to test C code • Thursday: we’ll investigate automatic code instrumentation, some novel coverage-based code strategies, and see how model checking and random testing compare • Download SPIN yourself, and play with it! • http://www.spinroot.com • Examples available on class website