Model Checking Java Programs using Structural Heuristics

1. Model Checking Java Programs using Structural Heuristics Alex Groce Carnegie Mellon University Willem Visser NASA Ames Research Center

2. Model Checking • Explores graph of reachable system states • Checking for local assertions, invariants and general temporal (logic) properties • Symbolic model checking • Explicit-state model checking

3. Java Code Bytecode void add(Object o) { buffer[head] = o; head = (head+1)%size; } Object take() { … tail=(tail+1)%size; return buffer[tail]; } JAVAC JVM 0: iconst_0 1: istore_2 2: goto #39 5: getstatic 8: aload_0 9: iload_2 10: aaload Model Checker Java PathFinder Special JVM

4. Depth-first Search push initial state on Stack while (Stack not empty) s = top(Stack) if s has no more successors pop the Stack else s’ = next successor of s if s’ not already visited mark s’ visited if s’ is a goal state then terminate push s’ on Stack

5. Problems with DFS • Produces lengthy counterexamples • If state-space is too large to fully explore • May expend all resources on a single path when shallow counterexamples exist • Failed runs give little information because states explored may be very “similar”

6. Directed Model Checking • Model checking as a search in a state space • Why not use heuristics to guide the search? • Need to know what we’re looking for • Can we find good heuristics for model checking? • Bug-finding rather than verification

7. Best-first Search priority queue Q = {initial state} while (Q not empty) s = state in Q with lowest f remove s from Q for each successor state s’ of s if s’ not already visited mark s’ visited if s’ is a goal state then terminate f = h(s’) store (s’, f) in Q

8. Two Kinds of Heuristics • Property-specific heuristics • Directed at a specific error • Number of unblocked threads as a measure of distance to deadlock • Static analysis for distance to an assertion check • Focus of most previous work in field

9. Two Kinds of Heuristics • Structural heuristics • Designed to explore the structure of a program in a systematic fashion • But what do we mean by structure?

10. Structural Heuristics • One obvious kind of structure in a program: • Control flow • Reachable control flow rather than just CFG • Motivation for branch coverage metrics used in software testing

11. Branch Coverage • Instrument model checker to calculate branch coverage • Using a simple coverage measure as a heuristic doesn’t work well • Easily falls into local minima (once any branches are taken, every state on that path has “better” coverage) • Doesn’t distinguish between branches explored once and branches explored many times

12. The Branch Counting Heuristic • Count the number of times each branch has been taken • Heuristic value is then: • Branches never before taken get lowest value • Non-branching transitions are next lowest • Otherwise, score is equal to the count (lower values are explored first)

13. Three Searches DFS Branch Counting CFG Each CFG state is a basic block that increments some variable x. ERROR BFS

14. Three Searches DFS Branch Counting CFG BFS

15. Three Searches DFS Branch Counting CFG BFS

16. Three Searches DFS Branch Counting CFG BFS

17. Three Searches DFS Branch Counting CFG Heuristic avoids taking BFS

18. Three Searches DFS Branch Counting CFG BFS

19. Three Searches DFS Branch Counting CFG Expands 15 states BFS Terminates only with depth limit Expands 25 states

20. Experimental Results • DEOS real-time operating system example • This version uses an integer valued counter, without abstraction

21. Results for DEOS All experiments performed on a 1.4GHz Athlon, limiting Java heap size to 512MB, all times are in seconds

22. The Interleaving Heuristic • An important (and very hard to find) class of errors in Java is concurrency errors • What kind of structure could we explore to catch these? • Thread-interdependency

23. The Interleaving Heuristic • Not clear how to heuristically define actual thread-interdependence • So we use an approximation: • Executions in which context is switched more often are given better heuristic values • Explores executions unlikely to appear in testing (JVM/JITs schedule quite differently)

24. The Interleaving Heuristic • Keep track on each path of which threads are executed at each transition • Give lower (better) heuristic score to paths in which the most recently executed thread has been run less frequently • Slightly more complicated in practice, counting live threads

25. Limiting the Queue • With heuristics we are more interested in finding bugs than in verification • So, we apply a technique from heuristic search literature: • Limit the size of the priority queue! • When queue has more than k states in it, remove all but k states with best heuristic values

26. Experimental Results • Dining Philosophers • Comparison to other results: • Godefroid and Khurshid in TACAS ’02 paper apply genetic algorithms to dining philosophers • Best result reported is 17 philosophers, 177 seconds, 50% success rate (on a slower machine) • HSF-SPIN • Not clear how to compare (times not given) • Best result they show is 16 philosophers, and SPIN (using partial order reduction) itself fails with 14 philosophers

27. Experimental Results

28. One Last Heuristic • The choose-free heuristic: • Works only for abstracted Java programs • Rewards transitions that do not involve nondeterminism introduced by the abstraction • Prefers counterexamples that do not result from loss of precision introduced by the abstraction • Structure of abstraction, not program

29. Previous Work • Edelkamp, Lafuente, and Leue • HSF-SPIN: SPIN + heuristic search framework • Bloem, Ravi, and Somenzi • Symbolic Guided Search: BDDs + heuristics • With BDDs heuristics can aid verification • Cobleigh, Clarke, and Osterweil • FLAVERS verification work

30. Conclusions • Structural heuristics: a useful class of heuristics • When model checking is used for debugging, we may not know what kinds of bugs we are hunting • Property-specific heuristics are also useful; approach is complementary, not replacement • Most-blocked can perform as well or better than interleaving in the Remote Agent example, depending on the k limit and search method

31. Future Work • Experiment with other, larger examples • Static analysis for property-specific heuristics • Language for properties/search/heuristics • Discover how heuristics work when symbolic execution is introduced into JPF • Counterexample analysis for “bug causality” • What other kinds of structure can be exploited with heuristics? • Counting occurrences of data values, perhaps