Going from Concrete to Symbolic Model Checking via Predicate Abstraction

1. Going from Concrete to Symbolic Model Checking via Predicate Abstraction Willem Visser Corina Pasareanu and Radek Pelanek Automated Software Engineering GroupNASA Ames Research Center

2. Overview • Abstraction • Classic over-approximation based • Counter-example based refinement • Under-approximation based • Refinement based on abstraction’s exactness • Lightweight framework for testing • Test generation environment built around JPF with symbolic execution • Measure predicate coverage • Evaluate against other test-case generation methods • Java Container classes

3. 1: p = T; 2: while (p) 3: p = !p ? F : T | F; 4: assert false; 1: x = 2; 2: while (x>0) 3: x = x - 1; 4: assert false; Abstraction Mapping p = (x>0) Predicate Abstraction

4. wp(a’,T) a a wp(a’,T) T T a’ a’ a → a’ a → a’ Abstraction Mapping For a,a’ in 2{preds}: if wp(a’,T) /\ a add transition a → a’ may transition must transition

5. 1: p = T; 2: while (p) 3:p = !p ? F : T | F; 4: assert false; 1: x = 2; 2: while (x>0) 3:x = x - 1; 4: assert false; Abstraction Mapping p = (x>0) Example Abstraction {x – 1 > 0} x = x – 1 {p} {x – 1 <= 0} x = x – 1 {!p} wp(!p,x=x-1) /\ p add p → !p wp(p,x=x-1) /\ p add p → p wp(!p,x=x-1) /\ !p add !p → !p wp(p,x=x-1) /\ !p !p → wp(!p,x=x-1) !p→ !p is must trans

6. X>0 X>1 X>0 X>0 X<=1 X>0 X>1 X>0 X<=1 X>0 X<=0 X<=0 X<=1 Refinement 1: p = T; 2: while (p) 3: p = !p ? F : T | F; 4: assert false; Infeasible Counter Example 1,2,3(F),2,4 1: x=2 {x>0}; 2: x=2 {x>0}; 3: x=1 {x<=0} {x > 1}x = x -1 {x > 0} may must

7. Let’s Go Outside the Box • Rather than over-approximate and refine, we under-approximate and refine • Clearly complements existing techniques • If we restrict ourselves only to feasible behaviors when under-approximating then all safety property violations will be preserved • Build on top of classic explicit-state model checking infrastructure

8. Classic Explicit-State Search PROCEDURE dfs() { s = top(Stack); FOR all transitions t enabled in s DO s' = successor(s) after executing t; IF s' NOT IN VisitedStates THEN Enter s' into VisitedStates; Push s' onto Stack; dfs(); END END; Pop s from Stack; } INIT { Enter s0 into VisitedStates; Push s0 onto Stack; dfs(); }

9. Explicit-State (1-step) αSearch PROCEDURE dfs() { s = top(Stack); FOR all transitions t enabled in s DO s' = successor(s) after executing t; IF α(s‘) NOT IN VisitedStates THEN Enter α(s‘) into VisitedStates; Push s' onto Stack; dfs(); END END; Pop s from Stack; } INIT { Enter α(s0) into VisitedStates; Push s0 onto Stack; dfs(); }

10. 1,p 2,p 3,p αSearch Map concrete states to abstract states for state storing 1: x = 2; 2: while (x>0) 3: x = x - 1; 4: assert false; Abstraction Mapping p = (x>0) Under-approximation of the behaviors Always traverse only feasible paths

11. A,0 A,p A,p B,1 C,0 B,p C,p B,p C,p D,1 D,0 D,p D,p E,2 E,1 E,!p E,p Must Transitionsp = (x < 2) May Transitionsp = (x < 2) Concrete, May & Must Concrete

12. A,0 A,p B,1 C,0 B,p C,p D,1 D,0 D,p E,2 E,1 E,!p Transition not “exact” Concrete & αSearch A,0 B,1 C,0 Abstraction Search p = (x<2) D,1 D,0 E,2

13. A,p,q A,0 A,p,!q A,p,q B,1 C,0 After Refinement Step p = (x<2); q = x < 1 A,p,!q A,p,q D,1 D,0 A,!p,!q A,p,!q E,2 E,1 Refinement & αSearch A,0 B,1 C,0 D,1 D,0 E,2 E,1

14. 1,p 2,p 3,p Example 1: x = 2; 2: while (x>0) 3: x = x - 1; 4: assert false; Abstraction Mapping p = (x>0)

15. 1,p Only 1 DP call 2,p {x – 1 > 0} wp(p,x=x-1) x = x – 1 3,p {x > 0} Refinement Check if the induced abstract transition is a must transition? If not, add new predicates • Add x > 1 to abstraction predicates and repeat search • Globally for all transitions • Locally only for the transition (location) it refines

16. Predicate Abstraction αSearch … … • Showing property holds • Over-approximation based • Counter-example driven refinement • Expensive computation to calculate abstraction • Finding defects • Under-approximation based • Abstraction driven refinement • Trivial computation to calculate abstraction mapping

17. αSearch tries to compute a finite reachable bisimulation quotient this is only possible if a finite reachable bisimulation quotient exists Issue unreachable reachable wp(p,T) T p if new predicates are infinitely required to refine the unreachable area the algorithm will not terminate

18. y >= 0 x+y >= 0 2x+y >= 0 y >= 0 x+y >= 0 y >= 0 p p,q p,q,r … p p,q p,q,r p p,q p,q,r Example x = 0; y = 0; while (y >= 0) y = x + y; The refinement only refines the unreachable state space!

19. BFS 1st iteration 18 concrete states and 12 abstract x+1 <= y, x <= y+1 and y >= 0 2nd iteration 26 concrete state and 19 abstract x+2<=y, y>=1 and x <= 1 3rd iteration 44 concrete states and 32 abstract y <= 1, x <= 0 and y >= 2 4th iteration 48 concrete, 36 abstract DFS 1st iteration 14 concrete states and 20 abstract x+1 <= y, x <= y+1, y >= 0 and x <= 0 2nd iteration 29 concrete state and 21 abstract x+2<=y, y<= 0, x <= -1 and x <= 1 3rd iteration 45 concrete states and 33 abstract Modified Bakery while true { x = y; x = x + 1; wait (x<=y); x = 0; } while true { y = x; y = y + 1; wait (y<x); y = 0; } Search Order Matters!!

20. Symbolic Execution and αSearch • Current implementation is for a simple input language • oCaml using Simplify as a decision procedure • We would like to integrate the technique in Java Pathfinder (JPF) that supports symbolic execution (using the Omega Library) • To allow application to programs with complex data structures (objects)

21. From Concrete to Symbolic X=1, Y = 0 X > Y Concrete Behavior Symbolic Behavior

22. Possible Approach • Execute the concrete program on valid inputs • Collect all predicates in path condition • Solve constraints over all combinations of these predicates • Use results as inputs for step 1 • When no new predicates are found, or, if an error is found, terminate

23. x > 0 && y < 10 y < 5 end Solve Constraints p1 = x > 0 && y < 10 p2 = y < 5 p1,!p2 → method(1,6) !p1,p2 → method(-1,1) !p1,!p2 → method(-1,6) Example method(1,1) + {true} public static void method(int x, int y) { if ((x > 0) && (y < 10)) { if (y < 5) { … } else { … } } else { if (x > 0) { … } else { … } } } method(1,1) + {p1,p2}

24. p1 = x > 0 && y < 10 !p2 = y < 5 Example (2) method(1,6) + {p1,!p2} public static void method(int x, int y) { if ((x > 0) && (y < 10)) { if (y < 5) { … } else { … } } else { if (x > 0) { … } else { … } } } x > 0 && y < 10 y < 5 end method(1,6) + {p1,!p2}

25. !p1 = x > 0 && y < 10 !p3 = x > 0 Example (4) method(-1,1) + {!p1,p2} public static void method(int x, int y) { if ((x > 0) && (y < 10)) { if (y < 5) { … } else { … } } else { if (x > 0) { … } else { … } } } x > 0 && y < 10 x > 0 end Solve Constraints !p1,p3 → method(1,11) method(-1,1) + {!p1,p2,!p3}

26. !p1 = x > 0 && y < 10 p3 = y < 5 Example (3) method(1,11) + {!p1,p3} public static void method(int x, int y) { if ((x > 0) && (y < 10)) { if (y < 5) { … } else { … } } else { if (x > 0) { … } else { … } } } x > 0 && y < 10 x > 0 end method(1,11) + {!p1,p3}

27. End of Part One • Showed under-approximation based search with refinement • Backward weakest precondition based • Forward symbolic execution based • Part Two • Rather than automated refinement we use user-provided abstractions • Motivation is to generate test-cases to achieve high behavioral coverage for Java container classes

28. Explicit-State (1-step) αSearch PROCEDURE dfs() { s = top(Stack); FOR all transitions t enabled in s DO s' = successor(s) after executing t; IF α(s‘) NOT IN VisitedStates THEN Enter α(s‘) into VisitedStates; Push s' onto Stack; dfs(); END END; Pop s from Stack; } INIT { Enter α(s0) into VisitedStates; Push s0 onto Stack; dfs(); }

29. General Idea SUT ENV (m,n) m is the seq. length of API calls & n is the number of values used in the parameters of the calls API … put(v) del(v) Evaluate different techniques for selecting test-cases from ENV(m,n) to obtain maximum coverage

30. Predicate Coverage Cover all combinations of a given set of predicates at each branch in the code Red-Black Tree Predicates root = null, e.left = null, e.right = null, e.parent = null, e.color = BLACK

31. Techniques Considered • Random selection • Classic model checking • State matching on complete state • Abstraction search • State matching on abstract (partial) state • Symbolic Execution • Complete matching using subsumption checks • Abstract matching

32. Framework SUT with minor instrumentation ENV Coverage Manager Abstraction Mapping + State Storage TestListener JPF

33. Concrete Test case number 7 for '32,L-R-P+RED': put(1);put(0);remove(0); Symbolic Test case number 7 for '32,L-R-P+RED': X2(0) == X1(0) && X2(0) < X0(1) && X1(0) < X0(1) put(X0);put(X1);remove(X2); Path Condition with solutions Symbolic TC Sample Output Branch Number Predicate Values Unique ID for the test Test case number 77 for '15,L+R+P-REDroot': put(0);put(4);put(5);put(1);put(2);put(3);remove(4); Test-case to achieve above coverage

34. Environment Skeleton M : sequence length N : parameter values A : abstraction used for (int i = 0; i < M; i++) { int x = Verify.random(N - 1); switch (Verify.random(1)) { case 0: put(x); break; case 1: remove(x); break; } } Verify.ignoreIf(checkAbstractState(A));

35. Symbolic Environment Skeleton M : sequence length A : abstraction used for (int i = 0; i < M; i++) { SymbolicInteger x = new SymbolicInteger(“X“+i); switch (Verify.random(1)) { case 0: put(x); break; case 1: remove(x); break; } } Verify.ignoreIf(checkAbstractState(A));

36. Abstraction Search • Map state to an abstract version and backtrack if the abstract state was seen before, i.e. discard test-case • Mapping can be lossy or not • Abstraction mappings can be created by the user/tester • Default abstraction mappings are provided

37. Default Mappings • Structure of the heap of the program • e.g. structure of the containers • Structure augmented with non-data fields • Structure augmented with symbolic constraints on the data in the structure • This requires checking constraint subsumption

38. Linearization Comparing Structures 1 1 1 2 3 -1 -1 4 -1 -1 5 -1 -1 1 2 3 -1 -1 4 -1 -1 5 -1 -1 2 5 2 5 3 4 3 4 1 1 1 2 3 -1 -1 4 -1 -1 5 -1 -1 1 2 3 -1 -1 4 -1 5 -1 -1 -1 2 2 5 3 4 3 4 5

39. Linearization + Mapping 1b 2b 3r -1 -1 4r -1 -1 5b -1 -1 1b 2r 3r -1 -1 4r -1 -1 5r -1 -1 1 1 2 5 2 5 3 4 3 4 Linearization takes a mapping object as parameter to indicate how each node in the heap should be linearized. In the example above each node gets, besides the unique identifier, a mapping of “r” if the original structure had a red node and “b” if the original structure had a black node in that position. If we also added the key values for each node the linearization might have looked something like: 1b6 2b4 3r3 -1 -1 4r5 -1 -1 5b7 -1 -1

40. Symbolic Execution Symbolic State x1 x1 > x2 & x2 > x3 & x2 < x4 & x5 > x1 + x2 x5 x3 x4 Symbolic Constraints Shape

41. Subsumption Checking x1 x1 > x2 & x2 > x3 & x2 < x4 & x5 > x1 + x2 x5 x3 x4 x1 x1 > x2 & x2 > x3 & x2 < x4 & x5 > x1 + x2 x5 x3 x4 If only it was this simple!

42. Getting Ready for CheckingExistential Elimination s1 x1 PC s1 < s2 & s4 > s3 & s4 < s1 & s4 < s5 & s7 < s2 & s7 > s1 + s4 x2 x5 s2 s3 x3 x4 s5  s1,s2,s3,s4,s5 such that x1 = s1 & x2 = s4 & x3 = s3 & x4 = s5 & x5 = s2 & PC x1 > x2 & x2 > x3 & x2 < x4 & x5 > x1

43. Bidirectional Subsumption Checking • If new => old • backtrack • If old => new • new is more general than old • replace old with new • to increase chances of getting a match in the future • Continue on path from new, i.e. don’t backtrack • Ultimately for each shape we want to use disjunction of constraints • Small technicality prevents us – bug in omega lib

44. Evaluation • Red-Black Trees • Out of Memory runs are not reported • Breadth-first Search unless stated • Sequence Length = Values for the non-symbolic searches • First compare under Branch Coverage

45. Exhaustive TechniquesBranch Coverage Optimal Branch Coverage is 39

46. Under-Approximation TechniquesBranch Coverage Optimal Branch Coverage is 39

47. Exhaustive TechniquesPredicate Coverage Optimal Predicate Coverage is 106

48. Under-Approximation TechniquesPredicate Coverage Optimal Predicate Coverage is 106

49. Observations • For a simple coverage such as branch coverage, all the techniques work well, including the exhaustive ones • But making the coverage more “behavioral”, even by a small increment, kills off the exhaustive techniques

50. Observations • Full Blown Model Checking doesn’t work here • Its close cousin, that only looks at the relevant state at the relevant time, scales much better • Branch - full coverage after: • MC: 536s & 584Mb • Complete: 10s & 17Mb • Predicate – best coverage after: • MC: 79 covered with 543s & 309Mb • Complete: 95 covered with 350s & 228Mb