Korat: Automated Testing Based on Java Predicates

1. Korat: Automated Testing Based on Java Predicates C.Boyapaty, S.Khurshid, D.Marinov CS751 Presentation by Radu Stoleru University of Virginia

2. Roadmap • Why do they do it? • Statement of the Problem • State of the Art • What do they do? • How do they do it? • Test Input Generation • Checking Correctness • Results & Evaluation • Questions & Comments University of Virginia

3. Roadmap • Why do they do it? • Statement of the Problem • State of the Art • What do they do? • How do they do it? • Test Input Generation • Checking Correctness • Results & Evaluation • Questions & Comments University of Virginia

4. Why do they do it? • Automated Testing. Why improve testing? • Through manual testing: • significant errors are not found • it takes 30% of development time • automated testing is an industry standard validation • Automated Testing consists of: • automated generation of test cases from specifications • automated execution of test cases • automated validation University of Virginia

5. Why do they do it? • Specification-based testing: • Z specification, UML statechart – no linked data structs • TestEra framework (’01) – new specification language • JML+JUnit (’01) – no test case generation • Static Analysis • Extended Static Checker (’98) – no complex structs • TVLA (’98) – only limited program properties • Software model checking • JavaPathFinder (’00), VeriSoft (’98) – no linked data structs University of Virginia

6. Why do they do it? • Korat: • automated generation of test cases for complex structs • complete evaluation of correctness automatically • generates counter-examples • no new specification language University of Virginia

7. Roadmap • Why do they do it? • Statement of the Problem • State of the Art • What do they do? • How do they do it? • Test Input Generation • Checking Correctness • Results & Evaluation • Questions & Comments University of Virginia

8. What do they do? • use JML for formal specification (class invariants, preconditions, postconditions) • generate test inputs using preconditions • builds Java predicate • builds a skeleton finitization • prunes input state space • generates isomorph-free test cases • evaluate correctness using postconditions • using JML/JUnit University of Virginia

9. Roadmap • Why do they do it? • Statement of the Problem • State of the Art • What do they do? • How do they do it? • Test Input Generation • Checking Correctness • Results & Evaluation • Questions & Comments University of Virginia

10. Example boolean repOk() { if (root == null) return size == 0; Set visited = new HashSet(); visited.add(root); List workList = new LinkedList(); workList.add(root); while (!workList.isEmpty()) { Node current = (Node)workList.removeFirst(); if (current.left != null) { if (!visited.add(current.left)) return false; workList.add(current.left); } if (current.right != null) { if (!visited.add(current.right)) return false; workList.add(current.right); } } if (visited.size() != size) return false; return true;} class BinaryTree { //@ public invariant //@ repOk(); Node root; int size; static class Node { Node left; Node right; } /*@ public normal_behavior @ requires has(n); @ ensures !has(n); @*/ void remove(Node n) { ... } } University of Virginia

11. N0 N1 N2 Input Size • 5 non-isomorphic solutions for 3 nodes: N0 N0 N0 N0 right left left left right right N1 N1 N1 N1 N2 right left right left N2 N2 N2 • (n+1)2n+1 candidates for n nodes (292 for 12 nodes) • how to find them quickly? University of Virginia

12. Search • Korat search algorithm: void koratSearch(Predicate p, Finitization f) { initialize(f); while(hasNextCandidate()) { Object candidate = nextCandidate(); try { if(p.invoke(candidate)) output(candidate); } catch (Throwable t) {} backtrack(); } } • given a predicate and a finitization, candidate inputs are generated • inputs are validated by invoking the predicate on them University of Virginia

13. Finitization • a set of bounds that limits the size of the input • Class Domain := a set of objects from one class {N0, N1, N2} • Field Domain := a set of values a field can take. For Node.left it is {null, N0, N1, N2} • generated automatically by Korat • can be further specialized Finitization finBinaryTree(int n, int min, int max) { Finitization f = new Finitization(BinaryTree.class); ObjSet nodes = f.createObjects(“Node”, n); nodes.add(null); f.set(“root”, nodes); // Field Domain f.set(“size”, new IntSet(min, max)); // Field Domain f.set(“Node.left”, nodes); // Field Domain f.set(“Node.right”, nodes); // Field Domain return f; } University of Virginia

14. State Space • using a finitization, Korat: • allocates a given number of objects • constructs candidate vectors using object fields: • ‘root’, ‘left’, ‘right’: {null, N0, N1, N2} • size: {3} BinaryTree N0 N1 N2 root size left right left right left right N0 left : [N0, 3, N1, N1, null, null, null, null] right N1 N2 University of Virginia

15. Search • for each candidate vector, Korat: • invokes repOk() and monitors the execution • builds a field ordering (list of fields ordered by the accessed time) • if repOk() returns true, output the structure • if repOk() returns false, backtracks on the last accessed field, using the field ordering University of Virginia

16. N0 left right N1 N2 Search • when repOk() returns false, the field ordering is: N0 left [N0, 3, N1, N1, null, null, null, null] right N1 N2 |root, N0.left, N0.right| • backtracking on N0.right, gives the next candidate: (increments the field domain index for the field that is last in the field ordering) [N0, 3, N1, N2, null, null, null, null] University of Virginia

17. Search N0 left [N0, 3, N1, N1, null, null, null, null] right N1 N2 N0 left [N0, 3, N1, N2, null, null, null, null] right N1 N2 • with backtracking, Korat prunes 44 candidates of type: [N0, 3, N1, N1, _, _, _, _] University of Virginia

18. Nonisomorphism N0 N1 • two candidates are isomorphic if:   ;  o, o’  OC,r ;  f  fields(o) ;  p  P . o.f == o’ in C <=> (o).f == (o’) in C’ and o.f == p in C <=> (o).f == p in C’ • isomorphism => state space partitioned • only the lexicographically smallest candidate is generated • it is used to increment field domain indices by more than 1. left left N1 N0 left left N2 N2 University of Virginia

19. N0 left N2 right N1 Nonisomorphism-Algorithm backtracking on a field f (pointer to object of of class cf): class domain: cf {N0, N1, N2} N0 [N0, 3, N2, null, null, null, null, null] left |root, N0.left, N0.right, N2.left, N2.right| N2 N1 [N0, 3, N2, null, null, null, null, N1] (?) University of Virginia

20. Generating Test Cases • to generate test inputs for method m, Korat • builds a class that represents m’s inputs • builds repOk() that checks m’s precondition • generates all inputs that satisfy repOk() class BinaryTree_remove { //@ invariant repOk(); BinaryTree This; BinaryTree.Node n; boolean repOk() { return This.repOk() && This.has(n); } } class BinaryTree { //@ invariant repOk(); ... //@ requires has(n); void remove(Node n) { ... } } University of Virginia

21. Checking Correctness • Korat uses: • JML toolset for generating oracles • JUnit for executing tests and reporting errors • to test a method m, Korat invokes m on each input and test the output using the oracle University of Virginia

22. Roadmap • Why do they do it? • Statement of the Problem • State of the Art • What do they do? • How do they do it? • Test Input Generation • Checking Correctness • Results & Evaluation • Questions & Comments University of Virginia

23. Results & Evaluation University of Virginia

24. Results & Evaluation University of Virginia

25. Roadmap • Why do they do it? • Statement of the Problem • State of the Art • What do they do? • How do they do it? • Test Input Generation • Checking Correctness • Results & Evaluation • Questions & Comments University of Virginia

26. Questions & Comments • non-Java environments? • clear enough explanations for algorithms? • proof for the search algorithm? • paper quality: outstanding / good / bad / awful ? • anything else you want to add? University of Virginia