1 / 37

Korat: Automated Testing Based on Java Predicates

Korat: Automated Testing Based on Java Predicates. Chandrasekhar Boyapati 1 , Sarfraz Khurshid 2 , and Darko Marinov 3. 1 University of Michigan Ann Arbor 2 University of Texas at Austin 3 University of Illinois at Urbana-Champaign. Examples of Structurally Complex Data. red-black tree.

salali
Download Presentation

Korat: Automated Testing Based on Java Predicates

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Korat: Automated Testing Based on Java Predicates Chandrasekhar Boyapati1, Sarfraz Khurshid2, and Darko Marinov3 1University of Michigan Ann Arbor 2University of Texas at Austin 3University of Illinois at Urbana-Champaign

  2. Examples of Structurally Complex Data red-black tree XML document 1 <library> <book year=2001> <title>T1</title> <author>A1</author> </book> <book year=2002> <title>T2</title> <author>A2</author> </book> <book year=2003> <title>T3</title> <author>A3</author> </book></library> /library/book[@year<2003]/title 0 3 2

  3. Testing Setup pass 1 0 3 2 fail 2 2 0 0 3 3 3 0 inputs outputs • examples of code under test • abstract data type • input/output: data structure • XML processing program • input/output: XML document testing oracle testgenerator code

  4. Manual Test Generation pass 1 0 3 2 fail 2 2 0 0 3 3 3 0 inputs outputs • drawbacks of manual generation • labor-intensive and expensive • vulnerable to systematic bias that eliminates certain kinds of inputs • testgenerator • manual testing oracle code

  5. Automated Test Generation pass 1 0 3 2 fail 2 2 0 0 3 3 3 0 inputs outputs • challenges of automated test generation • describing test inputs • (efficient) test generation • checking output • testgenerator • automated testing oracle code

  6. Running Example B0: 3 root N0: 2 right left N1: 1 N2: 3 class BST { Node root; int size; static class Node { Node left, right; int value; } } void remove(int i) { … } …

  7. Example Valid Inputs B0: 3 B0: 3 B0: 3 B0: 3 B0: 3 N0: 1 N0: 2 root root root root root right right left N0: 1 N0: 3 N0: 3 N1: 1 N2: 3 N1: 3 right left left left N1: 2 N1: 1 N1: 2 N2: 2 right right left N2: 3 N2: 2 N2: 1 • trees with exactly 3 nodes

  8. Example Invalid Inputs B0: 2 B0: 3 N0: 3 N0: 2 root root right right left left N1: 1 N1: 1 N2: 2 N2: 3 B0: 3 root N0: 2 right left N1: 1 N2: 3 • object graphs violating some validity property

  9. Validity Properties for Example underlying graph is a tree(no sharing between subtrees) correct number of nodes reachable from root node values orderedfor binary search B0: 3 N0: 2 root right left N1: 1 N2: 3

  10. Describing Validity Properties • Korat uses a predicate written in standard implementation language (Java, C#…) • takes an input that can be valid or invalid • returns a boolean indicating validity • advantages • familiar language • existing development tools • predicates often already present • challenge: generate tests from predicates

  11. Example Predicate boolean repOk(BST t) { return isTree(t) && hasCorrectSize(t) && isOrdered(t); } boolean isTree(BST t) { if (t.root == null) returntrue; // empty tree Set visited = new HashSet(); visited.add(t.root); List workList = new LinkedList(); workList.add(t.root); while (!workList.isEmpty()) { Node current = (Node)workList.removeFirst(); if (current.left != null) { if (!visited.add(current.left)) return false; // sharing workList.add(current.left); } if (current.right != null) { if (!visited.add(current.right)) return false; // sharing workList.add(current.right); } } return true; // no sharing }

  12. Input Space B0: 2 B0: 3 B0: 3 B0: 3 B0: 3 B0: 3 N0: 1 N0: 2 N0: 3 root root right right left left N0: 1 N0: 3 N0: 3 root root root root N1: 1 N1: 1 N2: 2 N2: 3 N1: 3 B0: 1 B0: 3 right N0: 1 root B0: 3 left B0: 1 B0: 1 N1: 2 N1: 1 N1: 2 root right N2: 2 left left N0: 2 N0: 1 root left root left right B0: 0 N0: 2 B0: 1 N0: 1 root N2: 3 N2: 2 N2: 1 left right right right N0: 1 left root N1: 1 N2: 3 right right left N1: 1 N2: 3 • all possible object graphs with a BST root(obeying type declarations)

  13. Bounded-Exhaustive Generation • finitization bounds input space • number of objects • values of fields • generate all valid inputs up to given bound • eliminates systematic bias • finds all errors detectable within bound • avoid isomorphic inputs • reduces the number of inputs • preserves capability to find all errors

  14. Example Finitization • specifies number of objects for each class • 1 object for BST: { B0 } • 3 objects for Node: { N0, N1, N2 } • specifies set of values for each field • sets consist of objects and literals • for root, left, right: { null, N0, N1, N2 } • for value: { 1, 2, 3 } • for size: { 3 }

  15. Example Input Space B0 N0 N2 N1 root size left right value left right value left right value N0 3 N1 N1 2 null null 1 null null 3 B0 N0 N2 N1 root size left right value left right value left right value null null null null null null null 1 1 1 N0 N0 N0 N0 N0 N0 N0 2 2 2 N1 N1 N1 N1 N1 N1 N1 3 3 3 N2 N2 N2 N2 N2 N2 N2 • 1 BST object, 3 Node objects: total 11 fields 3 4 * 1 * (4 * 4 * 3)3 > 218 inputs, only 5 valid

  16. Example Input B0 N0 N2 N1 root size left right value left right value left right value N0 3 N1 N1 2 null null 1 null null 3 B0: 3 root N0: 2 right left N1: 1 N2: 3 • each input is a valuation of fields

  17. Generation Problem • given • predicate • finitization • generate • all nonisomorphic valid inputs within finitization • simple “solution” • enumerate entire input space • run predicate on each input • generate input if predicate returns true • infeasible for sparse input spaces (#valid << #total) • must reason about behavior of predicate

  18. Example Execution B0: 3 root N0: 2 right left N1: 1 N2: 3 [ B0.root, N0.left, N0.right ] [ B0.root ] [ B0.root, N0.left ] boolean repOk(BST t) { return isTree(t) && …; } boolean isTree(BST t) { if (t.root == null) return true; Set visited = new HashSet(); visited.add(t.root); List workList = new LinkedList(); workList.add(t.root); while (!workList.isEmpty()) { Node current = (Node)workList.removeFirst(); if (current.left != null) { if (!visited.add(current.left)) return false; workList.add(current.left); } if (current.right != null) { if (!visited.add(current.right)) return false; workList.add(current.right); } } return true; } field accesses: [ ]

  19. Failed Execution B0 B0 N0 N0 N2 N2 N1 N1 root root size size left left right right value value left left right right value value left left right right value value N0 N0 3 - N1 N1 N1 N1 2 - null - null - 1 - null - null - 3 - • failed after few accesses for a concrete input • would fail for all inputs with partial valuation 1 * 3 * 4 * 4 * 3 * 4 * 4 * 3 > 212

  20. Key Idea • observe execution of predicate • record field accesses • prune large chunks of input space for each failed execution • use backtracking to efficiently enumerate valid inputs

  21. Search Step B0 B0 N0 N0 N2 N2 N1 N1 root root size size left left right right value value left left right right value value left left right right value value N0 N0 3 3 N1 N1 N2 N1 2 2 null null null null 1 1 null null null null 3 3 • backtracking on [ B0.root, N0.left, N0.right ] • produces next candidate input

  22. Next Input B0 N0 N2 N1 root size left right value left right value left right value N0 3 N1 N2 2 null null 1 null null 3 • execution returns true, generate this input B0: 3 root N0: 2 right left N1: 1 N2: 3

  23. Korat repOk finitization • solver for executable predicates • Korat systematically searches input space • encodes inputs as vectors of integers • creates candidate object-graph inputs • monitors execution of predicate • prunes search by backtracking on field accesses • avoids isomorphic inputs allnonisomorphicvalid inputs Korat

  24. Isomorphic Inputs B0: 3 N1: 2 root right left N2: 1 N0: 3 B0: 3 root N0: 2 right left N1: 1 N2: 3 • equivalent for all code and all properties • example for trees: permutation of nodes • n! isomorphic trees => generation infeasible • removing isomorphic inputs from test suites • significantly reduces the number of tests • does not reduce quality 

  25. Isomorphism Definition • inputs: rooted and labeled (object) graphs • edges have fields • nodes have object identities • isomorphism with respect to object identitites • definition: inputs I and I’ consisting of objects from a set O are isomorphic iff: O->O. o,o’ reachable in I. ffields(o). o.f = o’ in I => (o).f = (o’) in I’ and o.f = p in I => (o).f = p in I’

  26. Example Nonisomorphic Trees B0: 3 B0: 3 root root   B0: 3 N0: 2 N0: 2 root right left right left N0: 3 N1: 1 N2: 3 N1: 1 N2: 3 right left N1: 1 N2: 2 • different edges or primitives, not only nodes

  27. Nonisomorphic Generation • simple “solution” • generate all inputs • filter out isomorphic inputs • Korat does not require filtering • generate only one (candidate) input from each isomorphism class • only the lexicographically smallest input • search increments some field values for >1

  28. Correctness • Korat’s input generation • sound • no invalid input • complete • at least one valid input from each isomorphism class • optimal • at most one (valid) input from each isomorphism class

  29. Example Specification class BST { Node root; int size; static class Node { Node left, right; int value; } //@ invariant repOk(); //@ precondition true; //@ postcondition !contains(i) && …; void remove(int i) { … } boolean contains(int i) { … } … }

  30. Testing Scenario root B0: 3 invariantrepOk() pass right N0: 1 postcondition!contains(i) && … preconditiontrue fail right N1: 2 N2: 3 root B0: 3 right N0: 1 N1: 2 , 3 and and Korat remove finitization

  31. Korat Structure Generation linked data structures and array-based data structures

  32. Korat Structure Generation very large input spaces

  33. Korat Structure Generation pruning based on filed accesses very effective

  34. Korat Structure Generation correct number of nonisomorphic structures (Sloane’s)

  35. Korat Structure Generation 800Mhz Pentium III Sun’s Java 2 SDK 1.3.1 JVM

  36. Korat Method Testing several operations for each data structure

  37. Conclusions • Korat automates specification-based testing • uses method precondition to generate all nonisomorphic test inputs • prunes search space using field accesses • invokes the method on each input and uses method postcondition as a test oracle • Korat prototype uses JML specifications • Korat efficiently generates complex data structures including some from JCF

More Related