Test Input Generation with Java PathFinder

1. Test Input Generation with Java PathFinder Author: Willem Visser, Corina S. Pasareanu, and Sarfraz Khurshid (ISSTA 2004) Presented by: Jongbhin Park University of Illinois at Urbana-Champaign Some slides from Willem Visser

2. Test input generation • Test input generation for code that manipulates complex data structures is not easy (Java TreeMap) • Test Input generation technique • Strait model checking • model checking used in a black-box test input generation • model checking used in during white-box test input generation • Contribution: to show how efficient white-box test input generation can be done for code manipulating complex data

3. Red-Black Tree (for case study) Self-balancing Binary Search Trees Java TreeMap Implementation repOk(): conditions (1)-(4) (2) The root is BLACK (4) All paths from a node to its leaves contain the same number of black nodes. (3) If a node is red, then both its children are black. (1) Every node is either red or black

4. Test input generation for treemap – My experience • Statement coverage is just 2% for testing TreeMap library using AgitarOne. Vendor declare “80% code coverage or better (AgitarOne is a commercial tool for automated test case generation tool ) • Test cases for put and remove method generated by AgitarOne public void testPut() throws Throwable { TreeMap treeMap = new TreeMap(); Integer key = new Integer(-100); Object result = treeMap.put(key, key); assertEquals("treeMap.size()", 1, treeMap.size()); assert something…… } public void testRemove() throws Throwable { Object result = new TreeMap().remove(""); assertNull("result", result); } Simple Test Input Just Once

5. Java Code Bytecode void add(Object o) { buffer[head] = o; head = (head+1)%size; } Object take() { … tail=(tail+1)%size; return buffer[tail]; } JAVAC JVM 0: iconst_0 1: istore_2 2: goto #39 5: getstatic 8: aload_0 9: iload_2 10: aaload Model Checker Special JVM Java PathFinder • explicit state model checker for Java bytecode

6. Symbolic Execution • Using symbolic values, instead of actual data • representing the values of program variables as symbolic expressions • As a result, the outputs computed by a program are expressed as a function of the symbolic inputs

7. Symbolic Execution Tree x = 1, y = 0 1 >? 0 x = 1 + 0 = 1 y = 1 – 0 = 1 x = 1 – 1 = 0 0 – 1 >? 0 [Concrete Execution] [Symbolic Execution Tree]

8. Generalized Symbolic Execution • model checker generates and explores “symbolic” execution tree • off-the-shelf decision procedures check path conditions • Lazy Initialization • it initializes the components of the method inputs on an “as needed”, “on demand” basis • Conservative Precondition • For evaluating partially initialized structures • Returning false only if the initialized fields of the input structure violate a constraint in the precondition

9. Case Study • What to test: • Class: java.util.TeeMap • Method: Some helper method for put and remove • repOK: class invariant • Checks if the input is red-black tree • Returns true for valid structures, otherwise false • assume this definition can be encoded as a Java predicate, called repOk()

10. Test Input Generation for Complex Structures • Test adequacy measured by branch coverage • Model Checking and Symbolic Execution • Black-box over repOk() • White-box over implementation code

11. Testing by Model Checking • Encode environment of code under test as a nondeterministic program and model check • Testing for TreeMap methods publicstaticintN=5; publicstaticTreeMapt=newTreeMap(); publicstaticInteger[]elems; static{elems=newInteger[N];…} publicstaticvoidmain(String[]args){ while(true){ if(Verify.randomBool()) t.put(elems[Verify.random(N-1)]); else t.remove(elems[Verify.random(N-1)]); } } analyzing all sequences of put and remove calls on a set with maximally N elements

12. Black-box TIG Symbolic Execution • Symbolic execution of repOk() • Generate new structures only when repOk() returns true • Limit the size of the structures generated • Only correct structures will be generated • repOk() returns true after all nodes in the tree have been visited, hence they must all be concrete • symbolic (partial) structures can fail repOk()

13. White-box TIGSymbolic Execution • Consider code coverage criterion when generating test inputs • Use conservative repOk as a precondition during symbolic execution of source code • Use repOk to concretize input

14. repOk() x 2abstract and concrete During Lazy Initialization check conservative repOk Symbolic Execution of Code Concretize inputs by symbolic execution of (Concrete ) repOk() over symbolic structures - as with Black-box TIG- When coverage is achieved, solve the symbolic constraints to create concrete inputs Conservative repOk() : Symbolic Structure {true,false,don’t know} Concrete repOk() : Symbolic Structure Concrete Structure

15. Conservative repOk() • Eliminate symbolic structures that cannot be converted to a concrete structure that satisfy repOk() • Can accept symbolic structures that could lead to illegal concrete structures, i.e. it is conservative • Conservative repOk() can return TRUE, FALSE or Don’t Know • if FALSE, eliminate structure • if TRUE or Don’t Know, continue ... • Example: (2) Red nodes have only black children. TRUE Don’t Know FALSE

16. White-box TIG: cover branches in deleteEntry(Entry p) /* precondition: p.repOk() */ privatevoiddeleteEntry(Entryp){ if(p.left!=null&&p.right!=null){ Entrys=successor(p); swapPosition(s,p); } Entryreplacement=(p.left!=null?p.left:p.right); if(replacement!=null){ replacement.parent=p.parent; if(p.parent==null) root=replacement; elseif(p==p.parent.left){ p.parent.left=replacement; } else p.parent.right=replacement; p.left=p.right=p.parent=null; if(p.color==BLACK) fixAfterDeletion(replacement); ...

17. Symbolic structure before executing branch if(p.left!=null&&p.right!=null){ ... Symbolic structure(s) that cover the branch This structure “passes” the conservative repOk() Concretize The symbolic structure is used as input to repOk() and lazily executed to obtain the concrete structure Concrete structure that will cover the code Symbolic Execution for white-box TIG

18. Experimental Results • Three helper method (called by put or remove method) • deleteEntry(del)-Optimal BC(branch coverage): 86% • fixAfterDeletetion(fixD)-Optimal BC: 100% • fixAfterInsertion(fixI): 88% • All results were obtained using the JPF model checker

19. Model Checking

20. Black Box

21. White-box

22. Conclusions Showed how to use symbolic execution and method preconditions to do efficient black- and white-box test input generation for code manipulating complex data • Current limitations • Code instrumentation for symbolic execution by hand • Cannot detect dead-code • Need to deal with loops

23. JPF Status • developed at the Robust Software Engineering Group at NASA Ames Research Center • currently in it’s fourth development cycle • v1: Spin/Promela translator - 1999 • v2: backtrackable, state matching JVM - 2000 • v3: extension infrastructure (listeners, MJI) - 2004 • v4: symbolic execution, choice generators - 4Q 2005 • open sourced since 04/2005 under NOSA 1.3 license:<javapathfinder.sourceforge.net> • it’s a first: no NASA system development hosted on public site before • 11100 downloads since publication 04/2005

24. Getting and Running JPF • Getting JPF • svn co https://svn.sourceforge.net/svnroot/javapathfinder • Compiling JPF • Running JPF : generally speaking: like a VM (“java” replacement) • The Eclipse • Using Subclipse • Or import downloaded JPF

25. Sample Output (TreeMap)