Symbolic Executing Program Paths to Adaptively Generate Test Cases

1. Symbolic Executing Program Paths to Adaptively Generate Test Cases Liu Shi & Zhang Chun {liushi07, zhangchun08}@sei.pku.edu.cn

2. Agenda • Introduction & Motivation • Our Approach • Path selection • Symbolic Execution to get path info • Test case generation • Experimental study • Project related • Tools • Teamwork

3. Introduction • Automated software testing • Random testing VS. Systematic testing • Effective, covering, minimal • ART- Adaptive Random Testing • Evenly spread over the input domain • Simple, effective • Symbolic Execution/Model checking • Program analysis, exhaustive techniques • Find subtle error

4. Motivation • Random: low coverage, redundant • Systematic: space explosion problem • Combine random and systematic • Path Condition guided random test case generation • Using systematic techniques to improve the quality of randomly generated test cases • Tradeoff between Cost and effectiveness (Covering)

5. Agenda • Introduction & Motivation • Our Approach • Path selection • Symbolic Execution to get path info • Test case generation • Experimental study • Project related • Tools • Teamwork

6. Our Approach • First, control flow analysis to select several paths • Get a sequence of paths representing the method under test • Second, symbolic execution on “straight line program” • Get a sequence of symbolic constraints for each path • Finally, adaptively generate test cases using the results of previous 2 steps

7. Path Selection ENTRY • Represent all paths(covering) • Path Coverage is good but too large • Using Pair-Wise coverage • For every possible pair of split nodes • Steps: • Node Clustering • Pair-Wise assignment • Path Selection 2 3 12 4 5 13 14 6 7 8 9 10 11 15 EXIT

8. Node Clustering • Post dominator calculation • If X appears on every path from Y to Exit, then X post dominate Y • Backwards analysis • Intersection • IN[B] = OUT[B] ∪ B • Focus on the split node • Do clustering

9. Node Clustering ENTRY • Focus on the split node • 2:[2] • 3:[3, 6, 9] • 6:[6, 9] • 9:[9] • 12:[12] • Clustering • [2] • [3, 6, 9] • [12] 2 3 12 4 5 13 14 6 7 8 9 10 11 15 EXIT

10. Pair-Wise assignment ENTRY • For node [2] • {2=true, 2=false} • For nodes [3, 6, 9] • {3=true, 6=true, 9=false} • {3=true, 6=false, 9=true} • {3=false, 6=true, 9=true} • {3=false, 6=false, 9=false} • For node [12] • {12=true, 12=false} 2 3 12 4 5 13 14 6 7 8 9 10 11 15 EXIT

11. Path Selection • Recursively traverse the Control Flow Graph • When reaching split node, apply the Pair-Wise coverage assignment • When reaching the EXIT node, print a path • For loops, set a threshold • Results • [0, 2, 12, 13, 15] • [0, 2, 12, 14, 15] • [0, 2, 3, 5, 6, 8, 9, 10, 15] • [0, 2, 3, 5, 6, 7, 9, 11, 15] • [0, 2, 3, 4, 6, 8, 9, 11, 15] • [0, 2, 3, 4, 6, 7, 9, 10, 15]

12. Symbolic Execution • Use a symbol table to store variable-expression pair • This part is actually an instruction visitor over the AST • Refresh the symbol table for different kinds of instructions • E.g. for GETFIELD_I instruction GETFIELD_I% T3 int,R1 DemoClass,.gender,T-1 <g> Result : “T3 = R1.gender“ • For method invocation • Whether the callee is loaded

13. Symbolic Execution • Sample result Path: [0, 2, 3, 5, 6, 7, 9, 10, 14] …… …… {…… Return=(((R1.getBasic() * 1.1) + 100.0) + (50 * (R1.gra.ordinal() - test.DemoClass.grade.one.ordinal()))) ……} BR: R1.t == test.DemoClass.type.student BR: R1.sType != test.DemoClass.studentType.ms BR: R1.gen == test.DemoClass.gender.male BR: R1.gra != test.DemoClass.grade.overdue

14. Test case generation • For primitive type: • Randomly data generation within restricted input domain • Keeps a pool • Fixed: primitive values like ASCII char set, -1, 0… • Add “critical point” values of programs to the pool • 25% will be selected from the pool • For complex type: • Randomly Constructor selection • Recursively object construction • Randomly method sequence generation

15. Test case “distance” calculation • ART need distance calculation to select the “next” one from candidates • We map a test case to a single path from the sleted ones • Calculate the path distance • The difference between basic block sequences • We use Java reflection mechanism for test case execution

16. Agenda • Introduction & Motivation • Our Approach • Path selection • Symbolic Execution to get path info • Test case generation • Experimental study • Project related • Tools • Project Management

17. Experimental study • Quality of test inputs • Path coverage ratio • Statement coverage ratio • Compare with • ART Adaptive Random Testing by Localization • URT Undirected Random Testing We conduct the comparison on the “TriangleType” procedure

18. Statement Coverage Ratio

19. Path Coverage Ratio

20. Agenda • Introduction & Motivation • Our Approach • Path selection • Symbolic Execution to get path info • Test case generation • Experimental study • Project related • Tools • Project Management

21. Project related • Tool • JoeQ for Java class file static analysis • http://joeq.sourceforge.net/ • Control Flow Graph Analysis • Quad instruction representation and manipulation • Eclipse as development environment • Teamwork • 2-members group • Division/Responsibility, Discussion • Google Code • http://code.google.com/p/sefrt/

22. Q&A

23. Future work • Path selection • Unfeasible path reduce • Symbolic execution • User input value(variable) • Evaluation • With Conclic, JPF-se, ARTOO, Randoop • More complex

24. Related work • JPF-SE • Pure systematic • CUTE & jCUTE • More Systematic • Using random inputs to init the search • Randoop • Primarily random • Feedback directed • Focus on method sequences