Kobi Inkumsah Tao Xie Dept. of Computer Science North Carolina State University,

1. Improving Structural Testing of Object-Oriented Programs via Integrating Evolutionary Testing and Symbolic Execution Kobi Inkumsah Tao Xie Dept. of Computer Science North Carolina State University, Raleigh, NC

2. Motivation • Generating OO unit tests involves two tasks: • Task 1: generating relevant method sequences • Task 2: generating relevant method arguments • Symbolic execution good at Task 2 but not Task 1 • bounded exhaustive sequence but low bound • Evolutionary testing good at Task 1 but not Task 2 • concrete values can be evolved but largely random • Contribution: Integrating the two techniques to address their respective weaknesses

3. Evolutionary Testing – Chromosome • Encode a method sequence with argument values as a chromosome: Method sequence Method arguments E.g., eToc [Tonella 04]

4. Evolutionary Testing - Algorithm • Start with random population of chromosomes • Evolve population towards optimal solution by recombination and mutation P ← generateRandomPopulation() while P is not optimal CS ← evaluateFitness(chromosomes[P]) P´ ← performCrossOverOnPairsOf(CS) P ← mutate(P´) end while E.g., eToc [Tonella 04]

5. Fitness Calculation eToc [Tonella 04] public void withdraw(double amount) {L1 if (amount > balance) {L2 printError();L3 return; }L4 if (numberOfWithdrawals >= 10)L5 if (amount > 500.00) {L6 printError();L7 return;}L8 dispence(amount);L9 balance -= amount;L10 numberOfWithdrawals++; } Target

6. Re-combination eToc [Tonella 04]

7. Re-combination eToc [Tonella 04]

8. Re-combination

9. Mutation randomly mutated to 80.00

10. Evolutionary Testing - Summary + Method sequences are evolved guided by fitness - Method arguments are largely randomly picked

11. Dynamic Symbolic Execution amount: 20.0 Also called concolic testingEx. DART, jCUTE, Pex

12. Dynamic Symbolic Execution amount: 20.0 amount > 1.0 Also called concolic testingEx. DART, jCUTE, Pex

13. Dynamic Symbolic Execution amount: 20.0 Also called concolic testingEx. DART, jCUTE, Pex

14. Dynamic Symbolic Execution amount: 20.0 Also called concolic testingEx. DART, jCUTE, Pex

15. Dynamic Symbolic Execution amount: 20.0 Negates amount > 1.0 new constraint: amount <= 1.0 new value: amount: 1.0 (1.00); Also called concolic testingEx. DART, jCUTE, Pex

16. Evolutionary Testing - Summary + Method sequences are evolved guided by fitness - Method arguments are largely randomly picked + Method arguments are solved - Method sequences are fixed (or bounded)

17. Evacon Not that good at argument generation Not that good at sequence generation

18. Transform primitive arguments of method sequences (produced by evolutionary testing) into symbolic arguments. Benefits: Allow a symbolic execution tester (e.g., jCUTE) to do concrete and symbolic execution Transform any JUnit method sequence into a symbolic test driver. Argument Transformation

19. Argument Transformation - Example Sym exe test generation

20. Evacon Not that good at argument generation Not that good at sequence generation

21. Chromosome Construction • Construct chromosomes out of method sequences generated using symbolic execution Evolutionary test generation

22. Evacon - Summary Not that good at argument generation Not that good at sequence generation

23. Evaluation • We compare Evacon’s achieved branch coverage with four publicly available test generation tools: • eToc [Tonella 04] alone • jCUTE [Sen&Agha 06] alone • JUnit Factory[Agitar Labs 07] • Randoop [Pacheco&Ernst 07]

24. Experimental Subjects

25. Branch Coverage • Evacon-A achieves the highest branch coverage for 10 out of 13 subjects • The best branch coverage can be achieved by tool combination for 8 out of 13 subjects

26. Required Length of Method Sequences The length of the longest method sequence generated by Evacon-A or Randoop that achieves new branch coverage - The required length reaches up to 23- Bounded exhaustive testing may not be feasible

27. Branch Ranking • What does branch coverage of two tools: 85% > 75% tell? • Tool with 75% may be better at covering those difficult-to-cover branches when used in tool combination • Need take into account difficulties of branches being covered (esp. using tools in combination) • Proposed metric: #branches categorized into: • Branch-1: covered by only 1 tool under comparison • … • Branch-n: covered by only n tools under comparison Covering more branches in Branch-1 means uniquely covering more branches not being covered by the other tools under comparison

28. Branch Ranking #Covered Branches/#Branches in category Branch-n Evacon-A is best in terms of uniquely covering branches in Branch-1 Using Evacon-A + JUnit Factory is the best choice if only two tools are to be used(not necessarily Evacon-A + Randoop!)

29. Conclusion • A new integration of evolutionary testing and symbolic execution to achieve two tasks of OO test generation • An empirical comparison of our integration with state-of-the-art representative testing tools • A detailed comparison of the strengths and weaknesses of tools w.r.t achieving high structural coverage (e.g., branch ranking)

30. Questions? THANK YOU!

31. Experiments • We applied two types of Evacon integrations: • Evacon-A and Evacon-B • We measure branch coverage achieved by the tests generated by all six tools within the same period of runtime, except for JUnit Factory • For remaining tools we use Evacon-A’s runtime as the common runtime

32. Coverage Subsumption • Branch coverage of Evacon-A subsumed: • Evacon-B (in 12 of 13 PsUT) • eToc (in 7 of 13 PsUT) • jCUTE (in 3 of 13 PsUT) • JUnit Factory (in 1 of 13 PsUT) • Randoop (in 4 of the 13 PsUT) • Branch coverage of Randoop subsumed: • Evacon-A (in 1 of 13 PsUT) • Overall, branch coverage of Evacon-A + branch coverage of JUnit Factory subsumed branch coverage achieved by all tools in 5 of 13 PsUT