SoftTest : A Framework for Software Testing of Java Programs

1. SoftTest: A Framework for Software Testing of Java Programs Presented by: Priyanka Puri 1000676053 B. Childers, M. L. Soffa, J. Beaver, L. Ber, K. Cammarata, J. Litman, J. Misurda

2. Structural Software Testing • Assure quality, robust software • Collect coverage information about the program • E.g., branch coverage records which edges are hit • Uses program instrumentation

3. ..Continued.. • Automatically apply multiple test strategies • Multiple languages and platforms • Handle large programs

4. Static instrumentation approaches • Not scalable: Instrumentation left in place • Time, memory • Inflexible: Only certain tests, languages, platforms • Generally not detailed

5. Approach • Specification driven: User written test • Test plans: Recipe of how & where to test • Path specific: Instrument only what is needed • Dynamic: Insert & remove instrumentation

6. Test Framework Instance for Java Wizard for creating the test specification Source level coverage results From test specification determine how to test Test plan uses instrumentation primitives

7. Test specification • Apply different tests to various code regions. • may be applied with different coverage criteria • A GUI for • What tests to apply, where to apply, under what conditions • Select code regions • Conditions can be specified

8. Example test specification • DEFINITIONS • { • NAME: X, REGION_D, • LOCATION: FILE Example.java { • CLASS example, METHOD foo • } • } • BODY • { • DO BRANCH_TEST • ON REGION X UNTIL:90% • } • public class Example • { • public int foo(int x) • { • if(x==200) • x = x+100; • else x = x-100; • return x; • } • }

9. Test Planner • Test plan: Where and how to test program • Test table • Where to instrument in the target machine code • Test information (results) • Branch coverage: when to insert & remove instrumentation, records which edges are hit/not hit

10. • Test payload – What to do at an instrumentation point – Target machine code – Branch coverage: Update edge coverage

11. Branch Coverage Example 1 Test Table Test Payload Mark edge hit Insert at next point Remove instrumentation 2 3 4 Instrumentation Hit

12. Branch Coverage Example 1 Test Table Test Payload Mark edge hit Insert at next point Remove instrumentation 2 3 4 Instrumentation Hit

13. Branch Coverage Example 1 Test Table Test Payload Mark edge hit Insert at next point Remove instrumentation 2 3 4 Instrumentation Hit

14. Branch Coverage Example 1 Test Table Test Payload Mark edge hit Insert at next point Remove instrumentation 2 3 4 Instrumentation Hit

15. Branch Coverage Example 1 Test Table Test Payload Mark edge hit Insert at next point Remove instrumentation 2 3 4 Instrumentation Hit

16. Branch Coverage Example 1 Test Table Test Payload Mark edge hit Insert at next point Remove instrumentation 2 3 4 Instrumentation Hit

17. Branch Coverage Example 1 Test Table Test Payload Mark edge hit Insert at next point Remove instrumentation 2 3 4 Instrumentation Hit

18. Branch Coverage Example 1 Test Table Test Payload Mark edge hit Insert at next point Remove instrumentation 2 3 4 Instrumentation Hit

19. Branch Coverage Example 1 Test Table Test Payload Mark edge hit Insert at next point Remove instrumentation 2 3 4 Instrumentation Hit

20. Branch Coverage Example 1 Test Table Test Payload Mark edge hit Insert at next point Remove instrumentation 2 3 4 Instrumentation Hit

21. Branch Coverage Example 1 Test Table Test Payload Mark edge hit Insert at next point Remove instrumentation 2 3 4 Instrumentation Hit

22. Branch Coverage Example 1 Test Table Test Payload Mark edge hit Insert at next point Remove instrumentation 2 3 4 Instrumentation Hit

23. Branch Coverage Example 1 Test Table Test Payload Mark edge hit Insert at next point Remove instrumentation 2 3 4 Instrumentation Hit

24. Example CFG with block as its own successor

25. Test Virtual Machine • Provides functionality to insert and remove instrumentation at run-time. • Implements an interface for inserting and removing instrumentation with fast breakpoints*. • API provides primitives, such as • the placement of successor breakpoints, • storing test- specific data • removal of breakpoints, for constructing fast breakpoints with varying payloads

26. Test Analyzer • Displays the results of tests • For branch coverage, the analyzer displays the CFG for a method • CFG for target machine code

27. Observations • SoftTest significantly reduces the number of tests resulting in savings in scripting effort. • Significant impact on the acceptability of code coverage. • Provides cost effective approach to testing. • Low performance and memory overhead. • For the benchmarks, the memory overhead was 178 to 822 bytes for the test table. • The performance overhead for coverage varied from 1% to 5.5% of the total execution time.

28. Preliminary Results: Branch Coverage Testing [3] Overhead includes: Test Planner Instrumentation Run-time performance overhead

29. References • [1] http://veryoldwww.cs.pitt.edu/copa/eclipse.html • [2] SoftTest: A Framework for Software Testing of Java Programs • [3] http://www.cs.pitt.edu/coco/papers/SoftTest- Etx-2003 • [4] http://www.softwaretestingmentor.com/types-of- testing/static-testing.php

30. Questions?