Time-Aware Test Suite Prioritization

1. Time-Aware Test Suite Prioritization Time-Aware Test Suite Prioritization Kristen R. Walcott Mary Lou Soffa Department of Computer Science University of Virginia {walcott, soffa}@cs.virginia.edu Gregory M. Kapfhammer Robert S. Roos Department of Computer Science Allegheny College {gkapfham, rroos}@allegheny.edu http://www.cs.virginia.edu/~krw7c/ Improved Fault Detection Rates Problem Statement • Two case study applications were considered: • Gradebook (28 test cases) • JDepend (53 test cases) • In order to evaluate the effectiveness of a given tuple of test cases, forty bugs were randomly seeded into the test applications, and the Average Percent of Faults Detected (APFD) was calculated for each GA-prioritized test suite. • GA time-aware test suite prioritizations were compared to: • Random prioritizations • Initial test suite orderings • As can be seen above, experimental analysis shows that our approach can create time-aware prioritizations that significantly outperform other prioritization techniques. On average, our prioritizations outperformed random prioritizations, and they had up to 120% improvement in APFD over • ordering based prioritizations. • In future work, we will examine methods to reduce the time overhead • of the search heuristic and further enhance the algorithm. Regression test prioritization is often performed in a time constrained execution environment in which testing only occurs for a fixed time period. For example, many organizations rely upon nightly building and regression testing of their applications each time source code changes are committed to a version control repository. However, no existing prioritization method takes a time budget into account. By factoring in this time constraint, a better test suite prioritization can be produced. Test Suite Prioritization We wanted to study the improvement in fault detection when test suites are prioritized with a time budget in mind. To do this, we used a genetic algorithm (GA) heuristic search technique to create prioritizations. • Test suites prioritized by the GA have the following attributes: • Run within a given time • limit • Have the highest possible potential for bug detection based on derived code coverage information • Reverse test suite orderings • Fault-aware prioritizations