1 / 19

Test case prioritization

Test case prioritization. Paolo Tonella Fondazione Bruno Kessler tonella@fbk.eu. Motivation. The order in which test cases are executed affects: The rate of fault detection : good orderings reveal faults earlier than bad ones, or (alternatively) they reveal high risk faults earlier;

leighton
Download Presentation

Test case prioritization

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Test case prioritization Paolo Tonella Fondazione Bruno Kessler tonella@fbk.eu

  2. Motivation The order in which test cases are executed affects: • The rate of fault detection: good orderings reveal faults earlier than bad ones, or (alternatively) they • reveal high risk faults earlier; • reveal faults related to critical code sections earlier; • give satisfactory confidence in the system’s reliability earlier. • The rate of code coverage: good orderings meet the required coverage level earlier than bad ones.

  3. The need for a heuristic approach The faults revealed by a test case are unknown until the test case is executed and its output is evaluated against the oracle. Heuristics are required to estimate the ability of a test case to reveal faults.

  4. Main prioritization heuristics • Available heuristics take advantage of execution information, assuming this can be obtained at a relatively low cost (if the output is not validated) or can be estimated from previous runs: • Coverage-based prioritization. • Mutation-based prioritization.

  5. Coverage-based heuristics • Total statement coverage: test cases are prioritized by number of statements covered. • Additional statement coverage: test cases are prioritized by incremental number of statements covered. • Total branch coverage: test cases are prioritized by number of branches covered. • Additional branch coverage: test cases are prioritized by incremental number of branches covered.

  6. Statement coverage Total statement coverage: (TC3, TC1, TC2) Additional statement coverage: (TC3, TC2, TC1)

  7. Mutation-based heuristics • Total Fault-Exposing-Potential (FEP): test cases are prioritized by FEP. • Additional Fault-Exposing-Potential (FEP): test cases are prioritized by incremental confidence ΔC. Underlying assumption: very low execution cost. A mutant of a statement is generated (automatically) by randomly changing the statement (e.g., “x++” →“x--”). A mutant is killed by a test case if the original and the mutated program give different outputs when the test case is executed.

  8. Mutation-based heuristics • Additional Fault-Exposing-Potential (FEP): test cases are prioritized by incremental confidence ΔC. C estimates the confidence in the correctness of a statement. Initially, C is zero for every statement. When the test case with highest ΔC is selected, the confidence is updated as follows:

  9. FEP-based prioritization Total FEP: (TC2, TC1, TC3)

  10. ΔC-based prioritization ΔC: (TC2)

  11. ΔC-based prioritization

  12. ΔC-based prioritization ΔC: (TC2, TC3)

  13. ΔC-based prioritization

  14. ΔC-based prioritization ΔC: (TC2, TC3, TC1)

  15. Empirical validation APFD: weighted Average of the Percentage of Faults Detected. 100% faults APFD = 50% A B C D E Prioritized test cases

  16. Empirical validation APFD: weighted Average of the Percentage of Faults Detected. 100% faults APFD = 84% C E B A D Prioritized test cases

  17. Experimental results • All the heuristics outperform the untreated or randomly prioritized test suites. • In almost every case, additional FEP prioritization outperforms coverage-based prioritization, but with only a marginal gain in APFD. • Given their expense, FEP-based prioritization may not be as cost effective as coverage based techniques More details in: Gregg Rothermel, Roland Untch, Chengyun Chu, and Mary Jean Harrold, Test Case Prioritization. IEEE Transactions on Software Engineering, vol.27, no.10, pp.929-948, October, 2001.

  18. Taking advantage of user knowledge Pair-wise comparisons TCi < TCj (TCi, TCj) Prioritized TS 1: TCi1 1 3 5 7 9 AHP 2: TCi2 Intensity of importance n: TCin TS = {TC1, …, TCn} AHP (Analytic Hierarchy Process) determines the target ranking of the test cases in TS from user input (pair-wise comparisons).

  19. Analytic Hierarchy Process 1 Equal 3 Moderate 5 Strong 7 Very strong 9 Extreme TC1 TC2 TC3 TC4 TC1 1/1 3/1 1/5 1/9 1/3 1/1 3/1 3/1 5/1 1/3 1/1 5/1 9/1 1/3 1/5 1/1 A = TC2 TC3 TC4 N * (N-1) / 2 pair-wise comparisons elicited vmax A vmax = λmax vmax TC1 TC3 0.289 0.544 0.634 0.466 TC2 TC2 TC3 TC4 Principal eigenvector TC4 TC1 λmax = 7.248

More Related