1 / 65

Comparing and Combining Test-Suite Reduction and Regression Test Selection

Explore test-suite reduction and regression test selection strategies to enhance testing efficiency. Learn how TSR and RTS compare quantitatively, and discover the benefits of combining these methods. Investigate the fault-detection capabilities and criteria for detecting change-related faults. Evaluate the size decrease and fault-detection capability decrease metrics to assess strategy effectiveness. Develop insights on detecting change-related faults accurately and enhance overall testing strategies.

vernonm
Download Presentation

Comparing and Combining Test-Suite Reduction and Regression Test Selection

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Comparing and Combining Test-Suite Reduction and Regression Test Selection August Shi, Tifany Yung, Alex Gyori, and Darko Marinov FSE 2015 Bergamo, Italy 09/02/2015 NSF Grant Nos. CCF-1012759, CCF-1421503, CCF-1434590, CCF-1439957

  2. Testing is Important but Slow test0 test1 test2 test3 … testN-1 testN Code Under Test V0

  3. Regression Testing is Slow(er) test0 test1 test2 test3 … testN-1 testN Code Under Test V0 test0 test1 test2 test3 … testN-1 testN Code Under Test V1 test0 test1 test2 test3 … testN-1 testN Code Under Test V2

  4. Speeding up Regression Testing • Test-Suite Reduction • Regression Test Selection • Test-Suite Parallelization • Refactoring Tests • Many More

  5. Speeding up Regression Testing • Test-Suite Reduction • Regression Test Selection • Test-Suite Parallelization • Refactoring Tests • Many More

  6. Test-Suite Reduction (TSR) test0 test1 test2 test3 … testN-1 testN Code Under Test V0 test0 test1 test2 test3 … testN-1 testN Code Under Test V1 test0 test1 test2 test3 … testN-1 testN Code Under Test V2

  7. Regression Test Selection (RTS) Δ test0 test1 test2 test3 … testN-1 testN Code Under Test V0 test0 test1 test2 test3 … testN-1 testN Code Under Test V1

  8. Regression Test Selection (RTS) test0 test1 test2 test3 … testN-1 testN Code Under Test V0 test0 test1 test2 test3 … testN-1 testN Code Under Test V1 test0 test1 test2 test3 … testN-1 testN Code Under Test V2

  9. TSR versus RTS(Known Qualitative Comparison) Test-Suite Reduction Regression Test Selection How are tests chosen to run? Redundancy (one revision) Changes (two revisions) How often is analysis performed? Infrequently Every revision Can it miss failing tests from the original test suite? Yes No (if safe)

  10. How do TSR and RTS compare quantitatively?How can TSR and RTS be combined?

  11. How do TSR and RTS compare quantitatively?How can TSR and RTS be combined?

  12. TSR Background T = Tests S = Statements

  13. TSR Background T = Tests S = Statements Reduced Test Suite R = {T3,T5}

  14. TSR Background T = Tests S = Statements R = {T3,T5} Size

  15. TSR Background T = Tests S = Statements R = {T3,T5} Size

  16. TSR Background T = Tests S = Statements R = {T3,T5} Size Fault-Detection Capability

  17. TSR Background T = Tests S = Statements M = Mutants R = {T3,T5} Size Fault-Detection Capability

  18. RTS Background T = Tests S = Statements Δ S2 changed S6 added Vi-1 Vi

  19. RTS Background T = Tests S = Statements Δ S2 changed S6 added Vi-1 Vi Selected Tests Si,Δ = {T1,T2,T3}

  20. RTS Background T = Tests S = Statements Δ S2 changed S6 added Vi-1 Vi Si,Δ= {T1,T2,T3} Size

  21. RTS Background T = Tests S = Statements Δ S2 changed S6 added S2 changed S6 added Vi-1 Vi Si,Δ= {T1,T2,T3} Size Fault-Detection Capability Safe RTS does not fail to detect change-related faults

  22. How can TSR and RTS be combined?

  23. Applying RTS after TSR T = Tests S = Statements Vi-1

  24. Applying RTS after TSR T = Tests S = Statements Vi-1 R = {T3,T5}

  25. Applying RTS after TSR T = Tests S = Statements Δ S2 changed S6 added S2 changed S6 added Vi Vi-1 Ri = {T3,T5}

  26. Applying RTS after TSR T = Tests S = Statements Δ S2 changed S6 added S2 changed S6 added Vi-1 Vi Ri = {T3,T5}

  27. Applying RTS after TSR T = Tests S = Statements Δ S2 changed S6 added S2 changed S6 added {T3} Vi-1 Vi Fault-Detection Capability Size If RTS is safe, then as good as reduced test-suite Selection of Reduction (SeRe)

  28. Metrics to compare between approaches • Size Decrease: • TSR: • RTS: • SeRe: • Fault-Detection Capability Decrease • Currently, NO metric for fault-detection capability between approaches • We need a metric that takes CHANGE into account

  29. Map Tests to Faults T = Tests F= Faults Vi-1 Vi Need criteria that includes these change-related faults

  30. Detect all Faults? T = Tests F= Faults Vi-1 Vi

  31. Which is Better? T = Tests F= Faults Vi-1 Vi Detects 5 faults Detects 1 change-related fault

  32. Which is Better? T = Tests F= Faults Vi-1 Vi Detects 5 faults Detects 1 change-related fault Detects 4 faults Detects 2 change-related faults If criteria is to detect all faults, can get misleading comparisons with respect to these change-related faults!

  33. Finding Change-Related Faults T = Tests F= Faults Vi-1 Vi Safe RTS will not fail to select tests whose behavior differs after the change

  34. Finding Change-Related Faults T = Tests F= Faults Vi-1 Vi Si,Δ = {T1,T2,T3}

  35. Finding Change-Related Faults T = Tests F= Faults Vi-1 Vi Si,Δ = {T1,T2,T3} Faults(Si,Δ) = {F1,F2,F3,F4,F6}

  36. Finding Change-Related Faults Faults detected by non-selected tests cannot be change-related! ChangeRelatedFaultsi,Δ = Faults(Si,Δ) \ Faults(Oi\ Si,Δ) Faults(Si,Δ) \ Faults(Oi \ Si,Δ) = Faults({T1,T2,T3}) \ Faults({T4,T5}) Faults(S1,Δ) \ Faults(O1 \ S1,Δ= {F1,F2,F3,F4,F6} \ {F1,F3,F5} = {F2,F4,F6} Vi Vi-1

  37. Change-Related Requirements (CRR) Use testing requirements (statements covered or mutants killed) to approximate fault-detection capability of test suite T chosen from Oi Evaluate loss in change-related fault-detection capability of reduced test suite

  38. Evaluation Setup

  39. Projects

  40. Experimental Setup • Use Greedy heuristic to perform TSR • Remove redundant tests with respect to statement coverage • Statement coverage/mutants killed collected using PIT • http://pitest.org • Use Ekstazi to perform (safe) RTS • Select tests based on file-level dependencies • Tests selected at test class level • http://www.ekstazi.org • Simulate evolving reduced test suite and selection of reduction

  41. Evolving Reduced Test Suite test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 Code Under Test V0

  42. Evolving Reduced Test Suite test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 Code Under Test V0

  43. Evolving Reduced Test Suite test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 Code Under Test V0

  44. Evolving Reduced Test Suite test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1

  45. Evolving Reduced Test Suite test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1

  46. Evolving Reduced Test Suite test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 test4 Code Under Test V2 test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1

  47. Evolving Reduced Test Suite test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 test4 Code Under Test V2 test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 test4 Code Under Test V2

  48. Evolving Reduced Test Suite test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 test4 Code Under Test V2 test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 test4 Code Under Test V2

  49. Evolving Reduced Test Suite test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 test4 Code Under Test V2 test0 test1 test2 test3 Code Under Test V0 test0 test1 test2 test3 test4 Code Under Test V1 test0 test1 test2 test3 test4 Code Under Test V2

  50. Selection of Reduction • Can see result of selection of reduction by looking at tests chosen by TSR and RTS • Given and , can intersect the two to see what tests from are selected due to changes

More Related