1 / 14

CS527: Advanced Topics in Software Engineering (Software Testing and Analysis)

CS527: Advanced Topics in Software Engineering (Software Testing and Analysis). Darko Marinov September 18, 2008. Recent Schedule. Sep 9: Testing refactoring engines Sep 11: Static analysis Sep 16: Model checking TODAY: Sep 18: Test prioritization Sep 23&25: More on testing

mauli
Download Presentation

CS527: Advanced Topics in Software Engineering (Software Testing and Analysis)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CS527: Advanced Topics in Software Engineering(Software Testing and Analysis) Darko Marinov September 18, 2008

  2. Recent Schedule • Sep 9: Testing refactoring engines • Sep 11: Static analysis • Sep 16: Model checking • TODAY: Sep 18: Test prioritization • Sep 23&25: More on testing • Let me know if you have some preferred topic • Project proposal (more info on mailing list) • Your initial project idea will be due on Sep 25 • Proposal will be due the week after

  3. Report Feedback • Vilas and I will send you by Monday a summary about the reports we received • My spam filter caught some reports, so we should double-check we got all your reports • We’ll also comment on the quality of reports • These papers were from various areas • Your projects will focus on one area

  4. Paper Today • Empirical Studies of Test Case Prioritization in a JUnit Testing Environment by Hyunsook Do, Gregg Rothermel, Alex Kinneer (ISSRE 2004)

  5. Brief Summary • Motivation: regression testing is important for finding bugs • Problem: regression testing is slow • Solution: prioritize tests to find bugs faster • Evaluation: JUnit tests for 4 Java programs • Results: Prioritization helps

  6. Some Terminology • Regression testing • Test: test case, test input, test data… • Test suite: test set, test class… • Test (case) prioritization • Test (suite) minimization • (Regression) test selection • Code coverage

  7. Research Questions • RQ1: Can test case prioritization improve the rate of fault detection of JUnit test suites? • RQ2: How do the three types of information and information use that distinguish prioritization techniques (type of coverage information, use of feedback, use of modification information) impact the effectiveness of prioritization techniques? • RQ3: Can test suite granularity (the choice of running testclass level JUnit tests versus test-method level tests) impact the cost-effectiveness of prioritization techniques?

  8. Independent Variables • Prioritization technique • Untreated, random, optimal • Block-total, block-addtl • Method-total, method-addtl • Method-diff-total, method-diff-addtl • Test Suite Granularity • Class-level • Method-level

  9. Dependent Measure • APFD = 1 – (TF1 + … + TFm)/nm + 1/2n • n test cases • m faults • TFi is the index of the first test case to find fault i • Why is this between 0 and 100%?

  10. Threats to Validity • Internal • Are there bugs in experimental tools? • Are seeded faults the same as real? • External • Are programs representative? • Is process representative? • Construct • Is APFD an appropriate measure?

  11. Brief Summary of Results • RQ1: Can test case prioritization improve the rate of fault detection of JUnit test suites? • Yes • RQ2: How do the three types of information and information use that distinguish prioritization techniques (type of coverage information, use of feedback, use of modification information) impact the effectiveness of prioritization techniques? • Coverage little, feedback lot, modification little • RQ3: Can test suite granularity (the choice of running testclass level JUnit tests versus test-method level tests) impact the cost-effectiveness of prioritization techniques? • Yes

  12. Questions for Discussion (1) • Why no significant improvement when prioritizing on code modification? [DH] • Why is test-class level not as good as test-method? [ST] • How much the results vary based on how one writes a JUnit test class? [YL] • Why random better than untreated? [WX] • What new techniques would you propose? • How prioritize automatically generated tests?

  13. Questions for Discussion (2) • What is hot topic in testing: regression test selection, test suite minimization, test case prioritization, automatic test generation...? • What other prioritization techniques exist? • How does test case prioritization fit in TDD? • Is the extra effectiveness of prioritization techniques worth the extra cost (compared to random ordering)?

  14. Questions for Discussion (3) • How to apply this in the real world if we don’t know when all bugs have been found? • How Galileo framework works (more detail)? • Questions like this can always lead to a project • How to prioritize in integration testing? • How to prioritize based on, say, complexity or criticality of test units? • What are other regression test performance metrics besides APFD?

More Related