1 / 43

Value-Based Software Test Prioritization

Value-Based Software Test Prioritization. University of Southern California Qi Li, Barry Boehm {qli1, boehm }@ usc.edu November 4, 2011. Outline. Research Motivation Research Method Case Studies Conclusion. Research Motivation.

dexter-horne
Download Presentation

Value-Based Software Test Prioritization

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Value-Based Software Test Prioritization University of Southern California Qi Li, Barry Boehm {qli1, boehm}@usc.edu November 4, 2011

  2. Outline • Research Motivation • Research Method • Case Studies • Conclusion COCOMO Forum 2011

  3. Research Motivation • Value-neutral SE methods are increasingly risky [Boehm, 2003] • Every requirement, use case, object, test case, and defect is equally important • “Earned Value” Systems don’t track business value • System value-domain problems are the chief sources of software project failures • Testing & Inspection resources are expensive and scarce • 30%-50%, even higher for high reliability projects [Ramler, 2005] • Time-to-market [Boehm, Huang, 2005] • Empirical Findings [Bullock 2000, Boehm & Basili 2001 ] • About 20 percent of the features provide 80 percent of business value • About 80 percent of the defects come from 20 percent of the modules • About 80 percent of avoidable rework comes from 20 percent of the defects • About 90 percent of the downtime comes from, at most, 10 percent of the defects • … COCOMO Forum 2011

  4. Outline • Research Motivation • Research Method • Case Studies • Conclusion COCOMO Forum 2011

  5. Fundamental Theory • Value-based Software Engineering 4+1 theorem [Boehm, 2005] COCOMO Forum 2011

  6. Value-Based Software Test Prioritization What to be prioritized? Testing items: Testing Scenarios, Testing Features, Test Cases How to prioritize? Value-Based (Business Importance, Risk, Cost) Dependency Aware How to Measure? Average Percentage of Business Importance Earned (APBIE)

  7. Research Method: Value-Based • Risk Exposure (RE) • Where Size (Loss) is the risk impact size of loss if the outcome is unsatisfactory, Pro (Loss) is the probability of an unsatisfactory outcome • Risk Reduction Leverage (RRL) • Where REbefore is the RE before initiating the risk reduction effort and REafter is the RE afterwards. • RRL is a measure of the cost-benefit ratio of performing a candidate risk reduction or defect removal activity COCOMO Forum 2011

  8. Research Method: Value-Based • Value-Based Prioritization Drivers: • Business Case Analysis • Stakeholder Prioritization • Impact of Defect • Experience Base • Business Value • Defect Criticality • Defect-prone Components, Performers • Size of Loss • Probability of Loss • Risk Exposure COCOMO Forum 2011

  9. Research Method: Value-Based • Objects are to be ranked by how well they can reduce risk exposure • Combining with their relative option costs • =>Priority Trigger: • This proposed strategy enables them to be prioritized in terms of Risk Reduction Leverage (RRL) or ROI • Supposed to improve the lifecycle cost-effectiveness of defect removal techniques COCOMO Forum 2011

  10. Research Method: Dependency Aware Natural speech input Tertairy application functions Animated displays Secondary application functions User amenities Main application functions Value of software product to organization Basic application functions Operating System Data management system Investment High-payoff Diminishing returns Cost of software product [Boehm, 1981] COCOMO Forum 2011

  11. Research Method: Dependency Aware • Dependency: • Example: dependencies among test cases to be executed • Solution: Prioritization Algorithm (greedy alg) • Select the one with the highest RRL • Check dependency • 9->3->9->5->9->4->7 COCOMO Forum 2011

  12. Research Method: Metrics • Testing Cost Effectiveness • Average Percentage of Business Importance Earned (APBIE) COCOMO Forum 2011

  13. Outline • Research Motivation • Research Method • Case Studies • Conclusion COCOMO Forum 2011

  14. Case Studies • Exercise Test Prioritization based on Risk Reduction Level (RRL) • software testing scenarios to be walked through • software features to be tested • software test cases to be executed COCOMO Forum 2011

  15. Case Studies Study 1: Prioritize testing scenarios to be walked through [Li TR1 2011] • Case Study: • Galorath Inc. (2011 Summer) • Project: Installation Process Automation • Challenge: Testing all scenarios is impossible under limited testing resources (69 scenarios) COCOMO Forum 2011

  16. Case Studies Study 1: Prioritize testing scenarios to be walked through • Case Study Results: Value-based • Value-based prioritization can improve the cost-effectiveness of testing Value-neutral Value-inverse (worst case) COCOMO Forum 2011

  17. Case Studies Study 2: Prioritize software features to be tested [Li ICSP 2009] • Case Study: • Institute of Software, Chinese Academy of Sciences (2008 Spring) • System testing of Version3.1 SoftPM that covers 9 features COCOMO Forum 2011

  18. Case Studies Study 2: Prioritize software features to be tested • Case Study Results • Value-based prioritization can improve the cost-effectiveness of testing COCOMO Forum 2011

  19. Case Studies Study 3: Prioritize software test cases to be executed [Li TR2 2011] • Experiment • USC-CSCI 577b (2011 Spring) • 5 Real Client Course Projects • Acceptance testing phase COCOMO Forum 2011

  20. Case Studies Study 4: Prioritize software test cases to be executed • Experiment Results (Quantitative) Value-based • Project 1 as an example, • Value-based prioritization can improve the cost-effectiveness of testing Value-based Value-neutral COCOMO Forum 2011

  21. Case Studies Study 3: Prioritize software test cases to be executed • Experiment Results (Observance) • Defects with higher Priority and Severity are reported earlier and resolved earlier COCOMO Forum 2011

  22. Case Studies Study 4: Prioritize software test cases to be executed • Experiment Results (Qualitative) “Before doing the prioritization, I had a vague idea of which test cases are important to clients. But after going through the Value-Based testing, I had a better picture as to which ones are of critical importance to the client.” “I prioritized test cases mainly based on the sequence of the system work flow, which is performing test cases with lower dependencies at first before using value-based testing. I like the value-based process because it can save time by letting me focus on more valuable test cases or risky ones. Therefore, it improves testing efficiency.” “Value-based testing is very useful in complex systems with hundreds or thousands of test-cases. However in 577 it should not be difficult to run every test-case in every test iteration, making the prioritization less useful. The impact of value-based testing and automated test management on software quality is entirely dependent on the complexity of the project. If complete test coverage is possible in the time given, the benefit of VBST to software quality is minimal.” COCOMO Forum 2011

  23. Outline • Research Motivation • Research Method • Case Studies • Conclusion COCOMO Forum 2011

  24. Conclusion • Propose a Real “Earned Value” System to Track Business Value of Testing and Measure Testing Efficiency in terms of APBIE • Propose a Systematic Strategy for Value-based, Dependency Aware Test Processes • Apply This Strategy to Series of Empirical Studies with different granularities of Prioritizations • Elaborate Decision Criteria of Testing Priorities Per Project Contexts, Which are Helpful for Real Industry Practices COCOMO Forum 2011

  25. Backup Slides COCOMO Forum 2011

  26. Tight Dependency 3 9 5 4 9 9 7 • 9->3->9->5->9->4->7

  27. Tight Dependency 5 1 5 2 5 9 • Path 1 • Path 2 • Path 1->Path 2

  28. Study 1. Value-based Testing: Prioritize Testing Scenarios COCOMO Forum 2011

  29. COCOMO Forum 2011

  30. Study 2. Value-based Testing: Prioritize Testing Features COCOMO Forum 2011

  31. Business Importance (BI) • Use Karl Wieger’s Method to get the relative BI for each feature • BIi= WBenefit* Benifiti +WPenalty* Penaltyi COCOMO Forum 2011

  32. Quality Risks & Weights Allocation Analytical Hierarchy Process (AHP) COCOMO Forum 2011

  33. Quality Risks Probability 11/3/2011 COCOMO Forum 2011 33

  34. Testing Cost COCOMO Forum 2011

  35. Put the three together: Value Priority COCOMO Forum 2011

  36. Study 3. Value-based Testing: Prioritize Test Cases COCOMO Forum 2011

  37. Step1: Dependency Analysis COCOMO Forum 2011

  38. Step 2:Rate Business Importance COCOMO Forum 2011

  39. Step 3: Rate Criticality COCOMO Forum 2011

  40. Step 4: Rate Fail Probability COCOMO Forum 2011

  41. Research Proposal: Dependency Aware • Prioritization Algorithm • Value First: • Test the one with the highest Testing Value. If several test cases’ Test Values are the same, test the one with the highest Criticality • Dependency Second: • If the test case selected from the first step is not “Ready-to-Test”, which means at least one of the test cases in its Dependencies Set is “Not-Tested-Yet”. In such situation, prioritize the “Not-Tested-Yet” test cases according to “Value First” in this Dependencies Set and start to test until all test cases in the Dependencies Set are “Passed”. Then the test case with the highest value is “Ready-to-Test” • Update the prioritization: • After one round, update the Fail Probability based on updated observation from previous testing rounds COCOMO Forum 2011

  42. Research Proposal: Dependency Aware COCOMO Forum 2011

  43. Future Research Plan • Tool Support for Value-based Test Prioritization COCOMO Forum 2011

More Related