The ROI of Testing Presented By: Shaun Bradshaw Questcon Technologies sbradshaw@questcon

1. Quality - Innovation - Vision • The ROI of Testing • Presented By: • Shaun Bradshaw • Questcon Technologies • sbradshaw@questcon.com

2. Return on Investment

3. Return on Investment (ROI) • What is ROI? • ROI is a calculation that attempts to determine the actual or perceived future value of an expense or investment • By calculating ROI, an organization can assess whether the expense/investment is justified by the resulting savings/revenue

4. Return on Investment (ROI) • How is ROI calculated? • In its most basic form ROI is calculated as follows: ROI = (Benefits – Costs)/Costs • That is, the financial benefit after an investment or improvement is made minus the cost of the investment or improvement, calculated as a percentage of those costs • Benefits • Revenue generated • Cost reduction • Cost avoidance • Productivity improvements • Costs • Labor expense • Time expense • Tool expense

5. Return on Investment (ROI) • What makes testing valuable? How do we get to a positive ROI? • Produces information reliably grounded in observed system behavior [1] • Functions like a credit check [1] • Exposes risk and improves decision-making • Extends the life of the software • Improves the development process • Increases end-user confidence and satisfaction

6. Defect Preventionvs.Defect Detection

7. Importance of Testing Organizations rely on testers to do QA We cannot “test” quality into an application Test needs to be the “backdoor to QA” Requires a shift in the testing paradigm Test Finding Defects Defect Prevention vs. Defect Detection Measuring Quality

8. Rules & Results of Defect Prevention 60% Rule [2]Percentage of defectsintroduced in require-ments and design 85% Rule [2]Percentage of defects removed through reviews, inspections and testing 368:1 Rule [3] Defect Prevention vs. Defect Detection

9. 30 0 30 30 10 90 15 30 60 100 0 15 0 0 85 $0 $212,100 $0 $0 $166,600 $0 $378,700 $0 $0 $166,600 Defect Prevention vs. Defect Detection Testing Only Assume X is $100 & 100 total defects Requirements Design Code Test Production X 5X 10X 50X 368X+ # of defects introduced Accumulated defects Defects removed Cost Accumulated Cost

10. 30 30 30 10 0 34 30 35 15 2 26 29 30 13 2 $2,600 $14,500 $30,000 $65,000 $73,600 $2,600 $17,100 $47,100 $112,100 $185,700 Defect Prevention vs. Defect Detection Full QA Assume X is $100 & 100 total defects Requirements Design Code Test Production X 5X 10X 50X 368X+ # of defects introduced Accumulated defects Defects removed Cost Accumulated Cost

11. Quantitative Benefits

12. Quantitative Benefits • Reduced Defect Repair Costs • Test involvement earlier in the lifecycle reduces repair costs • Automation improves defect detection • Good test practices increase detection likelihood

13. Decreased Production Costs Defects in production can be significantly higher than 368x - the “Annuity Nightmare” Small rounding error goes undetected at a major financial institution Months until discovered Code repair is cheap Production cost – NOT CHEAP! Increased Revenue & Profits Reduced time-to-market for functioning software Increased market share Higher customer retention/goodwill Reduced maintenance costs (easier to enhance) Quantitative Benefits

14. Calculating Test Value

15. Defect Injection Rate Total number of defects introduced into an application When each defect was introduced (using Root Cause Analysis) Number of defects introduced in each phase of the SDLC Calculate the defect injection rate as: # of defects introduced total defects Key Metrics to Calculate Test Value

16. Defect Repair Cost Hourly rate of each resource involved in the repair (PM, BA, Developer, Tester, etc.) Defect repair time for each resource by phase of the SDLC Number of defects repaired in each phase of the SDLC Calculate the defect repair costs as shown in the example*: *Data in the example was drawn from [4]. Key Metrics to Calculate Test Value

17. “Test” Effectiveness Number of defects found during “testing” activities during any phase of the SDLC Total number of defects found including in production (use a “warranty period”) Calculate test effectiveness as: # of defects foundphase total # of defects found Key Metrics to Calculate Test Value

18. ROI Example

19. ROI Example

20. ROI Case Study • Established a test team made up of: • 1 test lead • 1 test analyst • 1 test consultant • Total 1st year investment: $237,500

21. ROI Case Study • Conducted 2 projects in first 8 months • Realized $173,000 savings in project-over-project defect repair costs • Calculated savings do not include other benefits such as: • Reduced production downtime • Reduced maintenance costs • Increased customer satisfaction & trust • Anticipated 46% ROI at the end of year 1

22. Questions sbradshaw@questcon.com 336-273-2428

23. References • Bullock, James. “Calculating the Value of Testing.” Software Testing & Quality Engineering: May/June, 2000. <www.stickyminds.com/> • Jones, Capers. "Software Cost Estimating Methods for Large Projects." CrossTalk: April, 2005. <www.stsc.hill.af.mil/crosstalk/2005/04/0504Jones.html> • Dabney, JB. "Return on Investment of Independent Verification and Validation Study Preliminary Phase 2B Report." Fairmont, WV: NASA IV&V Facility, 2003. <sarpresults.ivv.nasa.gov/ViewResearch/289/24.jsp> • US Dept. of Commerce, National Institute of Standards & Technology (NIST). “Planning Report 02-3: The Economic Impacts of Inadequate Infrastructure for Software Testing.” Technology Program Office, Strategic Planning & Economic Analysis Group. May, 2002.<www.nist.gov/director/prog-ofc/report02-3.pdf>