Can we calculate the profit of testing?

1. Can we calculate the profit of testing? Jurian van de Laar Philips TASS 10th Dutch Testing Day October 8th, 2004 - Leiden Version 2.0

2. Agenda • Introduction • Why talk about ‘profit’ of testing ? • Consequences of insufficient testing • Profit of testing: calculation example • Supporting practices • Profit calculation example refined • Conclusions

3. Introduction • Philips TASS International • Independent subsidiary of Philips Electronics • Technical and embedded software • Philips TASS Netherlands • Founded in 1978, Location Eindhoven • 160 software specialists • HW/SW Interfacing, Remote Services,Integration & Testing (I&T) • TASS I&T task force: • Increase I&T knowledge (ISEB) • Sharing I&T best practices • Enable customers to • compose reliable embedded software products • reduce the risk of product failure.

4. Software testing in our line of business • Commitment, metrics, change control • Rigorous testing vs strategy / techniques • Large quantities, complex or safety critical • Changing software in operation is difficult • Driven by TTM (market-windows)

5. Why talk about ‘profit’ of testing ? • To quantify the benefits of testing • Emphasis mostly on cost ;benefits hard to explain • Failures after production: taken into consideration? • Early fault detection contributes to total profit • To make test process more efficient • ‘Calculated risks’ sometimes not really calculated • Finding severe faults in testing prevents high costs

6. So, testing is important …but expensive. Rigorous … but exhaustive is impossible. ‘Bottleneck’ for TTM … a ‘calculated’ risk. How much to invest in ‘sufficient’ testing? Cost of testing Cost of failure The ‘profit’ of testing … Profit: prevented cost of failure - cost of testing

7. “Murphy’s Law ..” Source: Rijkswaterstaat, www.aanlegA50.nl “Two failures at the same time ..” “.. the other failure was in the control software.” “.. expects that the supplier will install faultless software this week.” Source: Eindhovens Dagblad Tuesday August 8, 2004 Source: Eindhovens Dagblad Tuesday July 27, 2004

8. Consequences of insufficient testing • Dissatisfied customers (claims) • Bad publicity (image/brand damage) • Service (helpdesk, logistics) • Fixing the faults (repairs)

9. Costs of fixing a fault relative to phase • Consequence worse if fault is found late • Statistics show exponential increase • Operation phase: 1000 x Cost Reqt. phase From: E. van Veenendaal, The Testing Practitioner, 2002. Source: B. Boehm, Software Engineering Economics, 1981

10. Can we calculate the profit of testing? • A simplified hypothetical example: • Costs of testing effort: 180,000 Euro. • Serious problem found in acceptance testing. • Fix in acceptance test phase costs 1,400 Euro. • If not found until in operation: costs 20,000 Euro. • ‘Profit’: n x (20,000 - 1,400) - 180,000 !! (where n is amount of serious problems found)

11. Supporting practices • Risk based testing • Using intake criteria and acceptance criteria • Measuring severity • Measuring field call rate

12. Risk management in unit testing • Classify modules using criteria for Impact and Likelihood • E.g. HIGH: give higher priority and more thorough testing • Result: important defects found sooner: lower costs

13. Using intake and acceptance criteria • Example 1: Mobile Phone Project • Intake: Every module tested and reviewed • Acceptance: customer acceptance test • Example 2: TV Project • Intake procedure: paper check release notes • Acceptance: team required on customer site

14. Severity Severity D: Acceptable Severity C: Cosmetic / Minor issue Severity B: Serious Problem Severity A: Product can not be produced, block acceptance

15. Indication to what extent the product doesn’t meet the expectations of the customer, resulting in complaints. FCR After the project: FCR • Meaning: • Simplified example: 4% FCR: Of every 100 sold products,4 products are returned by the customerto the service organization (warranty period) • Used for root cause analysis,not a financial driver

16. Supporting practices resumed • Risk based testing • Prioritize & Differentiate => Test Strategy ! • More effective testing, Lower cost of failure. • Using intake criteria and acceptance criteria • More explicit transitions between phases • Reduce costs of fixing faults from earlier phases • Measuring severity • Monitor seriousness of (potential) failure • Input for calculating cost/benefits of testing • Measuring field call rate • Analyze failures after product release • Input for calculating costs of product failure

17. Can we calculate the profit of testing? • Hypothetical example refined: • Found nserious problems (B) in unit testing. • Root cause analysis learned: type B yields 2% FCR. • 2% FCR would cost 20,000 x 50 = 1,000,000 Euro. 1,000,000 products sold Repair cost: 50 Euro per product Selling price: 500 Euro per product Expectations of supplier: 10% profit (50,000,000 Euro) 4% FCR (2,000,000 Euro) • 2% FCR reduction = extra profit of 1,000,000 Euro. • 52 B-problems NOT found would consume all profit! Taking cost of testing into consideration, e.g. (n x 800) + 180,000: extra profit is 819,200 Euro.

18. Conclusions Testing is not cheap … … insufficient testing is expensive. Risk based testing and criteria can increase effectiveness. Measuring during projects is important … … and if we continue to measure after release, costs and profit of testing can be calculated. Your contribution in refining and improving these calculations is more than welcome !!