Using Metrics to Reduce Cost of Re-work

1. Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton

2. Opening Question “What is the job of testing?”

3. Potential Response “The job of testing is to find defects.”

4. Myth Exposed Myth: The Job of Testing Is to Find Defects1 1The Seven Principles of Lean Software Development by Mary Poppendieck

5. “Quality is everyone’s responsibility” Myth: The Job of Testing Is to Find Defects “The job of tests, and the people that develop and run tests, is to prevent defects, not find them. A quality assurance organization should champion processes that build quality into the code from the start rather than test quality in later.”1 1The Seven Principles of Lean Software Development by Mary Poppendieck

6. At a recent software seminar “The common attributes of successful software projects are • Heroes • Chaos • Re-work • Untracked OT”

7. Today’s Discussion • Case Study – 4 projects • 2 Definitions • Testing Metrics (what to collect & analyze) • Metrics from first 2 projects • Process Improvement Steps • Success Story – Projects 3 & 4 • Some trends & conclusions

8. Project Background • Legacy mature system (mainframe) • Industry regulatory enhancements • Time-boxed: 3 to 4 months • Waterfall SDLC First 2 Projects • Project 1 – Dec09 – 1X • Project 2 – June10 – 2.3X

9. Definition: Re-work Re-work – AD time to fix and testing time to report and re-test System Testing and Production defects

10. Definition: Static Testing Static Testing - Testing of a component or system at specification or implementation level without execution of that software, e.g., reviews or static code analysis.

11. Static vs. Dynamic Testing From System Testing with an Attitude, p.14,by Nathan Petschenik

12. Testing Defect Analysis • When was the defect introduced (SDLC phase) • When was the defect discovered (testing phase)

13. Points of Origin for Defects2 • Requirements • Design • Code • User documents • Bad fixes (new defects due to repairs of older defects) 2Software Engineering Best Practices: Lessons from Successful Projects in the Top Companies by Capers Jones

14. Defect Discovery Phases • Static Testing • Document walk-throughs & reviews • Pre-testing • Test Plan review • All before Build is complete • System Testing • Production

15. Tracking the Cost of Re-work • Defect Fix Time • Re-test Time COR (Cost of Re-work) – calculated by multiplying time x rate – cost per defect

16. Re-work time during Test Execution is tracked as 2 Components Time to Fix a Defect Time to Re-test Defect Fixes Testing Rework Workflow

17. Project 1 - Defect Fix Cost

18. Issues from first 2 projects • High % of project = Cost of Re-work

19. COR = 23 to 29% Project

20. Issues from first 2 projects • High % of project = Cost of Re-work • High % of Requirements & Design (R&D) defects found later

21. 20-30% R&D Defects found later

22. Issues from first 2 projects • High % of project = Cost of Re-work • High % of Requirements & Design (R&D) defects found later • DDP (Defect Discovery %) – slightly better than average

23. Defect Discovery Percentage (DDP) DDP = % of Total Defects discovered before implementation Provides a metric to evaluate the overall effectiveness of testing. 90% - Good 95% - Very Good

24. DDP = average +

25. Process Improvement Steps • Developed framework in Quality Center to track COR • Presentations to business partners & executives • Lively discussion during presentations • AD appointed a new AD Lead • AD improved their internal code review • More emphasis on and participation during reviews, walk-throughs – Static Testing

26. Process Improvement Steps • More collaboration between AD and testing teams • Increased emphasis on unit testing • Risk-based testing approach • Improved regression work • Staff increased in domain knowledge

27. Projects 3 & 4 – Process Improved • Project 1 – Dec09 – 1x • Project 2 – June10 – 2.3x • Project 3 – May11 – 3.4x • Project 4 – June11 – 2.3x

28. Static Testing emphasized

29. System Testing Defects

30. % R&D Defects found later

31. DDP (Defect Discovery %)

32. COR as % Project Cost

33. Defect Discovery Rate DDR = % of Total Defects discovered during an SDLC or Test Phase • Static Testing • System Testing • Transition Goal - Trying to find out if we are finding defects earlier.

34. DDR Trends As DDR for Static Testing increases, DDR for System Testing and Production Defects decreases ----------------Coding-----------------

35. When is Unit Testing complete? “A necessary criterion for completion of developer testing is that the developers believe that the software is ready to be delivered to the users on the day it is delivered to the system-test team.” From System Testing with an Attitude, p. 33,by Nathan Petschenik

36. Today’s Discussion • Defect metrics – powerful story • Static Testing works • It is a collaborative effort • The positive effect ripples into Unit Testing, System Testing, UAT, and implementation • No shortcuts here – influence your project to do it right

37. Some Final Thoughts • Training needed for our staff (testing, AD, business, PM community) - how to be more effective in document reviews and walk-throughs. Perspectives • Tester – static testing > a paradise (a new maturity level)

38. Questions & Answers • ???