Readiness Index – Is your application ready for Production?

1. Readiness Index – Is your application ready for Production? Jeff Tatelman SQuAD October 2008

2. Agenda • Building a Metrics Program • Typical Test Organizations • Readiness for Change • Metrics and Productivity • Production Readiness • Introduction & Objective • Four Key Metrics • Reporting

3. Typical Test Organizations • No metrics collected • No collection of requirements • Limited formal reporting on project status • Some central test repositories but not dash boarding results • Minimal effort towards process improvement

4. Dimensions Of Readiness • Motivation • Investment • Skills • Education • Culture • Support Staff • Aids/Maturity • Process Maturity

5. Getting Started • Set Measurement Objectives • Select Measures • Develop Measurement Program • How to Use the Results

6. Set Measurement Objectives • Determine Approach • Which measures will be included as a priority • Focus on a few to start with to get ‘quick wins’ • Determine Scope • One application? • One project? • Determine Method • Identify best method to track and analyze agreed metrics • Determine Timescale • Full project lifecycle? • Time boxed approach, eg. 1, 3 or 6 months • Determine Deliverables • Report, audience, next steps

7. Metrics and Productivity • Strategic • Is it to increase customer satisfaction? • Is it to achieve CMM level 2? • Is it to achieve industry standards? • Tactical • Is it to improve vendor delivery? • Is it to increase timeliness of system delivery? • Operational • Is it to reduce defects found in testing? • Is it to improve requirements definition?

8. Measuring Unit Examples • Strategic • Quality • Customer satisfaction • Timeliness (delivery) • Personnel • Industry benchmarking • Tactical • Estimating and planning • System quality • System delivery cost/time • Budget • Productivity • Cost of quality • Operational • Schedule tracking • Effort tracking • Defects • Problem resolution • System availability

9. Develop Measurement Program • Establish Specific Measurement Objectives • Define a Critical Metrics Set • Select Measures to Support the Metrics Set • Put Collection Mechanisms into Place • Determine Timing of Data Collection • Establish Mechanism for EvaluatingResults • Communicate Results • Establish Process for Future Planning Based on Metrics

10. How To Use The Results • Conduct root cause analysis of the data • Identify areas which would have the biggest impact and look to how to improve • Implement procedures to continually improve the processes • Revisit original list and determine what to address next

11. Key Points to Remember • Don’t try to measure too much • Understand the goals of your team before you determine what to measure • Determine which metrics support these goals • Don’t let your metrics define the behavior of the team • Change or choose a set of relative metrics that can not be manipulated • Monitor and identify trends , define areas for improvement Allan Page Measurement that Matter, Better Software October 2005

12. Production Readiness Based On Mike Ennis’ Presentation at Star West 2006

13. How do you know when a product is ready to ship? • Quality Metrics • Customer Commitment • Release Dates Are Preset • Indefinite Testing • Adequate Test Coverage • Time & Resources • Release Criteria

14. Release Criteria • No open critical or high defects • Minimal number of medium & low defects that have been approved by Senior Management • Product is able to run for 72 consecutive hours • No open installation or configuration issues • All pre-defined performance criteria has been met

15. Determining What to Measure • Test Case Execution Percentage • The percentage of tests that have been attempted during the test cycle • Test Case Success Percentage • The percentage of tests that have passed during the test cycle • Number of Unresolved Defects • The number of open defects that are currently logged against the product • Defect Arrival Rate • The number of defects found in a given day, week or build

16. Readiness Metrics Objective • Provide management an overall picture to assess if the project is ready to be placed in production. • Evaluate an application go / no-go production readiness status by • Tracking test case execution and defect metrics over time • Calculating the Production Readiness Index based on the above metrics

17. Test Case Spreadsheet Example

18. Defect Spreadsheet Example

19. Setting Range For Each Metric • Test Case Execution Percentage • 10 = 100% • 9 = 90-99% • 8 = 80-89% • 7 = 70-79% • … etc • Defect Arrival Rate This Week • 10 = 0 • 9 = 1-2 • 8 = 3-4 • … etc

20. Creating and Analyzing Readiness Example RATING RANGES Defects Arrival Rate 9 10 = 0, 9 = 1-2, 8 = 3-4, 7 = 4-5, 6 = 5-6, 5=6-7, 4=… Total Open Defects 9 10 = 0-5, 9 = 5-10, 8 = 11-15, 7 = 16-20, 6 = 20-25 … Test Success % 7 10 = 100%, 9 = 99-97%, 8 = 96-94%, 7 = 93-90%... Test Completion % 6 10 = 100-95%, 9 = 94-90%, 8 = 89-85%, 7 = 84-80%.. TOTAL RATING 31 Green = 35-40, Yellow = 29-34 Red < 28

21. Spider Chart Example

22. Management Graph Example

23. Readiness Index Example

24. Final Thoughts • Managing the risks • Understand the relationships between the metrics • Learn to anticipate and minimize the risks before they happen • Always know the information behind the data • Are you ready to release? • Redefine your Release Criteria using the individual/overall rating scale • Use colors for presentation & effectiveness • Let the data speak for itself

25. Key Attributes Of a Good Measure • Usefulness • Does it tell us what’s happening, wrong or need to know? • Trustworthiness • Does not provide false indications • Timeliness • Does it indicate the health of a system...advanced warning? • Simplicity • Too difficult to read, it will get abandoned • Accessibility • Must be visible, easy to get to

26. Questions jeffcolorado@juno.com