Software Process Performance Measures

1. Software Process Performance Measures James Over Software Engineering Institute Carnegie Mellon University Pittsburgh, PA 15213-3890

2. Purpose • Why are you interested in process improvement? • Hopefully for the process performance benefits. • If so, process performance measurement is a key concern. • Many of the examples in this presentation are from the Team Software ProcessSM , however the concepts are broadly applicable. SMTeam Software Process is a registered service mark of Carnegie Mellon University

3. Team Software Process • The Team Software Process (TSP) is an integrated set of practices for developing software. • TSP is a process-based solution to common software engineering and management issues. • cost and schedule predictability • productivity and product quality • process improvement • Unlike other methods, TSP • teams are self-directed. • emphasizes measurement and quality management. • provides immediate and measurable benefits. • accelerates CMMI-based improvement.

4. TSP Performance Summary -1 * From a study of 20 projects in 13 organizations conducted in 2003 ** Of the unsuccessful projects, average schedule error was 222%

5. TSP Performance Summary -2 * From a study of 20 projects in 13 organizations conducted in 2003

6. An analysis of 20 projects in 13 organizations showed TSP teams averaged 0.06 defects per thousand lines of new or modified code. Approximately 1/3 of these projects were defect-free. These results are substantially better than those achieved in high maturity organizations. TSP Performance Summary -3 Source: CMU/SEI-2003-TR-014

7. TSP-CMMI Overall Coverage

8. Topics • Process management concepts • TSP measurement framework • Performance measures

9. SEI Process Management Premise • “The quality of a software system is governed by the quality of the process • used to develop and evolve it.” • - Watts Humphrey

10. Managed Process • The CMMI defines a managed process as a process with the following characteristics. • a performed process that is planned an executed in accordance with policy • the process employs skilled people who have adequate resources to produce controlled outputs • it involves relevant stakeholders • it is monitored, controlled, and reviewed • it is evaluated for adherence to its process description

11. Process Management • Process ownership: key responsibilities for designing, establishing, and implementing the process and the mechanisms for measurement and corrective action is assigned. • Process definition: the design and formal documentation of the components of the process and their relationships. • Process control: the function of ensuring that the process output meets specifications including • measurement • control variable(s) • feedback loop(s) • defect detection, correction, and defect prevention Source: Quality Process Management by Gabriel Pall

12. Process Management Concept Input Output Control Work Process

13. Example System test yield Control Process Yield Input Output Inspection Process Review Rate

14. Process Management Conclusions • A defined process is a prerequisite for process management. • The enactment of the process should not differ from the defined process in any substantive way. • The key determinants of process performance must be instrumented and measured. • Failure to measure or limited measurement scope can lead to sub-optimization or “process tampering” • The process and measures should be designed to support process management from the start.

15. Topics • Process management concepts • TSP measurement framework • Performance measures

16. Process Measurement Issues • Some common process measurement issues… • substantial variation in measurement reporting requirements across development groups and suppliers • few measures of quality • standards emphasize derived measures instead of common base measures • inability to summarize, aggregate, drill-down, extend • cannot benchmark or make comparisons • limited use as a management indicator • lack of accountability • Measurement framework “literally” tied to CMMI process areas and the examples of derived measures from CMMI.

17. Measurement System • Design and a “systems” approach solves many measurement issues. • Define a few common base measurement categories and establish standards for the most used instances. • Develop a measurement framework that relates the base measures to the key elements of software process work. • Create derived measures from the standard base measures. • Identify process performance models and benchmarks that predict future performance. • Integrate into monitoring and decision-making processes.

18. Base measurement categories Example derived measures Estimation accuracy Prediction intervals Productivity Cost performance index Planned value Earned value Predicted earned value Defect density Defect density by phase Defect removal rate by phase Defect removal leverage Review rates Process yield Phase yield Failure cost of quality Appraisal cost of quality Appraisal/Failure COQ ratio Percent defect free Defect removal profiles Quality profile Quality profile index … TSP Measurement Framework -1 Size Effort Defects Schedule

19. A model of key process elements and their relations provides a context for the base measures. processes and phases projects and sub-projects products and parts teams and team members tasks period (week, month, etc.) The model facilitates analysis aggregation and drill-down queries and views scalability Process Project Team Product Tasks Period TSP Measurement Framework -2

20. Estimated and Actual Size • Size is a measure of the magnitude of the software deliverable, e.g. lines of code or function points. • Size is estimated and actual size is measured for each component. • Five size accounting categories are used. • Base • Modifications to the base • Deletions from the base • Added or new • Reused • Size data are used to • estimate effort • track progress • normalize other measures

21. Effort is a measure of time on task. The TSP effort measure is called a task hour. Task hours are estimated and measured by process phase task day or week How many task hours are there in a 40 hour week? About 15 to 20. Estimated and Actual Effort

22. Schedule has two components resource availability task completion dates Planned task dates are calculated from estimates of resource availability and planned task hours. Actual date completed is recorded as tasks are finished. Actual resource availability is derived from actual task hours. Estimated and Actual Schedule

23. Defects are the measure of quality. Estimates of the number of defects injected and removed. A count of the actual number of defects injected and removed. Defect data includes component phase injected phase removed Estimated and Actual Defects • Definition: a work product element that must be changed, after it was completed, in order to ensure proper design, implementation, test, use, or maintenance.

24. Topics • Process management concepts • TSP measurement framework • Performance measures

25. TSP Performance Measures • The most often used TSP performance measures are: • Planned value, earned value, predicted earned value • Planned and actual task hours • Estimation error • Growth • Defect density • Percent defect-free • Quality profile and index • These measures support planning and tracking. • Combined with historical data and/or benchmarks, these measures also support process performance modeling.

26. Process Performance Models Process performance model Predicted project performance Project data Historical data and Benchmarks

27. Example: Quality Profile • Project data • Time in design, design review, coding, and code review • Defects found in compile and unit test. • Product size Process performance model Quality Profile Predicted value Likelihood of post system test defects • Benchmarks • Development time ratio criteria • Defect density criteria

28. Quality Profile Benchmarks • These software quality benchmarks predict post-development defects. • Modules that meet these criteria were found to be largely defect free in system test and after deployment.

29. Quality Profile • The quality profile is a process performance model that provides an early warning indicator for post-development defects. • The quality profile uses the five software quality benchmarks. • Satisfied criteria are plotted at the outside edge of the chart. Inadequate design review time results in design defects escaping to test and production. High quality component Poor quality component

30. Using the Quality Profile

31. Quality Performance Index vs. Post-Development Defect Density Post-Development Defect Density Quality Performance Index Quality Performance Index • The Quality Performance Index is the product of the five parameters in the quality profile. • QPI predicts the likelihood of post-development defects in a system.

32. Conclusion • Measurement and process management are inseparable, you should incorporate measurement in your initial processes. • A common problem with software process measurement is a lack of integrated, well designed measurement systems resulting in unnecessary complexity and usability issues such as lack of scalability and extensibility. • Process management can be successfully applied to the software process with a few, simple, derived measures that are integrated into a measurement framework.