SMU CSE 8314 Software Measurement and Quality Engineering

1. SMU CSE 8314 Software Measurement and Quality Engineering Module 33 Quantitative Process Management

2. Outline • Basic principles of QPM • Statistical process control techniques • Setting and using thresholds

3. Problems just seemed to creep up on us until they overwhelmed us We knew we were behind, but we didn’t know it was that bad! Quantitative Process ManagementTypical Symptoms of a Problem

4. Quantitative Process ManagementSymptoms of Another Problem The support costs are huge! Why did we ship that software with so many bugs in it? • We never expected there would be so many when we released it.

5. Quantitative Process Management Other Characteristics & Symptoms • You make decisions without factual data to justify them • Usually because you have not collected the right data • Or else you have not analyzed your data effectively Decisions may be inappropriate or harmful.

6. Quantitative Process Management Other Characteristics & Symptoms (continued) • You overreact to normal variations in performance • Or you allow things to get out of hand before taking action You don’t know the difference between a warning sign and a harmless variation

7. Quantitative Process Management Other Characteristics & Symptoms (continued) • You can determine where the problems are but cannot assess which ones are the most serious You may solve the unimportant problems and overlook the ones that matter.

8. Quantitative Process Management Purpose To control the performance of the software project quantitatively. (Software process performance represents the actual results achieved from following a software process.)

9. Quantitative Process Management Definition • Establishing quantitative goals for the performance of the project’s defined software process, based on knowledge of the capability of that process; • Taking measurements of the process performance; • Analyzing these measurements; and • Making adjustments to maintain process performance within acceptable limits.

10. Why Do QPM? • To enhance your ability to make informed decisions about what to do and how to prioritize your actions • To set goals based on fact rather than opinion • To give you the facts to determine whether changes actually improve the process • To make you more competitive

11. How Do You Get There? 1) Characterize the process • Establish a capability baseline 2) Stabilize and manage the process • Improve consistency in the baseline so you have an expected range of values 3) Improve the process to achieve goals • Achieve a desired range of values

12. How it is Done in Practice Step 1 - Characterize Step 2 - Stabilize and manage Step 3 - Improve to meet goals Each step is ongoing once started, but you should do them in order. This will help achieve the most effective results.

13. Step 1Characterize the Process 1) Observe behavior 2) Establish a capability baseline • Average behavior • Variations in behavior 3) Measure actual behavior 4) Revise baseline as needed 5) Repeat until the baseline represents normal behavior Step 1 is a matter of measuring, observing, and revising baselines until they represent normal behavior

14. Capability Baseline Typical Characterization

15. Warning • Too often, organizations set limits based on what they wantinstead of what they are capable of doing. • An essential concept for the first step is to characterize your actual capability. Then, in future steps, you can find ways to improve so you can achieve what you want.

16. Step 2Stabilize the Process • Observe variance over time • Seek sources of variance and improve the process to reduce them • Over time, variance is reduced and baselines become stable • Once behaviors are reasonably consistent, the process is said to be stabilized and you can establish an expected range of values

17. Typical Un-stabilized Process Note that average moves significantly, and individual projects have wild swings.

18. Typical Stabilizing Process Average is more consistent, and individual projects have less severe swings.

19. Expected Range of Values A Stable Process

20. With a Stable Process You Can ... • Characterize expected behavior • Averages • Variations • Establish an expected range of values • Identify significant deviations from normal behavior • These are signs of a problem that must be addressed, normally a “special cause” problem

21. Types of Variations • Special Cause variations - a specific project or product has a problem that causes it to vary substantially from the norm • General Cause variations - problems inherent in the organization, the culture, or the process that are not specific to any particular project

22. Step 3Improve to Meet Goals 1) Observe capability of stabilizing process 2) Establish capability goals • Desired average behavior • Desired level of variations of behavior • I.e., a desired range of values 3) Identify ways to improve behavior 4) Move capability toward the goal 5) Repeat until the behavior achieves the goals

23. Quantitative Process Management Goals from the SEI CMM/CMMI 1) The quantitative process management activities areplanned • Planning occurs before the project starts 2) The performance of each project’s defined software process is controlled quantitatively … continued

24. Quantitative Process Management Goals from the SEI CMM/CMMI (continued) 3) The process capability of the organization’s standard software process is known in quantitative terms

25. What do we Mean byProcess Capability (Goal 3)? • Capability is the range of values normally achieved by our process • Including a mean or average • And an acceptable level of variation from that mean In other words, we know what we are capable of and how we normally perform - expected range of values.

26. What do we Mean byControlled Performance (Goal 2)? • We can measure process performance against a capability standard • We can determine when actual performance is outside of an accepted range • We can correct performance to get it back within range

27. Typical Graph ofProcess Capability

28. Some Variability is Normal Performance between limits represents normal variability.

29. “Too Perfect” Driver - No Variation “Typical” Driver - Normal Variation “Dangerous” Driver - Too Much Variation What is Normal Variability?

30. Causes of Variability • People • They vary somewhat from day to day • Methods • They may not always work equally well on different applications • Machines • They may have alignment variations, etc. • Material • May vary from batch to batch • Environment • Changes in the weather, for example

31. Knowing Capability means ... • Knowing what you can do • Knowing what you cannot do • Knowing what degree of variation is normal and what is not

32. Typical Graph of aProject vs Capability - Month 4

33. Starting off in good shape Typical Graph of aProject vs Capability - Month 4

34. Typical Graph of aProject vs Capability - Month 6

35. Upward trend suggests a potential problem Typical Graph of aProject vs Capability - Month 6

36. Typical Graph of aProject vs Capability - Month 8

37. Crossing the limit shows a definite problem Typical Graph of aProject vs Capability - Month 8

38. Because We Know our Capability We Can Spot Problems Early • We avoid the tendency to think things are only a little worse than last time • Or to overreact to normal variation • We can see our problems when we have time to react and take action • We can more readily justify actions because the data show the trends and the risks

39. See the trend. Don’t wait for the crisis.

40. Note • A process whose capability is known is not necessarily a good process • It may not meet acceptable performance or quality or cycle time standards • Performance may be highly unstable • It may be inefficient or poorly designed • But you know what it is capable of in quantitative terms • So you know if you are performing at the level permitted by your process

41. References Benno, Stephen A. and Dennis J. Frailey "Software Process Improvement in DSEG -- 1989 -1995“, Texas Instruments Technical Journal, March-April 1995. Hudec, et. al., “Experiences in Implementing Quantitative Process Management”, Proceedings of 1996 SEPG Conference (SEI/IEEE).

42. END OF MODULE 33