SMU CSE 8314 Software Measurement and Quality Engineering

1. SMU CSE 8314 Software Measurement and Quality Engineering Module 17 Statistical Process Control

2. Statistical Quality Control Quality Control • Developed during World War II • Used statistical methods to assure quality without needing to test each sample (each product being produced) • These techniques became known as SQC Quality Assurance and Engineering • The statistical techniques have become more sophisticated and have been used in newer quality programs

3. SQC Conceptas applied in Manufacturing 1) Set a quality standard for each important product characteristic 2) Plan to meet the standard 3) Control the factors that cause the quality problems 4) Measure conformance to the quality standard 5) Act when the process is out of line with the standard

4. 1) Set a Quality Standard • Target • Tolerance Example: Pencil Diameter: 0.25” ± .001” We saw this in the Six Sigma specification limits

5. 2) Plan to Meet the Standard Original concept: plan the following • Manufacturing process • Raw materials • Training • Staff expertise In later years, this was added: • Design of the product

6. 3) Control the Quality Factors • Material purity • Level of training • Temperature • Pressure • Design • This is of greater concern with more complex products • Etc.

7. Upper Control Limit (Action Threshold) Upper Spec Limit Lower Spec Limit Lower Control Limit (Action Threshold) 4) Measure Conformance to Quality Standards x x x x x x x x x x x x x x x x x x Goal x x x x x x x x x x x x x x x x x x x x

8. For Product Development Processes, Limits may Vary Over Time

9. 5) Take Action when Out of Line • Thresholds are established so you can head off problems before they become serious • These thresholds assume a relatively slow degree of variance • Actions generally involve the factors that are known to cause quality problems • In its initial form, SPC did NOT do extensive root cause analysis, although that came as a later refinement

10. Some Key Assumptions about SQC • Raw materials and other inputs are well defined • Quality standards are well defined • Specification limits are reasonable (i.e., some variance in measurable characteristics is reasonable) These may not apply to typical software projects

11. Applying this to Software Do Not • Apply normal distribution • Assume well defined spec limits (?) • etc. Do • Characterize what is acceptable • Measure actual results • Set thresholds where feasible • Act when not acceptable

12. Applying SPC to Software

13. Cho’s Quality Cycle • This is an application of SPC methods to software • Accomplished by analyzing the factory situation and drawing software analogies • One might quibble with Cho’s analogies, but the principles are valuable [Cho saw software as a production process rather than a design process]

14. Manufacturing Quality Cycle 1 9 2 8 3 Quality Character- istic 7 4 6 5

15. Factors that Affect Quality - I 1) Product concept formulation • Same with software [but Cho said “not applicable”] 2) Specify the quality characteristics • Same with software [but Cho said “not applicable”] 3) Design the product • Same with software [but Cho said “not applicable”]

16. Factors that Affect Quality - II 4) Engineer the (manufacturing) process • Not a big deal for software • But we do engineer the development process (at an earlier stage) • [Cho saw this as software product concept formulation] 5) Purchase/acquire equipment and raw materials. • Same with software tools [but Cho said “SW Quality Characteristic & Specs”]

17. Factors that Affect Quality - III 6) Installation and training • Same with software tools & methods • Cho said this corresponds to “Software Design” 7) Manufacture • Corresponds to software engineering • Cho said “Software Code and Test”

18. Factors that Affect Quality - IV 8) Inspect for quality • Same with software [Cho agrees] 9) Sell, use, get feedback • Same with software [Cho agrees] 10) Repeat the cycle

19. Statistical Sampling to get Quality Information Defect Rate () where D = Number of defects N = Number of units produced  = D / N

20. Probability of Defects • The probability of finding x defects in a sample of n units is: This is known as the binomial distribution. n  b(x ) =  =x(1-)n-x  x 

21. Definitions for Previous Slide Mean = µ = n* Variance:

22. Estimating Defects • If you know the defect rate for a particular sample, say d defects in a sample of size n: d 0 = -------- n • Then you can estimate the number of defects in the entire population. This is known as the 100*c% confidence interval, where 0  c  1

23. Confidence Interval     \/0*(1-0) \/0*(1-0)   0 - tn-1,/2 ----------------- , 0 + tn-1,/2 ----------------    n n  where the tn-1,/2 is called the value of the Student t-distribution at n-1 degrees of freedom and  = 1-c is called the risk factor. Example: if you chose c = .5 (50% confidence) the formula would give you an interval over which you could have 50% confidence of being within.

24. Student t-distribution tn-1,/2 can be taken from tables or, if n > 30, it can be derived from the area under the normal distribution curve. area = /2 tn-1,/2

25. Procedure • Set a Goal for Defect Rate () • Perhaps something like  < 0.01 • Sample enough values to get a high confidence level that the defect rate is close to the sample • Perhaps 90% confidence • Act to change the process until sample rate meets the goal • See Cho for more details

26. For Software ... • This will only work if the units are small enough that there are many of them • Example: lines of code or functions or requirements or objects or screens BUT • We don’t normally define quality characteristics for early steps of the software process (requirements, design, etc.)

27. Software Counterpart • Define thoroughly what the software must do • Define a model to thoroughly analyze the problem to be solved • Define specific product requirements based on the above: • Requirements Specifications • Test Requirements • Design the process to fit the software & the development methods to be used

28. Requirements Specifications • Functions to be performed • Things the software must do • Quality characteristics to be achieved • Performance, timing, etc. • Inputs • Variables, data, etc. (including ranges) Rigor at this level is hard to achieve and maybe harder to get people to do

29. Test Requirements • Units to be Tested (code modules, lines of code, functions, etc.) • Defectiveness goals • Sampling techniques to be used • Testing methods and plans • How to demonstrate or otherwise verify un-testable requirements

30. Requirements Modeling Do Test Development and Software Development in Parallel Testing Requirements Test Design Test Code Coding Software Requirements Software Design Software Coding Design Modeling Testing

31. Some Things you Doto Design Tests • Decide on sampling approach based on units to be tested • Review definition of defect for each unit • Decide on a sampling plan • Construct random inputs based on the plan • Determine how to analyze the results • Determine statistical inferences to be used

32. Sampling Techniques for Software • Sample each software module • Sample each input variable • Sample the boundary values of each input • Sample several invalid values for each input • Weight the samples based on importance of module or function being implemented or frequency of use

33. The Key to Making Any of this Apply to Software The Goal is to Find and Prevent Defects It is NOT any of the following: • To identify the best & worst performers • To make someone look good • To compare performance of individuals or teams • To get accurate data

34. Summary • Industrial programs involve • Measures of quality, such as • Six sigma • Zero defects • Statistical measures • Methods of quality improvement, such as • Six sigma programs • Zero defects programs • Statistical quality control

35. Summary (continued) • These programs are applied to real projects to • Measure quality • Predict quality • Manage quality

36. References • Cho, C. K., An Introduction to Software Quality Control, New York: John Wiley & Sons, 1980. • Harry, Mikel J. and J. Ronald Lawson,Six Sigma Producibility Analysis and Process Characterization, Motorola University Press, Addison-Wesley, ISBN 0-201-63412-0

37. References (continued) • Schulmeyer, G. Gordon. Zero Defect Software. McGraw Hill, 1990. ISBN 0-07-055663-6. • Schulmeyer, G. Gordon and James McManus. Handbook of Software Quality Assurance, Second Edition(especially chapter 17). Van Nostrand Reinhold, New York, 1992. ISBN 0-442-00796-5.

38. END OF MODULE 17