INFO 636 Software Engineering Process I Prof. Glenn Booker

1. INFO 636Software Engineering Process IProf. Glenn Booker Week 9 – Quality Management INFO636 Week 9

2. Software Quality Management • The PSP focuses on managing defects to produce a quality product • This, in turn, helps improve productivity • Productivity still falls as the size of a system increases, due to its increasing complexity • To improve further, must improve your process quality INFO636 Week 9

3. What is Quality? • Crosby considered software quality to be “conformance to requirements” • Meeting user needs is often key to determine quality • We can distinguish between the quality of the product created, and the quality of the process used to create it INFO636 Week 9

4. Product Quality • A software product must provide the functions defined in its requirements, and must be a functioning application • Beyond that, how sensitive the user is to other aspects of usability, reliability, compatibility, etc. varies wildly • Poor quality products tend to have a lot of focus on testing to find defects INFO636 Week 9

5. Process Quality • Process quality focuses on meeting the developer’s needs • A quality process supports creating good products • Process consistency and repeatability are generally critical • Efficiency and usability are less obvious objectives INFO636 Week 9

6. Cost of Quality • Quality is a big cost issue for software • Each defect has costs for • detecting it • finding the source • determining a solution • fixing the problem (req’ts, design, and/or code) • reviewing the fix • testing the fix to see if it works • testing the fix to check for regression problems • updating documentation to reflect the fix INFO636 Week 9

7. Cost of Quality • The relative cost of finding and correcting a defect increases through the life cycle • During requirements or design phases, a defect has a relative cost of about 1 • Coding and testing phases, cost 10-40 • After release to the customer, cost 100-1000 INFO636 Week 9

8. Cost of Quality • Many of you also noted that the time to fix defects found in testing or later is much higher than those found earlier (see pp. 276-7) • PSP experience has shown that reviews are twice as efficient at finding defects as testing INFO636 Week 9

9. Testing Schedules • Many organizations focus on getting into system testing quickly, as a sign of progress • However, more time in requirements and design would produce higher quality code which needs less testing time INFO636 Week 9

10. Cost of Defect Removal • Typical developers inject (create) about 100 defects per thousand LOC • About half are found by compiler • The rest need to be found by inspection or testing • Using the PSP, about 70% of defects entering inspection are found INFO636 Week 9

11. Cost of Defect Removal • Inspection and review average about ½ hour per defect, whereas testing averages 8-10 hours per defect – so the cost benefit of inspection becomes clear INFO636 Week 9

12. An Example • Consider a 50,000 LOC project • About 5000 defects are initially created, but 2500 are found by the compiler (and are presumably fixed immediately) • Without reviews or inspections, those 2500 defects would take at least 2500*8 = 20,000 hours to find and fix using testing alone INFO636 Week 9

13. An Example • Using inspections, we would expect to find 1750 of those defects, and fix them in ½ hour each for 875 hours • The remaining 750 defects would be found by testing, and fixed in 750*8=6000 hours • Total defect removal time with inspection and test is 6,875 hours, versus 20,000 hours for testing alone • That saves $1.3M at $100/hr INFO636 Week 9

14. Cost of (Low) Quality • Poor quality can be measured to determine its impact on the project • Failure cost = the cost of finding and repairing failures • Appraisal cost = the cost of conducting reviews (called), and testing which finds no defects • Prevention cost = the cost of preventing the failures from occurring INFO636 Week 9

15. Cost of Quality (COQ) • For the PSP we simplify these to • Failure cost = time for compile and test • Appraisal cost = time for reviews and inspections • Prevention costs are low, since we have few projects to examine for defect trends • Based on this, define COQ measures INFO636 Week 9

16. COQ Measures • Failure COQ = 100*(compile time + test time)/(total devel time) • Appraisal COQ = 100*(design review time + code review time)/(total devel time) • Total COQ = Failure COQ + Appraisal COQ • Appraisal cost as % of total quality = 100*(appraisal COQ)/(total COQ) • A/FR ratio = Appraisal COQ/(Failure COQ) INFO636 Week 9

17. COQ Measures • The purpose of these measures is to help look for trends in quality over several projects • Hope to see higher ‘appraisal cost as % of total quality’ over time • This should lead to lower defect rates for later projects INFO636 Week 9

18. Quality Strategy • Important to keep product and process measures separate – both are important • Avoid productivity measures by themselves – they can help with planning, and be compared to other measures, but don’t use them as a sole objective INFO636 Week 9

19. Process Yield • Total process yield is helpful – the percent of defects removed before test • You can also evaluate yield for every life cycle phase, but only after hidden defects have been later discovered • The life cycle is a filtering process • Each phase has some kind of step to help find and remove defects to keep from feeding them into the next phase INFO636 Week 9

20. No One Answer • The PSP is a personal process for many reasons – one is that there is no one best way to find and remove defects • Need to look for the best ways for you to improve your work • Then reevaluate your strategy periodically INFO636 Week 9

21. Benchmarking • It can be helpful to compare your product and process measures to other people’s, or to industry standards • For example, plot the A/FR ratio over time • It should increase (p. 288) • Or look at A/FR versus the yield • Or Test defects per kLOC versus A/FR INFO636 Week 9

22. Benchmarking • Productivity can be compared to other measures, such as yield, to look for trends • Yield acts like the filter part of the life cycle process • Each step’s yield reduces the number of defects to be found later, so a small improvement in yield can be significant INFO636 Week 9

23. Injection and Removal • The key to understanding your software development process is to balance the injection of new defects with activities to detect and remove them • To achieve greater control, need to examine the factors which cause defects to occur INFO636 Week 9

24. Defect Causes • Education – didn’t know how to do it • Communication – weren’t informed about something • Oversight – forgot to do something • Transcription – knew what to do, but made a mistake • Process – the process misled your actions INFO636 Week 9

25. Yield Management • While the overall process yield doesn’t care if you have 80% review yield, and 40% test yield, or 40% review and 80% test, the costs and A/FR ratio of those situations are quite different • This is why so much emphasis is placed on the quality of early reviews (see recommendations on pp. 298-300) INFO636 Week 9

26. Defect Prevention • Finding and fixing defects is purely defensive • To be more proactive, need to prevent defects from being created • Analyze the types of defects which occur frequently in your experience • Determine the causes of common defect types (see slide 24 again) INFO636 Week 9

27. Defect Prevention • Then look for ways to prevent those defects from being made again, e.g. • Improve review checklists • Get additional instruction • Conduct more detailed design • Get independent reviews • And change your processes accordingly INFO636 Week 9

28. Defect Prevention • The critical follow-up is to re-examine your defects later, and see if your changes helped or not • Incidentally, defect prevention and continuous process improvement are the highest levels of process maturity • CMM and CMMI level 5 activities INFO636 Week 9