How Much Quality is Enough?

1. How Much Quality is Enough? Jon Bach Manager, Corporate Intellect Quardev Laboratories -- Seattle jonb@quardev.com QAI, November 2006

2. My message Testing may be a waste of time. (c) 2006 Jon Bach / Quardev, Inc.

3. Dynamic Quality Paradigm Perfect Item A Item B Further improvement is necessary. Unacceptable quality Awful (c) 2006 Jon Bach / Quardev, Inc.

4. Dynamic Quality Paradigm Perfect Item A Item B Further improvement is necessary. Unacceptable quality Awful (c) 2006 Jon Bach / Quardev, Inc.

5. Dynamic Quality Paradigm Further improvement: waste of resources. Perfect Unnecessary quality quality bar Item A Further improvement is notnecessary. quality bar Item B Further improvement is necessary. Unacceptable quality Awful (c) 2006 Jon Bach / Quardev, Inc.

6. My Message (rephrased) You may be striving for more quality than you need. (c) 2006 Jon Bach / Quardev, Inc.

7. Bio • Corporate Intellect Manager and Senior Test Consultant for Quardev Laboratories (www.quardev.com) – a Seattle test lab specializing in rapid, exploratory testing.  • Most known for being co-inventor (with brother James) of Session-Based Test Management -- a way to manage and measure exploratory testing. • In my ten-year career, I have led projects for many corporations, including Microsoft, where I was a test manager on Systems Management Server 2.0 and feature lead on Flight Simulator 2004.  • Writer, philosopher, and test practitioner.  • President of the Conference for the Association for Software Testing (Seattle, 2007).  (c) 2006 Jon Bach / Quardev, Inc.

8. Agenda How can testers and test managers help project stakeholders know whether they are releasing software with too little quality or are unnecessarily striving for too much quality? • Four criteria to assess if your software is “good enough” • Three context-driven questions to ask when deciding if software is ready to deliver • A simple spreadsheet for measuring… “goodness”? (c) 2006 Jon Bach / Quardev, Inc.

9. G.E. in action Is this software good enough? Street Maps USA Professional Resume Plus Tarot / Lotto Magic Frog Frenzy Safari Exotic Classic Cars God Bless America Screen Savers 1000 Best Solitaire Games (c) 2006 Jon Bach / Quardev, Inc.

10. And the actual retail price of your showcase IS… $10.06 (c) 2006 Jon Bach / Quardev, Inc.

11. Assumptions • Testing is expensive • Testers can’t find all the bugs • Developers can’t fix all the bugs found • Customers are waiting for your software (c) 2006 Jon Bach / Quardev, Inc.

12. “Good enough” concept • A way of thinking about quality in terms of utility and economy • “To get a result that is good enough, although not necessarily the best” – Herbert Simon, 1978 Nobel Laureate economist • “setting an aspiration level which, if achieved, they will be happy enough with, and if they don't, try to change either their aspiration level or their decision.” • Alternative perspective to Six Sigma which “strives for near perfection” • 3.4 defects per one million “opportunities” -- chances for nonconformance, or not meeting the required specifications. (c) 2006 Jon Bach / Quardev, Inc.

13. Quality and value Quality • “value to `some person’” -- Weinberg • “degree to which a set of inherent characteristic fulfils requirements" -- ISO 9000 • “conformance to specifications”-- Crosby • “better management of design, engineering, testing and by improvement of processes” -- Deming Value • “what someone is willing to do (pay) to meet a need” -- Weinberg • “price on the open market" – neoclassical economics (supply vs. demand) • Formula: Benefits / Price • Formula: Quality Received / Expectations (c) 2006 Jon Bach / Quardev, Inc.

14. Key Ideas • In a market-driven economy, you can’t think about fixing a bug without also thinking about the cost to fix it. • If the answer to “Can we fix this?” is “Yes”, the next questions should be: “Should we fix this? Does it create value?” • “Good Enough” Thinking aims for the right quality at the right price -- not too little, not too much -- considering some contexts (c) 2006 Jon Bach / Quardev, Inc.

15. The problem • There is bad software in the world, blamed on processes (and people) that aren’t “certified” • Its clinical definition is limiting: “Stopping a search for alternatives by choosing the first option that exceeds one’s aspiration level.” • Considered a synonym for “mediocre” – reinforced by a social definition that connotes sloppiness: “Good enough for government work” (c) 2006 Jon Bach / Quardev, Inc.

16. The “Good Enough” Framework 1) Sufficient benefits 2) No critical problems 3) The benefits outweigh the problems 4) In the present situation, and all things considered, improvement would be more harmful than helpful The answer must be “Yes” to all four criteria (c) 2006 Jon Bach / Quardev, Inc.

17. Criterion #1 – Sufficient Benefits The software helps users solve problems or meet needs by: Increasing their productivity Providing entertainment Helping them compete in the marketplace Establishing / enhancing reputation Meeting standards (c) 2006 Jon Bach / Quardev, Inc.

18. Criterion #2 – No critical problems The software should not exhibit anything that is deemed by stakeholders to be “critical”, which could include: Embarrassing typos Legal issues Failures or faults Poor feature set Localization: insults (c) 2006 Jon Bach / Quardev, Inc.

19. Criterion #3 – Benefits outweigh problems Risk analysis – what’s the likelihood of something bad happening, and how bad would it be? Customers don’t notice (or forgive) the faults The bugs are all low severity The feature set is competitive and appropriate It’s awesome (fun, cool, timely, useful) If sued, we can make enough back to afford it (c) 2006 Jon Bach / Quardev, Inc.

20. Criterion #4 – more <x> is more harmful than helpful If we keep testing or designing… …we’ll miss our ship window …we’ll break our budget …we’ll lose market share …we’ll go out of business …staff will quit / revolt / burn out (c) 2006 Jon Bach / Quardev, Inc.

21. When we use it • If prone to changing requirements, behind schedule • If your project is understaffed, with ill-defined test processes • If time-to-market is crucial • If people may not care if the quality is “perfect” • If it can help reach an acceptable level of RISK • Replaces “blind perfectionism with vigilant moderation” • Alternative to counting LOC -- “no more than 3.4 defects per million opportunities in any process, product, or service” (c) 2006 Jon Bach / Quardev, Inc.

22. Contextual considerations • Good enough for whom? • Good enough for what? • Good enough for when? (c) 2006 Jon Bach / Quardev, Inc.

23. Good enough for whom? • Its intended users • People in the marketplace • Beta participants • CEO / Product Manager • Trade show attendees • Trainees • Interview Candidates • Testers (c) 2006 Jon Bach / Quardev, Inc.

24. Good enough for what? • Its intended purpose • To compete in the marketplace • To get early feedback • Proof-of-concept • Trade show demo • Class exercise • Interview test • Testing (c) 2006 Jon Bach / Quardev, Inc.

25. Good enough for when? • RTM • Today’s Bug Bash • Beta • Until we get a bad review • E3 / Comdex / QAI • Until we patch it • Until we get a competitor • Until another Y2K (c) 2006 Jon Bach / Quardev, Inc.

26. The context factory: “Triage” Witnesses Project Manager Test Manager Documentation The triage meeting is a key tool of “good enough” software development. If you have one, you’re doing it. Product Support Release Manager Sales / Marketing Localization Dev Manager (c) 2006 Jon Bach / Quardev, Inc.

27. Some “what if” bugs to triage • Title of the software is misspelled on splash screen • Blue screen in driver.vxd when opening CD during app setup • Resumes written in Arial font print in Courier • Vegas-style game does not load or launch • 20 Jefferson St. in Newburyport, MA is missing • ScreenSaver GIFs look pixilated in 640 X 480 • 2000 Lotto numbers picked for me did not win anything (c) 2006 Jon Bach / Quardev, Inc.

28. About risk… “There’s more to quality than the absence of bugs” -- Kaner “There’s more to quality than the presence of bugs” -- Bach Cost of slipping vs cost of shipping Testing is a process of revealing risks… …triage is a process of assessing risks The word “evaluate” has the word “value” in it (c) 2006 Jon Bach / Quardev, Inc.

29. What you can do about risk… Think of some risks and prioritize them Perform testing to explore each risk Risks may disappear and new ones may emerge, so adjust your effort to stay focused on the current crop see http://www.satisfice.com/articles/hrbt.pdf (c) 2006 Jon Bach / Quardev, Inc.

30. An example see http://www.satisfice.com/articles/hrbt.pdf (c) 2006 Jon Bach / Quardev, Inc.

31. Helping stakeholders A few of the ways testers help stakeholders make “good enough” (economic) decisions: • Feature comparison with competing products • Performance in different configurations • Assigning severity to a bug • Talking to Product Support • Simulating different users • Compatibility with past products • Beta programs • Hosting Playtest sessions • Our past experience or “gut feelings” (c) 2006 Jon Bach / Quardev, Inc.

32. Measuring “good”-ness (c) 2006 Jon Bach / Quardev, Inc.

33. Good Enough gone wrong: part 1 How software ships with too little quality: • Complacency • Denial • Irrational Exuberance • Inadequate analysis of risk • Pressure to ship (economic, cultural, emotional, political) • Misunderstanding of testing • Monopoly • Test did not raise the right questions (c) 2006 Jon Bach / Quardev, Inc.

34. Good Enough gone wrong: part 2 How software ships with too much quality: • Complacency • Denial • Irrational Exuberance – “a heightened state of speculative fervor” • Inadequate analysis of risk • Pressure to fix bugs (economic, cultural, emotional, political) • Misunderstanding of testing • Monopoly • Test did not raise the right questions (c) 2006 Jon Bach / Quardev, Inc.

35. Take-aways? • As a tester, you are an agent to help stakeholders discover stories about VALUE (or lack thereof) in your software • Whether you mean to or not, you tell those stories in bug reports and test status reports • You can’t think about fixing a bug without also thinking about the cost to fix it. • 4-step formula: try it in your context (c) 2006 Jon Bach / Quardev, Inc.

36. Was this talk good enough? 1) Sufficient benefits 2) No critical problems 3) The benefits outweigh the problems 4) In the present situation, taking the time to test it further would be more harmful than helpful If the answer is “No” to any one of the four criteria, then I have more work to do to impart value… (c) 2006 Jon Bach / Quardev, Inc.

37. Sources / More info James Bach “The Challenge of Good Enough Software” http://www.satisfice.com/articles/gooden2.pdf SixSigma.org http://www.six-sigma.org/why.html Motorola http://www.motorola.com/content/0,,3088,00.html Michael Byron “Satisficing and Optimality” http://mbyron.philosophy.kent.edu/pubs/satisficing.html General Electric http://www.ge.com/en/company/companyinfo/quality/whatis.htm (c) 2006 Jon Bach / Quardev, Inc.