Good Enough Testing

1. Good Enough Testing James Bach james@satisfice.com http://www.satisfice.com The author thanks ST Labs, Inc. for supporting the work that led to some of the materials in this presentation.

2. Paradigms of Good Enough... • Not Too Terrible: “we’re still in business” • Infallibility: “anything we do is good” • Righteous Exhaustion: “perfection or bust” • Customer Oracle: “customers seem to like it” • Defined Process: “we follow a Good Process” • Static Quality: “we satisfy The Requirements” • Accountability: “we fulfill our promises” • Advocacy: “we make every reasonable effort” • Dynamic Quality: “we fulfill the mission”

3. unnecessary quality Perfect floating line Good enough quality bar unacceptable quality Awful Dynamic Quality Paradigm It’s more important to work on Item B. Further improvement would not be a good use of resources. Item A Further improvement is necessary. Item B

4. A Heuristic for Good Enough 1. X has sufficient benefits. 2. X has no critical problems. 3.Benefits of X sufficiently outweigh problems. 4.In the present situation, and all things considered, improving X would be more harmful than helpful. All conditions must apply. Problems Benefits

5. Good Enough... • …with what level of confidence? • …to meet ethical obligations? • …in what time frame? • …compared to what? • …for what purpose? • …or else what? • …for whom? Perspective is Everything

6. You can use this to: • Argue persuasively for improvement, even in the midst of chaos. • Plan ahead for high quality. • Understand what problems are keeping otherwise smart, caring people from doing excellent work. • Notice when “perfect” or “as good as humanly possible” is not good enough. • Structure a dialog about acceptable quality.

7. Law of Medium Numbers For medium number systems, we canexpect that large fluctuations, irregularities,and discrepancies with any theory will occurmore or less regularly. -- Gerald Weinberg Why not Quantify Good Enough? Because quality is an emergent property of a complex system, any quantificationof quality is incomplete.

8. Heuristics are an alternative to quantitative models... • We can measure some things. • We can observe some things. • We can reason about thoseobservations and measurements. A heuristic is useful method that sometimes doesn’t work. Heuristic means “serving to discover”

9. Test Project Dynamics:Star Model

10. Test Project Dynamics:Star Model

11. Test Project Dynamics:Star Model

12. Test Project Dynamics:Givens vs. Choices • Motivation: What testing does the situation require? VENS Motivation Capability CHOI • Capability: Can we perform that testing in this situation?

13. Analyzing a Test Process • Are GIVENS good enough? • Do CHOICES about process exploit the GIVENS and address the MISSION well enough? • Is MISSION is achieved well enough? How do you know?

14. Find important problems Assess quality Certify to standard Fulfill process mandates Satisfy stakeholders Assure accountability Advise about QA Advise about testing Advise about quality Maximize efficiency Minimize time Minimize cost MISSION:The most important part The quality of testing depends on which of thesepossible missions matter and how they relate. Many debates about the goodness of testing are really debates over missions and givens.