What Is the Value of Testing and How Can We Increase It? Paul Gerrard Technical Director

1. What Is the Value of Testing and How Can We Increase It? Paul Gerrard Technical Director Systeme Evolutif Systeme Evolutif Limited 9 Cavendish Place London W1G 0QD, UK Tel: +44 (0)20 7636 6060 email: paulg@evolutif.co.uk http://www.evolutif.co.uk/

2. “[A cynic is] A man who knows the price of everything but the value of nothing” Oscar Wilde (1854-1900), Lady Windermere’s Fan. “To win without risk is to triumph without glory” Pierre Corneille (1606-84), The Cid. Themes

3. Agenda • Let’s play Marketing • Quantity, Quality, Process or Product – Where is the Value? • The Most Valuable Product of Testing is Project Intelligence • Influences on the Value of Testing • Increasing the Value of Testing • Process “Improvements” and the Value of Testing • Summary.

4. Let’s Play Marketing • We need to consider: • Who are the customers of testing? • What exactly does the customer value? • How can we increase the value of our testing? • How can we increase their appreciation of our testing? • How do we select an “improvement” to make? • How do we judge the value of an “improvement”?

5. Quantity, Quality, Process or Product – Where is the Value?

6. Mine clearing • According to captured records, there are 100 mines in that minefield • We have identified and removed seven mines • I crossed it once but don’t remember how • Is it safe?

7. Test activity and information • I ran some tests • But I can’t remember what happened or how I could repeat those tests • Was this testing valuable? • I ran a hundred tests and the system failed 23 times • I can’t remember how I did it or where the system failed • Was this testing valuable? • The ACTIVITY of testing only has VALUE if it generates INFORMATION (this is the deliverable).

8. How much detail? • Which is best? • “We ran lots of tests” • “We found lots of faults” • “10% of our tests caused the system to fail” • “This test case caused a failure” • “This set of test steps caused the system to fail at precisely this point, in precisely this way…” • Test records should be detailed enough to plan, design, implement, execute tests and interpret test results and reproduce failure • AND NO MORE.

9. Quantity or quality? • Which would you pay more for? • 100 page test plan or 200 page test plan? • 100 incident reports or 200 incident reports? • 100 requirements faults or 100 coding faults? • 100 random tests or 100 ‘designed’ tests? • Quantity is one thing, but the nature and quality of the information and the way it was prepared makes a huge difference to its value.

10. Testing generates “project intelligence” • If you don’t know where you are, a map won’t help • If you don’t know the status of the deliverables, the best plan won’t help you manage your project • A project is like driving off-road in the dark • Testing provides the headlights for the journey • Testing provides data on the status of deliverables and generates “PROJECT INTELLIGENCE” • Stakeholders and management need intelligence to make decisions, command and control the project.

11. Testing Product (unchanged) Valuable Stuff Does testing ‘change’ anything? Product Testing doesn’t change the product in any way. We do it to get the ‘valuable stuff’.

12. What is the valuable stuff? • Test deliverables (infrastructure, test plans, specifications, scripts, data, results etc…) • TEST RECORDS – test documentation (but do only testers read it?) • PROJECT INTELLIGENCE – rarely documented as such though.

13. Military intelligence?

14. The Most Valuable Product of Testing is Project Intelligence

15. Documentation • PRODUCTS • Test Strategy • Test plans, designs, specifications • Procedures/scripts, expected results • Actual results • Incident reports • Log • End of Phase Report • Etc. • TANGIBLE, BUT LESS VALUABLE.

16. Test infrastructure • PRODUCTS • Test process • Configured test hardware • Test databases • Tools • Manual and automated tests • Skilled resources • Etc… • TANGIBLE, USUALLY AN OVERHEAD • (unless these are required deliverables).

17. TEST RECORDS • DATA • How the product SHOULD work (test specs) • How the product ACTUALLY works (test results) • Where the product is STABLE (incident trends) • Where the product is UNSTABLE (incident trends) • What they have COVERED in the product (test log) • What they have NOT COVERED (test plan – test log) • How (and where) the product FAILS (incidents) • TANGIBLE AND VALUABLE, BUT… • (valuable to testers, developers, NOT management).

18. PROJECT INTELLIGENCE • KNOWLEDGE OF PROJECT RISKS/BENEFITS • RISKS • Requirements are poor and/or unstable • Less of the product is ready, product is not as good as planned • They should have been less ambitious, done it differently • The project will slip • BENEFITS • Some features/benefits are available • Other features/benefits are blocked by risk uncertainty • INTANGIBLE AND VALUABLE • Crucial to management and stakeholders • Intelligence often arrives TOO LATE TO BE OF USE.

19. LOW VALUE Documentation Test infrastructure ** UNLESS these are specific objectives or deliverables of project HIGH VALUE Test Records Project Intelligence Where the value of testing lies • We can increase the value of testing MOST by: • Improved INTELLIGENCE • Providing INTELLIGENCE EARLIER.

20. Influences on the Value of Testing

21. Intangible value, tangible cost • Testers think testing is good and has undeniable value, but we are not independent! • Not everyone thinks that (software suppliers, in-house developers?) • If you are not a tester, the value seems intangible • But the cost IS tangible • Exercise, flossing, house maintenance, life insurance, testing can all be put off till another day • If you don’t do it now you store up problems for later • A bit like a lifestyle choice – it’s human nature to put things off.

22. Key factors in the value of testing • Adding value depends on how well we achieve and communicate: • Coverage • Incidents • Risk-based testing • Benefits-based testing • Time to deliver that value depends on the test process (more later) • Willingness of the organisation to accept and act upon project intelligence is key (more later).

23. Coverage • Breadth of coverage • Are all aspects of the product tested? • Depth of coverage • Are critical aspects of the product tested more thoroughly? • Can the measure be applied systematically? • Is the measure relevant to the faults to be detected or the risks of concern? • Can we communicate coverage in a meaningful, understandable way?

24. Incidents • Are the failures of most concern being found and logged as incidents? • Can these failures be reproduced easily? • Are the incidents being addressed in the right order? • Are the incidents of most concern being resolved quickly?

25. Risk-based testing • Are product risks identified early in the project? • Are stakeholders involved in the decision to rank these risks? • Is the decision to include risks in scope for testing done by consensus? • Is the decision to de-scope risks done by consensus? • Is test design driven by the need to address risk? • Is test progress reporting based on risks addressed?

26. Benefit-based testing • Does the project have cardinal objectives (benefits) defined? • Are all risks related to the benefits? • Are tests related to the risks that block the benefits? • Is test progress reporting based on the benefits that are threatened or available?

27. Increasing the Value of Testing

28. Contribution of the test process • Does testing generating valuable intelligence? • Does testing generate intelligence early enough? • Is testing flexible enough to react to new risks? • Are the test communication channels clear? • Improvement types: • Effectiveness • Efficiency • Perceived value of testing.

29. Effectiveness improvements • Improvements to “intellectual” test activities enhance the quality of intelligence • Implementing these activities earlier make intelligence available earlier • Examples: • Formal test design improves coverage, and improves the quality of test records • Improvements that promote earlier test activities tend to make the value available earlier (e.g. more reviews).

30. Efficiency improvements • These reduce the cost of doing testing • Improvements to the mechanical or clerical aspects of our test process make us more efficient (but don't affect the value of testing) • For example • Using a tool to automate tests might make it cheaper to do a task, but does it add value? • It might save time, allowing us to do other useful activities • Ultimately, effectiveness improvements are more valuable.

31. Organisational culture • Willingness to accept and act on the intelligence that testers generate is key • Primarily a cultural issue • If the intelligence isn’t valued by the culture - it doesn't matter what value it has • We need to EDUCATE our customers.

32. Increasing the perceived value of testing • No such thing as “absolute value” of testing • We must work on the PERCEIVED VALUE to the customers of our testing: 1. Improve the value in line with their expectations • Provide richer, deeper, focused, relevant information • “Create more complex, subtle, interesting wine” 2. Increase their appreciation of what we do • Educate them to understand testing products • “Train their palate to detect the complex flavours” • Improvements must address both issues.

33. Process “Improvements” and the Value of Testing

34. Test process or test product assessment? • Assessments of the “Test Process” may miss the point • “Process healthchecks” are useless because they don’t focus on the VALUE of test deliverables • Assessment of the perceived value of Test Products must be a key driver • We must focus on process changes that: • add value to the test products • enhance the perceived value of test products • make test products available earlier.

35. The challenges of testing improvement • Barriers are organisational, personal, not technical: • changing management perceptions so they support testing and improvement • overcoming management and practitioner resistance to change • design and implementation of workable processes and management controls • People, not tools, implement test strategies • Change is easier if people see VALUE being added.

36. Problems or symptoms? • Need to separate problems from symptoms: • “Management doesn't understand the value of testing” • “The cost of testing is high but difficult to pin down” • “Developers, testers, users never trained” • “The quality of the product delivered into testing is poor, so takes longer to system test” • “The perceived value of testing is low” • Need to understand symptoms to identify the source of problem and fix the problem • Improvements often tackle the symptoms only.

37. Right improvement mix is key • A mix of improvements is most likely to be required: • management awareness • tester training • developer training • improved definition of the test stages and their objectives • measurement of the quality of the product at each stage • etc. etc.

38. Constraints and expectations • Not all improvements a good idea straight away • some improvements are expensive • some save time, but bring dramatic change • some improve the quality of the testing, but take longer to implement • some tidy up ‘loose ends’ but have negligible value • Very few improvements save time, improve quality, cause minimal change and pay back after two weeks • Need to set/agree clear management expectations.

39. Do tools increase the value of testing? • With tool vendors present, I’d better be careful! • Need to distinguish between • Quantity, quality and the information provided • A tool might allow you do to testing faster, but it probably won’t increase the value of tests • It might make a task interesting, so a valuable task gets done • It might give you more time to do other valuable tasks.

40. To Summarise…

41. How to increase the value of testing • The value of testing lies in the project intelligence it generates • Produce documentation with purpose: • To allow testers to do their job (THE MEANS) • To communicate intelligence (THE END) • Process ‘improvements’ selected to deliver better intelligence, earlier, faster and cheaper • You need to train your customers to appreciate the project intelligence that testers produce.

42. “A poor tester knows the challenge and cost of testing but not its value” Don’t be cynical – testing isn’t just a costly activity to be gotten through, testers should aim to deliver value “To win without risk is to triumph without glory” Risks are inevitable - the testers’ role isn’t to avoid them, but inform management and reduce their uncertainty Satisfaction (and recognition) comes from overcoming risks, not doing the mundane tasks of development Make your management aware of software product risks and report progress towards their goal, not just incidents. Testers deal in success, not just failure. Themes revisited

43. What Is the Value of Testing and How Can We Increase It? Paul Gerrard www.evolutif.co.ukwww.riskbasedtesting.com