TestIstanbul 2013 Conference “Future Of Testing: New Techniques and Methodologies”

1. TestIstanbul 2013 Conference“Future Of Testing: New Techniques and Methodologies” The Future Of Test Management “Expect The Unexpected” Gerlof Hoekstra

2. Introduction Who am I, the world I come from • Gerlof Hoekstra • Started as software engineer • Early 90’s: Testing ! • Current job: Test consultant/test manager/solution consultant at Atos NL • From software testing to large scale business process integration tests • Complex application chains • Many stakeholders • Many software suppliers

3. Intended audience • Do you sometimes • Feel that your message is not heard? • Feel that nobody cares about your test plans and test cases? • Does your customer • Complain that testing takes too long, asks you: • ‘When you’re done?’, ‘Can’t you go faster?’, ‘You are delaying the project’ • Ignore your good advice and still go live with a flawed system? • Do you sometimes feel frustrated because • You only get half the time that you require for a proper test • Many of the defects you found are still not solved? • Do you worry • If your job still exists in a few years, when all projects are ‘Agile’?

4. Presentation subject • Being successful as a test manager often is very much dependent of the way you behave, set expectations, use words & symbols etc. • Very often (a combination of) subtle things • Some test managers I met and saw at work • are worried about the future • have been frustrated, burned out, replaced (yes, it can be a challenging job) WHICH QUALITIES CAN HELP TESTMANAGERS TO DELIVER MORE VALUE TO THEIR CUSTOMERS AND IN THE MEANTIME HAVE A MORE FUN JOB AND BE PREPARED FOR THE FUTURE

5. Agenda Introduction Mission Test planning & specification Showing test results The future of test management Closure

6. The Test Manager’s Mission How the first 20 minutes of your (to be) assignment can be crucial

7. Job Interview 1 ‘…. write a test plan, make sure the acceptance criteria are SMART, make test cases, have all requirements covered, get the test cases reviewed and formally accepted, execute the test cases, have blocking defects get solved, perform re-tests, regression testing, deliver test report and provide a go/no go advice’.

8. Job Interview 2 ‘I expect you, the future test manager, to assure the quality and decide whether the system can go live, how will you do that?’ ‘What the …?!??’ ‘I don’t…….’

9. My mission: Where I stand for (and not) • I am here to serve, to assist the stakeholders • My job is to provide USEFUL INFORMATION • that the stakeholders need • to make their own informed decisions • I am the weather man ! • I am NOT the quality police, nor the judge • I do NOT ensure product quality either

10. Then what DO I ‘ensure’ ? • My observations are • Objective, facts, no opinions • Made by skilled & trained investigators • You will receive forecasts that you will learn to appreciate and are • Pretty reliable • Early in time (!) • The information you will get from me is very clear, in terms that the stakeholders understand, no explaining or advice needed, enables them to make their own decisions! • The information gathering process is efficient I PROMISE YOU, IN CASE YOUR PROJECT IS HEADING INTO THE WRONG DIRECTION, I WILL BE THE FIRST TO TELL YOU, LOUD AND CLEAR !

11. Two ‘basic values’ • ‘Expect The Unexpected’ • Projects are subject of constant change and movement • One cannot possibly plan all activities ahead • Test strategy should be able to cope with change • Management style • Most testers are highly skilled and trained professionals • Test manager should treat them appropriate • Leave room for personal initiative and creativity

12. Agenda Introduction Mission Test planning & specification Showing test results The future of test management Closure

13. Discussion with an ‘old school test manager’ • How to we know your test set is complete? • Your test cases are not very detailed. • You did not define in the test plan the criteria when a defect is considered blocking. • You do not have certified testers in your team • You did not define acceptance criteria (coverage%, #test cases passed, #open defects) • You should complete the system test before you start the acceptance test • The most important deliverable (go/no go advice) is missing in your plan • --- • --- Review test plan v0.5

14. Let me introduce you to ‘The Doctor’You don’t have to be a control freak to get missions accomplished

15. How to prepare for a football match?

16. Our first holiday abroad

17. Detailed preparation versus Flexibility • Have a simple but clear plan, not too detailed • Have attention to details by the time when really needed, be prepared ! • Prepare / train your team so that they are able to respond when the Unexpected happens

18. Writing a Test Plan – The Theory The creation of the test plan involves the following activities: • Establishing the assignment • Understanding the assignment • Determining the test basis • Analysing the product risks • Determining the test strategy • Estimating the effort • Determining the planning • Allocating test units and test techniques • Defining the test products • Defining the organisation • Defining the infrastructure • Organising the management • Determining the test project risks and countermeasures • Feedbackand consolidation of the plan    

19. Writing a Test Plan – The Theory The creation of the test plan involves the following activities: • Establishing the assignment • Understanding the assignment • Determining the test basis • Analysing the product risks • Determining the test strategy • Estimating the effort • Determining the planning • Allocating test units and test techniques • Defining the test products • Defining the organisation • Defining the infrastructure • Organising the management • Determining the test project risks and countermeasures • Feedbackand consolidation of the plan     Now look at this !?!

20. What I preferInstant, continuous feedback and collaboration

21. Test Plan Example

22. Some additional thoughts • You can not write a test plan on your own, hiding in your office • Received no review comments? The reviewer did not read/understand the plan or is not interested. Don’t consider the plan done! • Strip to the bone, minimize #words, avoid meaningless phrases, use images & tables • A test plan is NOT a contract! • Expect the plan to change

23. Coverage MapsA great way to plan your test Tip: you can use James Bach’s San Francisco DePOT acronym

24. Test specification • How many test cases & scripts do we really need? • How detailed should they be? • Scripted vs Exploratory testing? IT IS ALL ABOUT BALANCE My observation: • Many test educations (and as a result, many certified testers) heavily over-rate the value of specifying a lot of detailed test scripts • Often, you can get away with less test scripts than you might think

25. Example, a limited number of formal test cases, based on a decision table , completed with test charters

26. Agenda Introduction Mission Test planning & specification Showing test results The future of test management Closure

27. A good Test Report • Has useful information for the stakeholders • Is easy to understand for the stakeholders • Does not need any go/no go advice • Is brief and visual • Includes a forecast • Makes clear what you have (not) covered

28. Be VERY careful with coverage metricsTesting is investigation in an infinite space: numbers like 80% could easily be misunderstood

29. Be VERY careful with coverage metricsOnly testers understand these numbers… (do we?) ‘We have reached 91.7% multiple condition coverage’ Do our stakeholders know what this means for them? Be honest, do you know? I’m not sure if I do…

30. Old school test reporting • We covered 95% of the area • We found 60 mines • We disarmed 58 • The 2 mines we did not disarm are here • These are MSM MKII mines containing 4Kg of Composition B explosives IS THIS USEFUL INFORMATION ? WOULD YOU APPRECIATE MY GO/NO GO ADVICE ?

31. Alternative: Use maps to show coverage & test results

32. Alternative: Use maps to show coverage & test results

33. What is wrong with the good old traffic lights? • Do not encourage to take very much responsibility • Don’t think, just obey the law !! • Is not how I want to communicate my test results • Remember: it’s not the test manager who decides! • Weather symbols send a better message • This is likely to happen • You have been informed (warned?) and now it’s up to you to make a decision • B.t.w.: a weather forecast is never 100% reliable

34. Example

35. Longer term forecastingTesting in the Early Project Phases (and I mean testing, not just reviewing) Getting your message heard • Don’t complain, just report the facts • Friendly ask some questions (‘Are you OK with this?’) • And the best: let stakeholders experience the problem  no further comments needed

36. Agenda Introduction Mission Test planning & specification Showing test results The future of test management Closure

37. Trend 1: Changing Software Development Paradigms • Agile • Continuous deployment • Smaller iterations • Test Driven Development • Quality is not ‘owned’ by the testers (anymore)

38. Trend 2: Systems Integration • Chains, ever increasing in scope and complexity • Many software suppliers involved • COTS software, combined with bespoke software • Multi-stakeholder • Cross-company integration • Distributed data • Across the globe, 24/7 business processes ORGANIZING AN END-TO-END TEST CERTAINLY IS NOT A TRIVIAL TASK

39. The future

40. A shift in qualities To: From: • Ensure quality • Go/No Go decision • Find defects • Number of test cases • Number of defects • Enforce, control • Proof • Follow rules & standards • Strictly follow detailed plans • Certification • Requirements freeze • Acceptance criteria • Quality assistance • Deep insight to stakeholders • Prevent failures • Coverage map • Business consequences • Serve, help, facilitate • Forecast • Investigate • Prepared for change • Diversity • Requirement management • Participation, co operation

41. What can we learn from The Doctor? • Passionate about his mission • He enjoys his journey • No dogma’s • Not afraid of the unexpected, embraces change • Surrounded by a few smart and pro-active companions • Uses a few simple yet powerful tools • Easy going, BUT can be very persistent WHEN IT MATTERS

42. Almost Done!But tell me, when is testing done?

43. Thank you ! gerlof.hoekstra@atos.net +31 6 512 88 478