When to Compromise on Testing

1. When to Compromise on Testing eurostar 2000 Copenhagen6th December 2000 Graham ThomasTesting ManagerWall Street Systems

2. Abstract Several times over the last few years working as a tester I have found myself making compromises on the way that I have been testing, and generally felt very uncomfortable about doing so. Everyone will tell you that compromise in testing is inevitable, but that never makes it any easier. It is never possible to get the perfect mix of resources, skilled testers, equipment to test upon, enough time to plan and prepare for testing, or even to run all of the test scripts, let alone re-test all of the software fixes. This talk does not offer a silver bullet solution, but will take you through the testing lifecycle, identifying the areas where compromise is most commonly called for, and show you the techniques that I have found successful in managing and controlling that compromise without losing integrity. And also a few of the pitfalls! WALL STREET SYSTEMS

3. Dictionary Definition • Settlement of dispute by mutual concession • Adjustment of conflicting opinions by modification of each • Bring oneself under suspicion by indiscrete action WALL STREET SYSTEMS

4. Agenda • Strategy • Planning • Resources • Preparation • Training • Monitoring & Reporting • Execution WALL STREET SYSTEMS

5. Strategy - Compromises • We know how to test, just produce a plan. • We are doing rapid development but we want proper testing! • Only one phase of testing actually carried out • Do we even have a testing strategy? WALL STREET SYSTEMS

6. Strategy - Tips • Brainstorming Workshop • Bring all parties together • Moderate the session • The presentation of the material is as important as the strategy document • Keep the strategy document short 15-20 pages, with plenty of diagrams. WALL STREET SYSTEMS

7. Strategy - Tips • Who, How, Why, What, Where & When. I keep six honest serving-men (They taught me all I knew); Their names are What and Why and When And How and Where and Who. Rudyard Kipling -The Elephant's Child WALL STREET SYSTEMS

8. Planning - Compromise • You’ve got 10 weeks. • We have to go live on Jan 1st • We don’t have planning tools for the testing team! • You do not control the resources needed to carry out testing WALL STREET SYSTEMS

9. Planning - Tips • Embed the testing plan in the overall project/program plan • Establish the dependency on development deliverables • Use past experience to develop a planning template • Update your plan WALL STREET SYSTEMS

10. People - Compromise • We have some new recruits that you could have! • The business don’t want to get involved, they’re too busy • All our testers are committed, get some external resource • Organisation does not recognise testing as an IT role WALL STREET SYSTEMS

11. My Ideal Testing Team • Works for nothing • Prepared to die trying WALL STREET SYSTEMS

12. People - Tips • Mix of business, technical and testing skills • Second from business and IT • Consultants can kick start your project and lead by example • A well motivated and rewarded team can achieve the impossible • Don’t overwork the test team! WALL STREET SYSTEMS

13. Environment - Compromise • You can test in the development environment • Production support has priority over the test environment • We can’t afford new kit, you’ll have to use what we've already got! WALL STREET SYSTEMS

14. Environment - Tips • Dedicated Environment • As production like as possible • Agree SLA (Service Level Agreements) with all systems owners • Take a back-up before you start • Document the configuration WALL STREET SYSTEMS

15. Preparation - Compromise • So when do you start the testing? • It’s only a small system, what is taking so long, I can test it in 5 minutes? • Why do I need to prioritise this? It’s all important to me! • I am not producing any documentation, you can have a look at the finished code! • We use a copy of production for test data WALL STREET SYSTEMS

16. Preparation – Tips • Clearly identify how long the preparation activity will take • Prioritise everything • Circulate test planning output for review • Use a repository for test planning • Use the training material in the absence of systems documentation • Back-up your baseline WALL STREET SYSTEMS

17. Training - Compromise • We can’t afford training for the testers • You don’t need training in the testing tool, just use it and see how you get on • Test team has no formal training in structured testing or testing techniques WALL STREET SYSTEMS

18. Training - Tips • Training in • Techniques and methods • Tools and utilities • System under test • Mentoring • Quick start consultancy rather than formal training • Train on-site with your own systems, data and configuration • Encourage personal development WALL STREET SYSTEMS

19. Execution - Compromise • Development are going to deliver late but we don’t want to change the go-live date! • Not all of the system will be ready in time • Problem reports dismissed by development • Carry on testing even when code should be returned to development • Testing in parallel with development WALL STREET SYSTEMS

20. Execution - Tips • If you don’t predict your results before you run the test, you are not doing testing • Don’t automate if you don’t have time • Report faults as soon as you find them • Tight control is required when executing the regression test pack • Defer low priority tests • Batch up fault fixes WALL STREET SYSTEMS

21. Reporting - Compromise • Different view required by management each week • Volume of faults makes it difficult to keep up with reporting schedule • Different view maintained by development team • Reports are several days out of date • Pressure to embellish the story WALL STREET SYSTEMS

22. Reporting - Tips • Monitor, Measure and the Manage • Maintain reporting information in a repository • Be flexible when reporting • Produce accurate daily status reports • Keep a consistent view • Don’t bombard management with stats • Be honest WALL STREET SYSTEMS

23. Reporting - Tips 1. Progress against Plans 2. Issues Raised v Closed 3. Test Script Coverage 4. Issue Reporting WALL STREET SYSTEMS

24. Summary – What I will do • Write a good test strategy presentation • Embed testing in the project plan • Prepare thoroughly • Back-up my baseline • Report faults as soon as I find them • Prioritise test execution • Provide timely management information • Take it personally when we find a live bug WALL STREET SYSTEMS

25.  Summary – What I wont do • Just get on with it! • Allow testing to be used as development contingency • Carry out testing without a plan • Be pressured into curtailing preparation • Test on an unrepresentative environment • Test in parallel with development • Put a gloss on testing progress WALL STREET SYSTEMS

26. Conclusion • You can still test successfully whilst making compromise • A testing methodology is just a means to an end • Your measure of success is the quality of the delivered system, and not adherence to the methodology • Remember both sides give a little WALL STREET SYSTEMS

27. Contact Details Graham Thomas Testing Manager Wall Street Systems  0044 20 7827 0430 graham.thomas@wallstreetsystems.com WALL STREET SYSTEMS