Test Automation Failures - lessons to be learned

1. DEV-TEST (SPRING) ‘99, UNICOM Seminar, Experiences with Test Automation Commonwealth Institute, London, 28 May 1999 Test Automation Failures -lessons to be learned Ståle Amland, Amland Consulting, Hulda Garborgsv. 2, 4020 STAVANGER, NORWAY Mob:+47 905 28 930, Phone: +47 51580587, Fax: +4751585524 e-mail: stale@amland.no WEB: www.amland.no

2. Presentation Outline • Introduction - The Projects • Problems • Test Tool Problems • The Testers and their Problems • Test Environment Problems • Timing and “volume” of test automation • Recommendations • Test Tools - recommendations • The Testers - recommendations • The Environment - recommendations • Timing and “volume” - recommendations • Summary Amland Consulting

3. Introduction - Project A • Retail banking applications (server only) • IBM Mainframe, CICS / DB2, Client / Server, “waterfall” • Test bed: PC terminal emulator • Test Phase: System Test, 6 months, 3,000 man-days • Test tool: AutoTester • Test Documentation: • What to test! NOT How to test. • Product specialist used for testing • “Manual” Test Process: • Recording for audit trail and “manual playback” • Recording of scripts to be transformed into automated test scripts • Resources (plan: 100% automation) • 15%of all transactions were tested • 25% of available testing resources • 2.5% of all faults were detected Amland Consulting

4. Introduction - Project B • “Home Banking” application for small businesses • Windows multi platform Client/Server application, RAD project • 100 windows, 5 hardware platforms, 15+ mainframe systems • 25 developers in 6 months, 25 testers in 3 months • 8 mainly test management • 2 test automation • Test Phase: User Acceptance Testing (UAT) • Test tools: SQA Suite • Test Documentation: • Structured documentation: Input / Output • Used Test Tool for management • Automated Test Process: • Based on record and playback Amland Consulting

5. Test Tool Problems • The Test Tool does not support the Development Tools: • Will not recognise objects • Must rely on relative positioning • Different Tools = Different Problems • If you do Automation “wrong” - selecting the “right” tool will not save you! Amland Consulting

6. The Testers and their Problems • Product people as testers • Conflict: • Testing the system “manually” vs. • Recording tests for later automated regression testing • Result: demotivation - poor testing! • Project B - RAD: • Instability of software in UAT Amland Consulting

7. Test Environment Problems • Project A: • Data Interference between testers • Database “unique key” problems • Relative Positioning: • If resized: Tests recorded on one PC would not replay on other PCs • Project B: • …same problems + • Test tool did not support development tools without separate .DLL • Because of outsourcing: Supplier would not include test tool .DLL Amland Consulting

8. Timing and “Volume” Problems • Tried to automate too much at the same time • Hectic recording process • Lost too much information • Massive Test script maintenance effort • Only managed to automate 15% • 100% automation objective became demotivating • No prioritisation took place Amland Consulting

9. Test Tools - Recommendations • Prerequisites: • Tool must support development and test environment and cover all (planned) production environments • Next - must handle... • documentation, planning, tracking and defect handling simultaneously from • automated testing • semi-automated testing • manual testing • Ability to record and log user input • For QA or “Manual Playback” Amland Consulting

10. The Testers - Recommendations • Allow time for training! • “On the job training” is needed • First project will have low productivity • Process should Combine Manual and Automated Testing • Dedicated testers for automation • Benefit to know programming • Build a maintainable test suite • Record manual tests: • “Readable” not necessary “replayable” Amland Consulting

11. Test Environment - Recommendations • If possible: integrate test automation into development • Separate environments for manual and automated testing • Implement Configuration Management • Allow enough time for Test Environment Maintenance • It is very time consuming! • Document shared test data carefully • Periodically restore to initial state Amland Consulting

12. Timing and “Volume” - Recommendations • Start with “simple” functions • I.e. changes to design / code / user interface are unlikely • No “unique key” problems • “Prove” benefits • Identify time-consuming tests • Pilot Project: • Training of testers • Build training material • Do NOT plan 100% coverage first time! Amland Consulting

13. Summary • Automated Testing • Benefits? YES • Complicated? YES • High resource requirements? YES • Tools very useful in • planning • progress tracking • defect tracking • documentation management • …as well as automation • All projects will benefit from the use of a recording tool as part of test documentation and audit trail / due diligence Amland Consulting

14. If you want to automate your testing, listen to people with experience before you start! Amland Consulting

15. Contact Details Ståle Amland Amland Consulting Hulda Garborgsv. 2, N-4020 STAVANGERNorwayMob: +47 905 28 930 Phone: +47 51 58 05 87 Fax: +47 51 58 55 24 e-mail: stale@amland.no Amland Consulting