Susan Windsor Insight Through Intelligence WMHL Consulting Limited, MD

1. Future for Functional Test Automation? TM Forum – April 2006 Susan Windsor Insight Through Intelligence WMHL Consulting Limited, MD Title slide

2. Future for Functional Testing Challenges for Functional Automation Strategic Direction What does this mean to you? AGENDA

3. Future for Functional Testing

4. Increased Testing Demand………..? • Global development, with large projects having multi site, multi geography and multi suppliers to contend with. • Corporate and regulatory requirements growing all the time • Business demands new applications, faster and cheaper to obtain competitive advantage • Closer alignment of IT and business to repair lack of business confidence

5. Or, increased competition for roles? • Business testers using frameworks reduces the number of traditional functional testers required • Agile development methods mean developers undertake more unit and component testing • Growth of outsourced testing to different geographies Is the role of the UK functional tester (who is neither technical nor business specialist) dead? With the focus on skilling up functional testers, are we fiddling whilst Rome burns?

6. Challenges of Functional Automation

7. Record and Playback • “all you need to do…….…!” • Easy to record the scripts • Extremely fragile • Expensive to maintain • On average, each test run requires at least 50% of the scripts to be recorded again Please tell me no one does this anymore!!!

8. Scripting • Use the scripting language to write scripts that do what you want • Build in as much robustness as possible • But, you’re building an application to test an application, which also needs testing! • Maintenance costs can be very high • Needs programmer skills • Cost of implementation not affordable at project level • Challenging for Test Managers to incorporate in overall test plans

9. Table Driven • Like scripting but more flexible and greater re-use • Remove items from the script that change, e.g. data • Reduces maintenance costs a bit • Implementation costs about the same • Still requires specialist skills to implement • Still a challenge for Test Managers to control

10. Functional Test Automation is Broken! • Focus on technology rather than business needs • 80% of functional testing still manual • 60% to 70% of automation tools used for non-functional testing • Typically, traditional functional automation stops at 100 scripts, regardless of test coverage requirement • Critical factors • Cost of implementation and maintenance prohibitive • Insufficient and expensive skills required • Inability to asset share over different technologies

11. Handover Milestones Initiation Build & Test Sys Test Support & Maintenance Int Test Acceptance Different companies, different teams, different skill sets, tight budgets (cost of automation not always considered at initiation time!!) Outsource Outsource Outsource External Influences Consultants Business Partners Products Suppliers Project Structure also a challenge

12. Strategic Direction

13. Automation Frameworks • Wouldn’t it be good if…….. • Tests could be documented in common format, regardless of whether they are manual or automated • The format for the tests resulted in quicker preparation time than traditional manual tests • Both developers and business testers could understand and use the same test format • Script maintenance was no longer a requirement • Tests could use any of the test execution tools required by the underlying technology Faster planning, faster execution, and far less technical skill required

14. Business Analysts already using them, and use will grow • Home grown frameworks built within organisations to meet business demands • Niche suppliers providing frameworks; try a Google search – 512,000 hits this week • Seen a handful that appear mature • Latest review by Paul Herzlich (OVUM analyst) • Market Leaders such as Mercury developing Business Process Tester (BPT) This is the industry direction now

15. Governance Requirements Analysis & Design Project Management Test Management Test Case Definition – Automation Framework Test Case Documentation Manual Test Execution & Documentation Non UI Component Test Execution Harness UI Automated Test Execution Tools Test Requirements Sign Off Test Results Where Frameworks Fit

16. Frameworks & Industry Needs • It’s too early to automate in our project • Define the test cases in the Framework now, and automate later • We haven’t got manual testing under control yet! • Use the Framework as the standard test case definition and use for manual and automated scripts • We do Agile development, functional automation doesn’t fit • Define test cases before the development commences and use the documentation to obtain business signoff against requirements, as soon as code is written, you can automate

17. Frameworks & Industry Needs - 2 • Automation consultants are too expensive to increase our automation test coverage the way we’d like to • You can fully utilise your manual testers to build test cases to generate automated scripts, your automation specialist can concentrate on the more exciting aspects of test automation and multiple projects can be support by a single technical resource. • Our offshore partner’s use automation but we are concerned at the lack of visibility over exactly what their tests do • Full test documentation can be generated to demonstrate what the automated scripts will do so that you retain control

18. Frameworks & Industry Needs - 3 • We are tied into our Automation Supplier and even though we’re not comfortable with that, we can’t see a way out • Transfer your test case definitions into Framework now, initially prove your test cases via interaction with existing tools and you can switch when ever you like • We can’t afford the cost of automated testing on this project • Using a Framework will reduce the cost of automation implementation by between 58% and 75% over traditional approaches • We find the cost of maintenance prohibitive to extend our test automation further • Using a Framework will reduce the cost of automation maintenance costs by between 75% and 85%

19. What Does This Mean to You

20. Discussion Time • What is your organisation’s direction? • What is your view on Automation Frameworks? • Do they have a future? • Do you require different skills? • What is your future?

21. Thank You Susan Windsor WMHL Consulting Limited, MD susan@wmhl.co.uk Closing slide