Commercial Test Automation Tools and their real benefits

1. Commercial Test Automation Tools and their real benefits Ananya Das

2. An Automation Fable • Once upon a time in the peaceful land of Testing, the wizard proposes , “ Let there be automation ..” and the King agrees “Yes.. that is the way to be” • Enter Chandan Nilekani “delegate” • Reviews commercial tools, selects , buys , costs …. • Assigns to Narad Murthy “overworked “ • Narad has several other responsibilities , tool is complicated , tries customer care • Finally gets an expert – works and works – several days pass … but no automation • Enter NazimPremji “short timer” • Fresh graduate, new ideas, trainings • Works full time … library, gets some parts working • After 6 months … gets a great offer ….. • Enter Shankar Nadar “hardluck” • Tries to execute scripts, no documentation • Incomplete scripts, poor error messages, gets some bugs … king happy • Next version of product.. .Changes …. Changes scripts … long hours … gets working • Product shipped • Customer very unhappy … lots of bugs … why ?? • Automation Programming error had dismissed these bugs … • King – “Off with their Heads !!!!!!!!!“

3. Some Commercial test Automation tools

4. Evaluating a commercial tool • How easy is it to use? Learning Curve? • Continuous Integration (CI) friendly? Call & Log without manual intervention? Work with existing systems ? • How does it interact? IE (client side)? HTTP layer? • What types of tests does it support? • Web? WinForms? • Performance? • Framework for Integration Test (FIT)? SQL? • Does it support recording? • Does it support scripting? • Languages? • Easy creation of scripts ? Object Oriented Scripting Support? • Interactive Debugging Tools? Easy integration with External libraries? • Ability to identify objects using multiple parameters? • Ability to run multiple scripts consistently in a Batch mode? • Ability to store test application's objects so that scripts are well maintainable? • Reusability of Scripts and Tests? • Reporting mechanism? • Object parametrization? Ajax? Popup handling? Keyword driven? Automatic Exception Handling? • Need to support IE7? How easy is it to transfer tests to/from platform?

5. Evaluating a commercial tool • How well is the tool supported? • Commercial? • Community? • Support for mocking + stubbing? • Test case organization? • Organizing tests similar to structured manual test plans? • Support cross frame script calls? • Web Objects? • Cost/Licensing? • Integrated with other test components? • VPN / Remote Desktop friendly? • Active Community of Users? • Forums? • Blogs?

6. Advantages of Commercial Tools

7. Truth or Myth ? • Commercial test tools are expensive ? • Myth • Under the influence of this myth some companies, especially the small ones: • Try to develop their own test automation tools • Use scripting lang. like Perl and Ruby,use shareware test tools • Do not consider test automation at all • Reality • It has been calculated that if implemented properly, the cost of this tool is just 3% of the person who uses it, but productivity gain can be very significant

8. Truth or Myth ? • Recorders make test automation easier • Myth • The Truth: • They make recording easy … • But they also make it easy to create automation that is difficult and expensive to maintain • A recorder can serve as a training device to help you learn a scripting language

9. Truth or Myth ? • You can’t develop test automation for an application until it is stable • Myth • The Truth: • You should develop test plans as soon as the Requirements are documented, You can also develop automated tests from wire-frames, mock-ups and prototypes • In the case of wire-frames, you can even test some of your methods • If tests are structured properly, only a small portion of your work will have to be maintained when GUI changes are made to an application that has not yet stabilized.

10. Truth or Myth ? • Test automation is most effective for regression testing • Part Truth: • Test automation is very effective for regression testing • It is also effective for first pass testing when a data driven approach is used • Automated tests should be designed to find bugs not just verify that regression tests don’t break

11. Truth or Myth ? • It is difficult to automate a dynamic application where data changes constantly • Myth • The Truth: • Data can be separated from test code so that only the data needs to be changed • Better yet, in some cases input data can be captured at runtime to meet the criteria required for a test • Or a database query can be executed to get data at runtime • Verification data can also be captured at runtime so that tests remain flexible in a dynamic environment

12. Truth or Myth ? • Test automation is a software development task • Truth: • Automated tests should be designed, developed and tested • You need to have some kind of a programming background to implement test automation. Test Automation is not as complex as C++/C#/Java development. • Automated test components are assets that should be treated like application source code • Test automation standards should be developed • Test automation should be subject to peer reviews

13. Truth or Myth ? • Productivity gains will be realized from test tools right away • Myth: • Test automation is an investment • Quality gains will be realized right away • Productivity gains will be realized over time as skills develop and manual testing tasks are offset by completed automation

14. Truth or Myth ? • Automated tests are difficult to maintain • Myth • The Truth: • Separate business rules, test methods and data to promote maintainability • If properly structured, automated tests can be maintained across many releases of a target application • It’s important to run automated tests on every build and make incremental changes as needed

15. Why is Automation great • Speed up testing to accelerate releases • Allow testing to happen more frequently • Reduce costs of testing by reducing manual labor • Improve test coverage • Ensure consistency • Improve the reliability of testing • Allow testing to be done by staff with less skill • Define the testing process and reduce dependence on the few who know it • Make testing more interesting • Develop programming skills • BUT ………….

16. Then … why failure ? • About 63 % of the Automation projects fail …. Why ? • Misconceptions • Testing tools don’t work and are difficult to use • This application is too complex and the schedules are too tight • Senior Management will never support it • Automation will eliminate the need for all manual testing • Manual testing must have taken place before automating a feature

17. Critical Success Factors • Clearly define Automation Objectives • Get management support for the project • Clearly demonstrate the short-term & well as long-term financial benefits to be accrued by investing quality resources and time in the project. • Ensure that all stakeholders are in agreement on the end objective of the project. • Top Management, the IT team, the developers, the tester, and the automation team--may have different requirements from the project. However, success will be elusive unless they all agree to a common goal. • Treat Automation as Software development • Study the requirements • Planning and tracking • Set realistic goals , assess existing skill sets and resources • Evaluate the testing tools; perform automation feasibility; • Identify reusable functions

18. Critical Success Factors • Design and develop a framework • Design and document extensible automation architecture, which clearly outlines the foundation and rules for the project. • Incorporate flexibility and scalability into the architecture to be able to cope with growth in the requirements • Also, provide the testing team with a proof-of-concept test suite and make test case maintenance a top priority • Evaluate testing tools • Validate the tools and approach at the earliest and evaluate its compatibility with the organization’s requirement and the AUT • Check the availability of trained/experienced resources

19. Critical Success factors` • Decide what needs to be automated • Include short and simple transactions and tests that are executed regularly. • Execution with multiple datasets • Do not automate where it is difficult to predict results or are not going to be repeated. • Avoid automating long or complex transactions, cross multiple applications • Spend time on Automation Strategy and Plan • List of activities • Organization test objective • Reusable and Modular test script identification and planning • Run Automation as a full time project • Look at it as a long term investment • Co-operation with functional testers

20. GM Regression Automation

21. ROI ROI = BENEFIT/COST • Automation Cost = Price of Tool + Dev Cost + Maint. Cost + Execution Cost • Manual Testing Cost = Development Cost + Maintenance Cost + Execution Cost • ROI = (Manual Testing Cost - Automation Cost)/Automation Cost Looks right, Doesn’t it? NO ….. • You can’t compare Automated Testing and Manual Testing. They are not the same and they provide different information about the AUT. • You can’t compare cost of multiple execution of automated tests vs. manual tests. You would never dream of executing that many test cases manually.

22. Automation ROI

23. Conclusion … Long Live the King • Commercial Automation tools are useful if implemented and managed correctly • ROI should be managed • CSFs should be considered seriously • And then ... the King said – “ Let there be automation ….” And there were profits, productivity, growth and success all thru the Kingdom THE END