Building an Automation Framework around Open Source Technologies

1. Building an Automation Framework around Open Source Technologies Charles Thangamuthu 20th November 2010 Disclaimer – This is a personal case study and does not represent PTC’s view on the topic

2. Thanks SiliconIndia!

3. Before we step in…. What this presentation is What this presentation is not • An example of how open source tools can be used to form an automation framework • To say there are solutions readily available for many of your problems – don’t have to reinvent the wheel • More of a case study • Does not say open source technology is better than commercial testing tools • Does not say the components talked about are the best in their class • Does not talk about a PTC proprietary or certified framework • Does not claim all your automation woes are solved

4. Topics • Automation Framework • Definition • Requirements • Functions • Generic Architecture • Framework components • Automation Tool • Version Control System • Continuous Integration • Test Runner • Data Manager • Advantages • Flexibility • Scalability • Benefits

5. Test Automation Framework Definition - What is it? • Automation Framework is not a single tool or process but a collection of tools and processes working together to support automated testing of an application • Is an integrated environment that sets the rules, assumptions, concepts and tools to aid automated testing • It integrates function libraries, test data, object details and various reusable modules • These components act as building blocks which can be assembled to represent a business process

6. Test Automation Framework Requirements – What do we need? Platform and OS independence Data driven capability Customizable result reporting Ease of debugging and logging Version control of tests Execute unattended runs Repeat tests as needed Extensible and Customizable Status notification Distributed test execution Automation Framework

7. Test Automation Framework Functions – What does it do?

8. Test Automation Framework Architecture – What does it consist of? Version Controller Automation Tool Test Scripts Library Functions Config Files Data Files Loggers Application under test (AUT) Test artifacts Data Manager Test Runner Integration (CI)

9. Automation Tool Automation Tool Selection - Considerations Technical Information • Operating system support • Resource requirements Features & Usability • Installation • Test Execution • Result reporting Event Capturing • GUI abstraction features • GUI spying features • Capturing comfort Scripting Support • Supported languages • Object orientation • Compiler / Interpreter In-built Functions • Browser handling • XML handling • Interaction with File System • Wait /Sleep functions

10. Automation Tool Automation Tool Selection – Considerations continued… Integrated Development Environment • Editor features • Debugger • Version management Data Driven Testing • Supported file formats Result Reporting • Output formats • Flexibility for customization Documentation • Ease of use • Users manual • Online help Community Support • FAQs • WIKIs • Forums • Technical support

11. Automation Tool Focus on Selenium Selenium

12. Automation Tool Why Selenium?

13. Automation Tool • Other popular Automation Tools • Fulfills the following requirements • Win Runner • HP Quick Test Professional • IBM Rational Functional Tester • ParasoftSOATest • Rational Robot • Silk Test • QFTest • Test Partner Platform and OS independence Data driven capability Customizable result reporting Ease of debugging and logging Version control of tests Execute unattended runs Repeat tests as needed Extensible and Customizable Status notification Distributed test execution Automation Framework

14. Version Control System Git Purpose • Is a way of ensuring the evolution of the tests corresponds to the evolution of the application features • Allows concurrent updates to the same test • Maintains the history and baselines of the automated test • Is a free, fast and open source Distributed Revision Control System • Can work offline • Everyone has a local sandbox • Branching and Merging are fast and easy

15. Version Control System • Other Version Control Systems • Fulfills the following requirement • Clearcase • Subversion • Fossil • Aegis • Bonsai • Concurrent Version System (CVS) • Revision Control System (RCS) Platform and OS independence Data driven capability Customizable result reporting Ease of debugging and logging Version control of tests Execute unattended runs Repeat tests as needed Extensible and Customizable Status notification Distributed test execution Automation Framework

16. Continuous Integration System Purpose HUDSON • Gives early warning of broken or incompatible code and conflicting changes • Does immediate unit testing of all changes in code • Initiates and monitors repeated jobs like test automation runs at predetermined schedule or event • Is a Free, Open Source CI server written in Java • Monitors executions of repeated jobs like • Building a software project • Executing automated tests • Sending e-mail notifications • Can execute • Apache Ant and Apache Maven based projects • External jobs on a remote machine • Supports SCM tools including CVS, Subversion, Git and Clearcase • Easy to install and configure • Supports plug-ins with RSS/E-mail/IM Integration

17. Continuous Integration System • Other CI Systems • Fulfills the following requirements • CruiseControl • CruiseControl.NET • CruiseControl.rb • Apache Gump • Apache Continuum • BuildBot • CABIE Platform and OS independence Data driven capability Customizable result reporting Ease of debugging and logging Version control of tests Execute unattended runs Repeat tests as needed Extensible and Customizable Status notification Distributed test execution Automation Framework

18. Test Runner Features of RADAR • Is an automation framework component that provides a simple workflow for execution of tests, result collection, re-running failures, and comprehensive reporting • Is an application built in-house on top of JUnit • Some of its features are • Compile APIs/Tests as needed • Generate Test Suite as per requirement • Selective execution of Tests in unattended mode • Re-runs the failed tests for the number of times specified • Creates comprehensive report with total number of tests passed and failed, time taken by each test, reasons for failure and logs • Localization Support • Even Video capture!

19. Data Manager Features of ROSLIN • An essential feature for repeating tests and testing in parallel • Is a set of APIs (developed in-house) allowing users to save and restore data in the database and LDAP • Creates multiple identical database schema clones from a Master and assigns them to multiple AUT installations Virtualization could be used as well

20. Test Runner & Data Manager • Fulfill the following requirements Platform and OS independence Data driven capability Customizable result reporting Ease of debugging and logging Version control of tests Execute unattended runs Repeat tests as needed Extensible and Customizable Status notification Distributed test execution Automation Framework

21. Test Automation Framework Example Version Mgmt Automation Tool GIT JUnit tests Scripts XMLs Java APIs Annotations Log4j, Dom4j AUT SELENIUM RC Test artifacts Data Manager ROSLIN Test Runner Cont. Integrator RADAR HUDSON

22. Advantages - Flexibility Framework design considerations • Reduce maintenance efforts to a minimum • Provide a means to help people with lesser programming expertise to create automated tests • A common infrastructure for the Development and QA teams to build and execute automated tests What is a Solvent layer? • Solvents are a set of API that help create Selenium tests, without executing Selenium commands directly • Solvents package the logic and Selenium commands required to perform various user actions into simple, reusable classes • If you are wondering what’s in the name - Solvents reduce the amount of Selenium required to create a test (sometimes to none) thereby diluting Selenium, hence the name ‘Solvent’

23. Advantages - Flexibility Layout with Solvent Layer Additional Layer

24. Advantages - Flexibility Advantages of the additional layer • It aids reusability of code • They can be ‘wired’ together to form test scenarios • Changes need to be made at a single place and not in multiple tests • Makes the test / script more readable POINT TO NOTE – The automation framework is flexible enough to accommodate an additional layer as needed

25. Advantages - Scalability Other possibilities • The framework could be integrated with tools that help determine code coverage like Corbertura, EMMA and Jtest • Assess if the testing done is sufficient • Analyze if the testing focus is as planned • Mechanism for code review like Gerrit could be integrated with the framework • Enforce standards, conventions and best practices for creating tests • Inspect the overall quality of the tests

26. Advantages - Scalability Newly Added Gerrit Enhanced Version Mgmt Automation Tool GIT POINT TO NOTE – The automation framework can be enhanced by adding additional capabilities XMLs Java APIs Annotations Log4j, Dom4j Eclipse Ant JUnit Tests AUT SELENIUM RC Test artifacts Data Manager ROSLIN Test Runner Cont. Integrator Newly Added EMMA RADAR HUDSON

27. Benefits Software Development Ecosystem A consistent, flexible and scalable framework will help automation play a role at every stage in the software development process Not just test automation, some aspect of process automation is also in-built

28. Benefits Key Advantages • Maintainability – Significantly reduces the test maintenance effort • Reusability – Due to modularity of test cases and library functions • Manageability – Effective test design, execution, and traceability • Accessibility – To design, develop & modify tests whilst executing • Availability – Scheduled execution can run unattended on a 24/7 basis • Reliability – Due to advanced error handling and scenario recovery • Flexibility – Framework independent of AUT or environment • Measurability – Customizable reporting of test results ensure quality