Building an automation framework around open source technologies

1. Building an automation framework around open source technologies

2. Agenda • Challenges • Choosing an automation framework • Generic Architecture • How does it work? • Features of an ideal test automation framework • Additions to Selenium framework • Other components of test automation framework • Beyond test automation Datamatics Confidential for Limited Circulation Only

3. Challenges today Datamatics Confidential for Limited Circulation Only

4. ROI Facts Datamatics Confidential for Limited Circulation Only

5. Technology Facts • Over 73% of applications developed today are web enabled applications • Prevalence of Firefox, IE, Safari and Chrome as the major browsers, with multiple versions • Incompatibilities between these browsers primarily due to javascript implementation • Testing needs for Web Applications is multiplied due to permutations of these challenges Datamatics Confidential for Limited Circulation Only

6. Test Automation Facts • New requirements are introduced on a daily basis • Testing window is getting smaller day by day • Test automation can never replace human testers • Not all test can / should be automated • Test automation is a supplement to help / boost overall testing • Use test automation for area that are least prone to changes and yet require frequent tests Datamatics Confidential for Limited Circulation Only

7. Test Automation Tool Facts • Only people that claim 100% automation • Learning curve is very higher with every variant • Product support for emerging technologies • Extendibility Datamatics Confidential for Limited Circulation Only

8. Myth#1: Test automation is a goal never reached • Automation is a project on top of our goal list • Many testers intimidated with test automation • Developers don’t want to learn test automation • GUI automation is hard so lets not do automation • Automate everything and fire the testers • For some - panacea and others - waste of time Automation testing is an investment – will take time and effort to bring to fruition Datamatics Confidential for Limited Circulation Only

9. Myth#2: Record / Playback • Main feature recommended by any test automation tool vendors • Capture user actions and later on play them back for regression testing. • Simple steps to follow • Design test case in test automation tool • Use capture feature to record user actions • Enhance script with verification points • Add delay and wait state points for synchronization • Playback and observe results in log file Datamatics Confidential for Limited Circulation Only

10. Record / Playback Myth • Learning’s • Scripts resulting in hard-coded values • Costs associated with maintenance is astronomical • Scripts are often not reliable (pop-up, messages) • If tester makes a mistake – re-record the script • If application changes – re-record the script • Scripts cannot be recorded until software works Nothing is ever tested by this approach Datamatics Confidential for Limited Circulation Only

11. Test Automation using Open Source Framework Datamatics Confidential for Limited Circulation Only

12. Choosing a test automation approach • Test script modularity • Test library architecture • Keyword or table driven testing • Data driven testing • Hybrid-Keyword & Data driven both. Datamatics Confidential for Limited Circulation Only

13. Choosing a technology layer • Platform Independence • Execute unattended runs • Distributed test execution • Extensible and Customizable • Ease of debugging and logging • Status reporting • Repeat tests as needed (CI) • Version control Datamatics Confidential for Limited Circulation Only

14. Generic Architecture Data Manager Test Manager Version Control Automation Tool Test Artifacts Application Under Tests (AUT) Integration (CI) Datamatics Confidential for Limited Circulation Only

15. Choosing a Test Automation Tools • Abbot Java GUI Test Framework • Anteater • Apodora • Arbiter • Autonet • AutoTestFlash • Avignon • Blerby Test Runner • CanooWebTest • Capedit • Concordion • Crosscheck • csvdiff • CubicTest • Cucumber • DBFeeder • DbFit • Eclipse TPTP • EMOS Framework • Enterprise Web Test • Expect • expect-lite • Feed4TestNG • Frankenstein • GITAK • Harness • httest • IdMUnit • Imprimatur • ItIN - Infopath testing in .Net • ITP • ItsNat, Natural AJAX • WebDriver • WebInject • WebTst • ivalidator • Jacobie • Jameleon • jDiffChaser • Jemmy • JFunc: JUnit Functional Testing Extension • JSystem • Latka • Linux Test Project • LogiTest • LReport • MActor • Marathon • Maveryx • MaxQ • miTester for SIP • SIP Testing Tool • Mockito • MozUnit • OLVER • Om • org.tigris.mbt • PAMIE • Pounder • pywinauto • QAliber • QAT • QMTest • Rasta • Robot Framework • Sahi • Samie • Sauce Labs • Selenium • SharpRobo • soapui • STAF • Solex • SWAT • SWTBot • Systin • T-Plan Robot • tclwebtest • TestPlan • TextTest • Tomato • Toster • Watij • WatiN • Watir • WET Datamatics Confidential for Limited Circulation Only

16. Test automation tool Datamatics Confidential for Limited Circulation Only

17. How does it work? Datamatics Confidential for Limited Circulation Only

18. Why Selenium? • Support for Windows, Linux and Mac OS • Supports browsers – Explorer, Firefox, Safire • Supports AJAX calls • Non-developers can develop test cases • Graphical Selenium IDE • Export test cases as JUnit test cases • When test fails, Selenium can send an email w/ screenshot • Selenium Grid enabled test execution on multiple machines in parallel reducing overall execution time Datamatics Confidential for Limited Circulation Only

19. Features of an ideal test automation framework Datamatics Confidential for Limited Circulation Only

20. Features Missing within the selenium suite Datamatics Confidential for Limited Circulation Only

21. Additions to Selenium framework Reusable Framework Keywords Datasheets Business Components Utility / Generic Functions Reporting Functions Recovery Functions SELENIUM Application Under Test (AUT) Datamatics Confidential for Limited Circulation Only

22. Features of an ideal test automation framework Datamatics Confidential for Limited Circulation Only

23. Features of an ideal test automation framework Datamatics Confidential for Limited Circulation Only

24. Generic Architecture Data Manager Test Manager Version Control Automation Tool Test Artifacts Application Under Tests (AUT) Integration (CI) Datamatics Confidential for Limited Circulation Only

25. Continuous Integration • Apache Continuum • Apache Gump • Beebox • Bitten • Buildbot • Cabie • Cerberus • Control Tier Cruise Control cDash Draco Hudson Jhbuild LuntBuild Sin Tinderbox Datamatics Confidential for Limited Circulation Only

26. Version Control • Aegis • ArX • GNU arch • Bazaar • Bonsai • Cervisia • Codeville • Cogito • CVSNT • Darcs • Fossil • Git • Subversion • Libre Source • Mercurial • Monotone • Repo • CVS OpenCVS OpenGrok Revision Control System Apache Subversion SVN Tortoise Vesta DCVS Datamatics Confidential for Limited Circulation Only

27. Looking beyond automation testing • Test reports are automatically prepared • Defects are automatically captured in defect tracking system • Test metrics are updated automatically • Traceability matrix is updated automatically • Select test cases automatically based defect status updates • Calculating code and test coverage automatically • Suggesting improvements in test coverage Datamatics Confidential for Limited Circulation Only

28. Thank You!!!