400 likes | 607 Views
It’s Automation, Not Automagic !. …or proving Einstein was right about insanity . Presented by Jim Hazen. Introduction. Jim Hazen Veteran of the Software Testing Trenches Experience in software testing, both commercial and consulting work. Working with automation and tools since 1989.
E N D
It’s Automation, Not Automagic! …or proving Einstein was right about insanity Presented by Jim Hazen
Introduction • Jim Hazen • Veteran of the Software Testing Trenches • Experience in software testing, both commercial and consulting work. • Working with automation and tools since 1989. • Implemented Automation Tools on DOS, OS/2, Windows and Web platforms. • Experience with tools; including HP/Mercury, IBM/Rational, and other ‘scripting’ languages.
Sage Words to Remember • “Insanity: doing the same thing over and over again and expecting different results.” – Albert Einstein
Agenda • What is ‘Test Automation’ • A Brief History • Different types of test automation, tools and uses • Automation Myths • How to correct them • Resources & Skills needed • Setting expectations for Management & Others • Summary • Q&A
What ‘Test Automation’ Really Is • “Automation” definition • Webster’s – “the technique of making an apparatus, a process, or a system operate automatically” • In Software Testing we use ‘tools’ to drive our processes • People only think of tools that execute a test as ‘Automation’ . • Other tools can be used for Test Design, Test Data Generation, Defect Tracking, Test Management, etc. • “Automation” is using a tool during any part of the process of testing.
A Brief History • Test “Execution” tools have been around a long time! • Mainframe – Home grown ‘macro’ tools1, Hyperion • PC DOS – AutoTester, SuperKey (macro tool) • PC GUI – Automated Test Facility (ATF), Rational Robot, QA Run, Winrunner, etc. • PC Web – QTP, SilkTest, Selenium, TestComplete, etc. • Mobile – MonkeyTalk, Robotium, etc.
Other Types of Test Automation Tools • Other Test Execution Tools • xUnit (JUnit & NUnit), SOAPUI, FitNesse, etc. • Test Design & Test Data Generation • Hexawise, AllPairs, Tosca Test Suite, etc. • Test Execution Management • HP ALM, MS Team Foundation Server, TestLink, etc. • Test Environment Management • VMWarevCloud Director, etc. • Defect Tracking & Reporting • Bugzilla, JIRA, etc.
Old Myths • You can automate 100% of your tests • Automation will “cure” all problems with testing • Automation tools are usable by anyone • Record/Playback is all you need • Automation allows Management to cut test staff • One tool is all you need • Automation will speed up testing process * Oldies, but goodies!!
New Myths • You don’t have to know how to program • Opensource tools are “free” • Commercial tools are “expensive” • GUI level automation is a waste of time • “Codeless” and “Scriptless” tools • Commercial tools can’t support agile • Developers only need to focus on Unit Tests
How to Correct Them • 100% Automation – Certain tests shouldn’t or can’t be ‘automated’. 80/20 Rule; automate 20% of functionality used 80% of time. • Automation solves all testing problems – it helps, but it creates new problems too. • Automation can be done by anyone – Not everyone has the ‘mindset’, it is both learned and inherent (logic). • Record/Playback is all you need – Proven time & time again to be false, and to fail.
How to Correct Them cont. • One tool is all you need – There isn’t a one-size fits all. • Automation speeds up testing process – The “Illusion of Speed” problem. Both execution and creation. • You don’t need to know programming – Anyone can Automate problem. • Opensource tools are free – Costs due to ramp up time (money) and skilled staff to do it (more expensive).2 • Commercial = Expensive – Not really in grand scheme of things.2
How to Correct Them cont. • GUI level automation is a waste of time – Not really
How to Correct Them cont. • Codeless/Scriptless tools – Just a different UI. You’re still writing a ‘test’ which is a set of ‘instructions’ (code). • Commercial tools don’t do Agile – Yes they do, it’s how you use the tool that counts. • Developers only create Unit Tests – Other tools require someone to ‘build’ the fixtures/code to ‘talk’ to the SUT.
Why Failures Happen Source: IDT Survey3 85% Failure Rate! Too High!!
Einstein was right! • We keep repeating history, why? • Always a ‘new’ group of management and testers. • Companies always looking to do testing faster, cheaper and with less resources. • Vendors and others who sell “Automagic”. • Instant gratification mindset. Automation takes time. • Not learning from past mistakes. • Software not designed with “testability” in mind. • Testers not involved early with decisions, not included by management.
Resources for Automation • Dedicated Staff • Skilled & experienced staff • Dedicated Equipment/Environments • Reduce risk of competition of resource or teams stepping on each other • Dedicated Budget • Automation isn’t cheap, plan appropriately • Dedicated Management Support • Can’t have any of the other 3 without it
Skills for Test Automation • It’s a different view and purpose for the script code. • Creating ‘software’ to ‘test’ software; i.e., PROGRAMMING! • Need basic understanding of programming methods. • Understand different types of technology layers • GUI, API, System, Web, Services, Database, etc. • You need to know how to program in different languages when needed. • C#, Java, VBScript, Jscript, XML, SQL, etc.
Test Automation Skills Cont. • Need to understand how to ‘hook’ the software under test and interact with it, and at what level. • API, or Object, or Service, or Protocol, etc. • Understand how tool ‘interacts with’ the software under test; what functions/methods are to use. • GUI Object ‘methods’ (functions) • Ex: SwfWindow(“LoginForm").SwfEdit(“ID”).SET “Jim” • Need to understand how to “pick apart” the software and “drive it” programmatically.
Axioms to remember • SMART • Standalone • Test/Script able to run by itself, minimize dependencies • Modular/Maintainable • Scripts and code need to be modular in design and easy to maintain • Appropriate • Build according to need for project, don’t try to automate 100% • Reusable • Build for reuse; keep script code compact and focused. • Testable & Traceable • Make it easy to debug your code, and be able to link together.
Axioms to remember cont. • SEARCH4,5 • Setup • Test/Script needs to check for pre-test conditions and flag if needed. • Execute • The logic and data needed to run the test itself • Analyze • Validations, Error Handling, etc. • Report • Progress, Error, Summation of run information (Logfile) • Cleanup • Test/Script needs to ‘cleanup’ (temp files, reset data, etc.) • Help (or Home)
Setting Realistic Expectations • Management & Other Groups • 100% automation is an unreachable goal.6,7 • Automation is its own form of software development. • Ramp-up time will be required, longer than you think. • Benefits realized over time, no immediate payback for investment.
Expectations cont. • Management & Other Groups • No tool supports all systems & technologies out of the box. • Not all testers can write scripts. Specialized resources are a must and need to be dedicated to the work. • Nothing is “Free”, there are always costs involved. • Don’t believe the marketing hype! Don’t buy the snake oil.8 There is no Magic!
Summary • What is ‘Test Automation’ – A lot more than you think! • History – Lots of tools over the last 30 plus years. • Different types of test automation tools – Not just for execution of a test only. • Automation Myths – Old and New; lots of failures due to them. • How to fix the problems – Be realistic, dispel the myths. • Resources and Skills needed for test automation – Again, a lot more than you think. It is Programming! • Setting expectations for Management & Others – Without proper expectations you are painted into a corner.
Remember… It’s automation, Not Automagic!
Contact Info • Jim Hazen • Company: Freelance Contractor • Home email: calkelpdiver@gmail.com • LinkedIn profile: http://www.linkedin.com/profile/view?id=2835000&trk=tab_pro
References • “Experiences of Test Automation: Case Studies of Software Test Automation”, Chapter 5, Graham & Fewster, Addison-Wesley, 2012 • “Test Automation - Let's Talk Business”, Igor Gershovich, Connected Testing, 2006. • “Implementing Automated Software Testing”, IDT Survey, ElfriedeDustin et al., Addison-Wesley, 2009. • “How We Test Software At Microsoft”, page 187, Alan Page, Microsoft Press, 2009. • “How to Automate Testing: The Big Picture“, Keith Stobie & Mark Bergman, 1992. • “Seven Steps to Test Automation Success”, Bret Pettichord, http://www.io.com/~wazmo/papers/seven_steps.html, 2001. • “Success with Test Automation”, Bret Pettichord, http://www.io.com/~wazmo/succpap.htm, 2001. • “Test Automation Snake Oil”, James Bach, 1996
Thank you for attending this session. Please fill out an evaluation form.