Automated Testing with SilkTest: Strategies That Really Work

1. Automated Testing with SilkTest: Strategies That Really Work Santa Clara Valley Software Quality AssociationSeptember 14, 1999 Presented by John W. GreenAutomation Expertise

2. Introduction • 6 years of automation experience • 15 years of software QA • 3.5 years at Segue as Trainer and Sr. Consultant • Founded Automation Expertise in 1998. • This document and SilkTest examples and FAQ are available at: www.automationexpertise.com You may need to go to http://207.114.165.79 Follow the link for Resources

3. Successful Automation • Skills • Tool/Software Marriage • Organizational Support • Realistic Goals • Training • Good Test Plans • Examples of Success

4. Skills • Two types of skills are critical to be successful as an automation engineer. • Testing • Programming • Sometimes people have one of the skills sometimes they have both. Good automated testers need to be subject matter experts. That means they understand exactly how the product they are testing shouldbehave. They know how users use the product. They can prioritize between the thousands of possible testcase to find the subset that are likely to: find bugs, provide significant code coverage.

5. Programming Skills • Good automated testers need to be good programmers. They need to be able to create intelligent, reusable, maintainable modules (classes/methods, functions and testcases) that are capable of behaving like a user.

6. More about Skills • Sometimes testers need to become detectives to determine exactly how the product should behave so that they can create a testcase which verifies that the product behaves as expected. This is not quite as simple at a company which doesn’t really have time for specifications. Some QA people at web companies say they’re lucky to do an acceptance test on every third modification to the live web site. • Most importantly, the tool’s learning curve should be consistent with the time frame in which the team want’s to use the tool. Setting realistic goals is critical.

7. Tool/Software Marriage • Should be a good marriage. • Tool should work with your hardware • Tool should be compatible with your development environment. • Tool should scale to allow “as close as possible” simulation of user interaction with the software. • You should at least believe that the vendor will continue to support your needs.

8. Organizational Support Automated Testing is often brought to a company amidst some controversy. The issues usually are: • Should we automate or not automate? • Will Automation cost us money? • Will Automation save us money? • Will Automation be too difficult? • Is Automation essential? • Did we choose the right tool?With mixed organizational support for automation from development, QA and others, the project is likely to fail.

9. Support Issues • An automation team can create some GUI testcases without support. They need a number of issues resolved to be successful. These include: • test data issues • custom object assistance • agreements on correct behavior • access to dedicated machines • unique databases • performance standards • GUI standards, etc. • configurations to be tested • These are all issues which the team and development should work together to resolve.

10. Realistic Goals • You can’t automate everything!!! • It will take time for the team to become experts. • Suggestion: Choose 25 testcases to automate. You can select a lot of tests from one product area or tests from a variety of areas. • Define them in excruciating detail. • Get assistance, if needed, at automating the 25 testcases you have selected. • Use this as a pilot project to see if the more tests can/should be automated.

11. More on Goals • The team will develop skills and the automation “guru” may appear during the pilot project. • Estimates of the cost of automation range from 1-8 times the cost of manual testing. Maintenance costs should be considered and will be relative to the number of changes to the application in the future and to how well the tests were written in the beginning. • The tools themselves are very expensive and must also be considered in the cost equation.

12. Selecting the Right Tool • Narrow things down to the vendors who say they will support your environment. • Choose 3-6 testcases and define them in detail. Select tests which exercise your application in several different ways. Ask each vendor to come in and automate these testcases while you watch.

13. Selecting Tools cont. • Then after they’ve left, rerun the tests, then see how the test perform when you move the window, cause errors, etc. • Next, compare the code each tool generated to see which one is more: readable, maintainable and flexible. See if your team can learn to use the language of the tool.

14. Things Automation Tools Can Do • Repeat boring tasks without complaining. • Repeat boring tasks without quitting. • Works all night. • Control several machines at once. • Get data out of a database easily. • Insert data into a database fairly easily.

15. More Things Automation Can Do • Reboot a remote NT machine and login as someone else. • Find all the links on a page that are outside the current domain and check them. • Create a test on one browser and play it back on another browser.

16. Advanced Automation Tools • Understand and recognize objects. • Allow intelligent recovery from errors. • Have object-oriented languages built-in. • Allow DLL calls. • Allow calls to application code. • Allow control of multiple machines. • Are cross-platform. • Are cross-browser.

17. Training • Develop the team’s skills. • Provide an in-house trainer, if possible. • Skilled contractors can perform this. function if their focus is on providing mentoring, training and support to full-time people. • All tool vendors offer classes. • It takes time to become expert. • Productivity increases as skill increases.

18. Good Test Plans • Test planning, I’ve left for next to last, but it is probably the most critical of the success factors for automation.Define what’s important to test.Decide how you’re going to execute the required test. For some tests, its obvious. If you need to stress test your web server with 500 users, you’ll need a tool to do that. However, for many of tests in your plan, you will have the choice of automation or manual execution.

19. Which Tests To Automate • Tests which are good candidates for automation include: • Time-consuming manual tasks • Functional tests of stable modules • Tests which will need to be repeated often. • Tests which are inexpensive to automate. • Stress/Load Tests • Multi-user scenarios • Either way, define what tests are important first, execute them and report the results. This should be done effectively no matter what execution method you select. • Test planning helps you to focus on what’s really critical in your application.

20. Success Example • Using SilkTest, we created a series of multi-machine scenario tests which could randomly select treeview items on each of six machines which were “clustered” together. Each treeview item launched an specific module. The “random” selections generated on each machine were repeatable by using a “random seed”. Also, the random items selected could be recorded and played back, at fairly close to the original sequence. The tests could be run for an unlimited amount of time.

21. Example 1 Results • This process resulted in the identification of more than 10 significant previously unidentified new major defects. Each defect we identified and which was fixed, lead us to another defect at another layer, using the same tests.

22. Success Example 2 • A team of manual testers created test plans using Excel, and then passed some of those plans to the automation team to be automated. Using the some advanced features of SilkTest, we created an interpreter that could read and execute the tests defined in the spreadsheets. The manual testers were unfamiliar with SilkTest. The language which the manual testers used to define tests was mutually agreed to, and involved the use of numerous functions defined to perform many of the test activities. • The focus of the test group shifted from execution of their tests to creation of new and different test cases.

23. Conclusions • Test Automation is development • Tests should be carefully planned • Tests should be carefully selected • Expectations should be very reasonable (low) for the first project. • Be concerned about hiring, developing and retaining the skill sets required for automation. • Let someone on the team learn the tool really well in order to take full advantage of its features. • Build modular, maintainable, well-documented flexible tests.