Automation Squirrel Systems

1. Automation @ SquirrelSystems Our automated solution at a glance By: Justin Hollander

2. About the Presenter Employed at Squirrel Systems since August 2007 as a QA Developer/Analyst Volunteered one year in Guatemala to help build an online system for the Central American district. Trained staff and volunteers to use the systems. Prepared manuals and other documentation BCIT Alumni and currently attending BCIT

3. About the Presentation Benefits of Automation Description of automation needs at Squirrel Systems Design goals vs. cost considerations Previous attempts at solving the automation / autoretest problem The current design of our autoretest solution Problems we overcame Future Considerations Questions and Answers

4. Our Automation Needs Intended to reduce overhead of manual certification checklists To allow other testers to do more functional / exploratory testing on other parts of the system that cannot be tested in an automated fashion Consistency in testing. Day in and day out.

5. Previous Attempts Attempt 1: Batch processing, relied on playback utilities, with an Access Database(DB) to manually track successes and failures of the running scripts Attempt 2: Improved batch processing(more readable scripts), relied on playback utilities, without the Access DB to track the successes and failures of the running scripts

6. Problems with Attempt 1 Script information and state were manually tracked Batch scripting is not as flexible or powerful as a programmatic solution to get the options you need for a comprehensive automated solution Results were not archived and therefore could not be linked Output was in a raw text only format and very cumbersome to process manually Too many false positives as comparison was blind, and not intelligent, nearly impossible to trust results

7. Problems with Attempt 2 Same as attempt 1, except now, without the Access DB, there was no way to track the running scripts Solution was scaled back due to resource constraints and the project had a hard time getting traction and support Ultimately, the project was suspended

8. What we Learned Automation/autoretest is more complex than we thought There is no �easy� solution Externalizing our test input data while using our playback utilities to retest the system is impossible At a minimum; output data must be analyzed better to eliminate false positives Batch scripting isn�t working A more dedicated approach is required

9. Autoretest, TAKE 3! We decided to try this again Decision to scrap previous work and start again from design phase Requirements were well understood, however, the right skill-set was missing until now

10. High level Design vs. Cost Design goals Cost Considerations To use existing playback tools within a framework to retest recorded �tracks� Use intelligent comparison algorithms to rule out false positives Output results in a human friendly format in order to ensure a quicker analysis of the potential problems Externalize the test input data from the system Re-usable parts to extend framework to all parts of POS system Limited to 40 man-hours a week(Basically me ?) Development team on a tight schedule and have limited resources for this project No money for off-the-shelf solution Management wanted a solution rather quickly(6 months) Money already invested; 2 previous attempts in solving this in last 5 years have failed

11. The Current Design Client/Server model Written in Java with easy to use administrative GUI Externalized data as much as possible Used intelligent output comparison algorithms One DB configuration supports one or more test scripts Supports XML processing Comparison output formatted in HTML Automatically detect if software that is being tested is the current build. Will update if needed

12. The Current Design cont.. Test outputs and comparison results archived to achieve the possibility of linking of historical data to present results Script information stored in a relational SQL DB Script information linked to GUI using custom written SQL driver Script states linked to scripts via GUI and they are not static Client has a head-less mode supported by multi-threaded timer mechanism Extensible due to its polymorphic nature and can easily support additional forms/levels of testing

13. Screenshot of GUI