Hierarchical GUI Test Case Generation Using Automated Planning

1. Hierarchical GUI Test Case Generation Using Automated Planning Atif M. Memon, Student Member, IEEE, Martha E. Pollack, and Mary Lou Soffa, Member, IEEE

2. Introduction • Graphical User Interfaces – important in interacting with today’s software • Makes software easier to use • Makes software testing more difficult than conventional software - must find a PATH from some initial state to a solution

3. What is a GUI? • Consists of labels, buttons, menus, and pop-up lists • User interacts with components to create events Object Window GUI Window Microsoft WordPad

4. Challenges in Testing GUI Applications • GUI’s becoming more complex  Complexity in testing correctness and underlying software • Enormous amounts of possible interaction • Determining coverage of Test Cases • Regression testing • Goal: Automatic generation of test cases • Solution: Planning Assisted Tester for grapHical User interface Systems

5. State of the Art Pros Cons Finite state machine models Automated test case generation Not scalable Scalable Semi-automated Test case quality depends on user expertise Mimic novice users A new technique both scalable and partially automated that reduces dependence on user expertise

6. 35,000 foot View of the Mountain PATHS… • Performs hierarchical structure of GUI  create hierarchical operators • Test designers describe operators(preconditions, effects)‏ • Takes goals as input and generates multiple sequences of events (“plans”) automatically • Plans becomes the test cases for GUI …easier to specify goals than all the possible PATHS toward the goal

7. PATHS: AI Planningwith Operators • Operator (preconditions, effects)‏ • Given to the AI planning algorithm • Guaranteed to result in the goal state when executed in the initial state Initial State Goal State

8. The PATHS Test CaseGeneration Process • Setup Phase – PATHS creates a hierarchical model of the GUI and returns list of operators from the model to test designer • Design preconditions and effects in simple, planning system language • Plan Generation Phase – Test designer describes scenarios by defining a set of initial and goal states for test case generation • PATHS generates a test suite for test case generation

9. Phase 1: Setup • PATHS partitions GUI into several classes using structural properties. • Menu-open Events – expand set of GUI events available to user • Unrestricted-focus Events – open GUI windows that do not restrict user’s focus, expand set of GUI events available to user (ex. Toolbar) • Restricted-focus Events – open GUI windows that have special property once invoked, monopolize GUI interaction, restrict user view of GUI to specific range until window is terminate (ex. Pop-up window) • System-interaction Events – interact with underlying software to perform some action

10. Phase 1: Operator Types • System-Interaction Operators – derived from those GUI events that generate interactions with the underlying software • Ex. Prefix_Suffix  File_Open = <File, Open> • Abstract Operators – represents a sequence of GUI events that invoke a window which monopolizes the GUI interaction • Treat this restricted-focuswindow as a separate planning problem and decompose

11. Phase 1: Setup Example GUI Events Planning Operators Open.Up File_ New File+ Open.Select File_ Open* Edit+ New Open.Cancel File_ Save Open* Open.Open File_ SaveAs* Save SaveAs.Up Save As* SaveAs.Select Edit_ Cut Cut SaveAs.Cancel Edit_ Copy Copy SaveAs.Save Paste Edit_ Paste

12. Phase 1: Operator Event Mapping • Transforms GUI menu events into planning operators These results Returned as Planning operators to the Designer …who then specifies preconditions and effects for each planning operator

13. Defined Task Planning Operators PATHS Tests Cases Achieving Goal Phase 2: Test Case Generation • Test case designer inputs defined operators into PATHS and identifies a task (initial state, goal state)‏

14. Abstract Operator Decomposition • Operators from the higher level need to be decomposed into lower level operations by • Inserting expansion from operator-event mappings, or • Making an additional call to the planner Hierarchical mechanism aids in regression testing since changes made to one component to do not necessarily invalidate all test cases.

15. Plan Generation: AI Planning Problem • Planning Problem – P(Λ, D, I G) • P(Operator Set, Finite Set of Objects, Initial State, Goal State)‏ • Solution – <S, O, L, B> • <Set of Plan Steps, Ordering constraints on S, Causal Links, Binding constraints on S> • Si < Sj – Step i must occur before Step j (not necessarily right before)‏ • Causal Links -- <Si, c, Sj> • Ensure no step threatens a required link

16. Partial Order Plans… • …aids in deriving total order plan • Add ordering constraints • i.e. Linearization of partial-order plan • … are solutions to a planning problem iff every consistent linearization of the partial-order plan meets the solution conditions Want to learn more about how the Authors implemented AI Planning…? We invite you to read about it in the paper. (IPP and HTN)‏

17. Evaluation: Empirical Approach Evaluation: Empirical Approach Evaluation: Empirical Approach • A PATHS prototype was developed based Microsoft WordPad • 325 GUI events • Reduced to 32 system-interaction and abstract operators 10-fold reduction of operatorsstreamline plan generation(phase 2 of PATHS)‏

18. Evaluation: Run Two Experiments • Two experiments were run using the PATHS prototype • Experiment 1: Generate test cases for multiple tasks • Is PATHS practical and efficient? • Experiment 2: Compare hierarchical vs. single-level test case generation • Is the hierarchical modeling approach more efficient than single-level?

19. Experiment 1: Generating Test Cases for Multiple Tasks • Use PATHS prototype to generate a suite of test cases for 9 WordPad tasks • Deploy experiment on Pentium-based computer running Linux on 200Mb RAM • Track times required to generate high-level plans and subplans PATHS is practical and efficient

20. Experiment 2: Hierarchical vs. Single-Level Test Case Generation • Use PATHS prototype to generate test cases for 6 WordPad tasks • Deploy experiment on same computer (Pentium-based running Linux on 200Mb RAM)‏ • Track plan lengths • Track times required to generate a single-level and a hierarchical test case Hierarchical test case generation is more efficient

21. Conclusions • Applying AI planning to the problem is a good match • Hierarchy of the model is efficient in generating test cases • 4- to 6-fold saving in plan length • Test case generated in seconds rather than hours • New technique is scalable when software releases new version, operators can be recycled • New technique is highly automated

22. PATHS: Contributions • Use well-known and used AI planning technique capable of solving large-space problems • Exploits structural features present in GUIs to reduce model size & complexity, and efficiently generate test cases • Hierarchical Planning makes regression testing easier • Increased portability – platform specific details incorporated last in test case generation • Reuse of commonly-appearing operator definitions

23. Open Questions • PATHS is practical, but is it useful? • Preliminary exploration achieves 20-fold reduction in GUI events • How does one measure usefulness? • How well does PATHS scale? • Manual analysis of operators will vary from application to application • How good are the test cases generated by PATHS? • Coverage • Negative test cases

24. Related / Future Work • Use library of operators to minimize human intervention • How to automatically generate preconditions and effects of operators