A Comprehensive Framework for Testing Graphical User Interfaces

1. A Comprehensive Framework for Testing Graphical User Interfaces Atif M. Memon atif@cs.pitt.edu Dept. of Computer Science University of Pittsburgh Advisors: Mary Lou Soffa & Martha E. Pollack

2. Research focus 50% of code GUI Interactionsbetween the GUI and the underlying code Underlying Code

3. Challenges of GUI Testing • What is the test input to the GUI? • (Test case generation) • How much testing is enough? • (Test coverage) • Is the GUI executing correctly during testing? • (Test oracles) • What test results can be salvaged from the previous test runs to test a new version? • (Regression testing) • How to represent the GUI to handle all the above? • (Representation)

4. Current Techniques • Capture/replay tools • GUI test case generation • FSM [Esmelioglu and Apfelbaum] and VFSM [Shahady and Siewiorek] models • Genetic algorithm technique [Kasik and George] • Visual TDE for GUIs [Foster, Goradia, Ostrand, and Szermer]

5. Goals of My Research • Develop a comprehensive environment that integrates techniques (tools) to address various aspects of GUI testing • Goals of developing the techniques • Integrated: for compatibility • Automated: avoid manual repetition of tasks • Efficient: to meet tight testing deadlines • Robust: to recognize a GUI error as soon as it occurs and report the problem • Portable: apply to multiple platforms • General: apply to a wide variety of GUIs

6. Framework * * * * • New representation for GUIs • Test case generation based on AI planning [ICSE 1999, AIPS 2000, IEEE TSE Feb 2001] • New event-based coverage criteria [FSE 2001] • Unique test oracles [FSE 2000] • Repairing test cases for automated regression testing • Automated test executor [QW 2000] • A comprehensive framework that integrates all the above components

7. Overview of the Framework GUI Implementation: Tools (Languages/Toolkits) GUI Specifications GUI Representation GUI Implementer Executing GUI Regression Tester Test Oracle Test Case Generator Test Coverage Evaluator Test Executor

8. GUI Representation GUI Implementation: Tools (Languages/Toolkits) GUI Specifications GUI Implementer GUI Model Test Designer Executing GUI Regression Tester Test Oracle Test Case Generator Test Coverage Evaluator Test Executor

9. Creating the GUI Model • Modeling • GUI’s state in terms of objects & their properties • Events as state transducers • GUI’s hierarchical structure • GUI components • Event-flow graphs • Integration tree • Classifying events

10. Modeling the GUI’s State Optional Objects Caption(Button, “Cancel”) • A GUI at time T is modeled using: • Objects O = {o1, o2, o3, …, om} • Properties P = {p1, p2, p3, …, pl}, where pi is an ni-ary (ni>= 1) Boolean relation of the form: True/False Property(oj, oa, ob, …,ox, value) Property Name Optional value of Property Object GUI’s state: S = {p1,p2,p3,…,pn}

11. Example: Modeling the GUI’s State WState(Form1, Width(Form1, Scroll(Form1, wsNormal) 1088) TRUE) Caption(Button1, Enabled(Button1, Visible(Button1, Height(Button1, “Cancel”) TRUE) TRUE) 65) Form1 Button1 Label1 Align(Label1, Caption(Label1, Color(Label1, Font(Label1, alNone) “Files of type:”) clBtnFace) (tfont))

12. Caption Cancel TabOrder 0 All Properties of Button1

13. Determining Objects & Properties • Manual Examination • Specifications (reduced set) • GUI being tested • Toolkit/language (complete set) • All available properties Now we know how to represent the GUI’s state

14. GUI Events • GUI’s state is not static • Events change the GUI’s state • Events E = {e1, e2, e3, …, en}, associated with a GUI are functions from one GUI state Si to another state Sj • Notation: Sj = ei(Si)

15. Example: An Event Sj = e(Si) State: Si Background color is not yellow w19 set-background-color (w19, yellow) Event: e Background color is yellow State: Sj

16. Representing Events • Infeasible to give exhaustive specifications of the state mapping for each event • No set limit to the number of objects a GUI can contain at any point in time • There can be infinitely many states of the GUI • Model the GUI events using operators, which specify their preconditions and effects

17. Operators • An Operator is a triple • <Name, Precondition, Effects> • Name identifies an event and its parameters • Precondition is a set of positive literals • Effects is a set of positive or negative literals • Operator Op is applicable in any state Si in which: • All the literals in Precondition(Op) are TRUE • The resulting state Sj is determined by using Effects(Op) • All positive literals in Effects(Op), and • All literals that were TRUE in Si, except • Those that are negative in Effects(Op)

18. Operator Example Operator :: set-background-color Parameters: wX: window; col: color; Precondition: isCurrent(wX), background-color(wX, oldColor), oldColor != col. Effects: background-color(wX, col). set-background-color (w19, yellow) Primitive Operator Now we know how to represent the events of the GUI

19. Model GUI Hierarchically • Hierarchy • GUIs are designed as a hierarchy of components • Hierarchical model makes testing intuitive and efficient

20. Modal Windows in GUIs Main

21. Modal Windows in GUIs Main invokes Print

22. Modal Windows in GUIs Main Print Components Properties

23. Integration Tree Main FileNew FileSave FileOpen PageSetup Print ViewOptions FormatFont Properties • Definition: Integration tree is a triple <N, R, B> • N is the set of components in the GUI • RєN is a designated component called the Main component • B is the set of directed edges showing the invokes relation between components, i.e., (Cx, Cy) єB iff Cx invokes Cy.

24. Representing a Component follows Cut Copy Paste About … Contents … Open … Save Open … Save Definition: Event ex follows ey iff ex can be performed immediately after ey. File Edit Help Event-flow Graph

25. Event-flow Graph File Help Edit Open Save About Contents Cut Copy Paste To File, Edit and Help To File, Edit and Help • Definition: Event-flow graph is a 4-tuple <V, E, B, I> • V is the set of vertices, representing events, • E is the set of directed edges, showing the follows relationship, • B is the set of events first available (shown in red), • I is the set of events that invoke other components (dotted lines).

26. Classifying Events • Classification • A new classification of events aids in creating the hierarchical model of the GUI • Opening modal windows • Restricted-focus events • Closing modal windows • Termination events • Opening modeless windows • Unrestricted-focus events • Opening menus • Menu-open events • Interacting with underlying software • System-interaction events

27. GUI Modeling Steps • From the GUI’s specifications (formal/informal), • Identify objects and properties • Create operators for GUI events • Using the event classification, identify GUI components, create event-flow graphs and integration tree

28. Test Case Generator GUI Implementation: Tools (Languages/Toolkits) GUI Specifications GUI Implementer GUI Model Test Designer Executing GUI Regression Tester Test Oracle Tasks Test-Case Generator Test Coverage Evaluator Test Executor Test Case

29. GUI Test Cases • Individual user events • Not enough ! • Sequences of user events lead to different states • Test case: contains sequence of user events

30. Definition: GUI Test Case • legal event sequence • e1;e2;e3;… ;en is a legal event sequence • if (ei, ei+1) is an edge in an event-flow graph • or ei invokes component Cx and ei+1 is an event in Cx • A GUI test case is a triple • (S0, e1;e2;e3;…;en, S1; S2;… ;Sn) • S0 is a GUI state, and • e1;e2;e3;…;en is a legal event sequence • Si = ei(Si-1), 1 <= i <= n

31. A Test Case for WordPad SelectText (“text”) SelectText (“This”) OK Font 18 Format This is the text. This is the text. Underline OK Format Font S0 Expected State Sequence in Test Oracles

32. Selecting Event Sequences • Infinitely many • Manual • Expert chooses sequences • Automated • Randomly choose sequences • Structural (event-flow graph & integration tree) • Automatically generate events for COMMONLY USED TASKS Multiple Event Sequences This is the text. This is the text. Goal State Initial State

33. A Plan for a GUI Task SelectText(“This”) This is the text. SetFontSize(18) SelectText(“text”) MouseClick(U) This is the text. Initial State Goal State This is the text. This is the text.

34. Overview of Test Generation

35. Using Primitive Operators • One operator for each event Operator :: set-background-color Parameters: wX: window; col: color; Precondition: isCurrent(wX), background-color(wX, oldColor), oldColor != col. Effects: background-color(wX, col). set-background-color (w19, yellow)

36. Exploit the GUI’s Structure • Reduce the number of operators • System more efficient • Easier for the test designer

37. Operator Abstractions Two types of abstractions • Combine buttons  create system-interaction operators • Decompose GUI hierarchically  create component operators • Key idea: the user interacts with one component at a time.

38. Create System-Interaction Operators File Mail Recipient Send To Sys-Interaction Operator: File_SendTo_MailRecipient = <File + SendTo + MailRecipient >

39. Create Component Operators Set Language English (United States) Using Primitive Operators Only Default... OK Cancel Main GUI’s Operator Set Using Abstraction … Set Language SelectFromList() Default OK Cancel ... Language Window’s Operator Set Main GUI’s Operator Set SelectFromList() Default OK Cancel … Set Language ...

40. Using Component Operators Advantage Alternative decomposition leads to a new test case Language Window’s Operator Set SelectFromList() Default OK Cancel High Level Plan ... SetLanguage() ... Planner SelectFromList (“English(US)”) OK Sub Plan Define Abstraction SetLanguage() Component Operator

41. Effects of Exploiting the GUI’s Structure • Reduction in planning operators • 362 operators  32 operators • Ratio 10:1 for MS Wordpad • 20:1 for MS Word • Efficiency

42. Initial State This is the text. Goal State This is the text.

43. Test Case Planner Planner FormatFont Underline OK OK FormatFont 18 Mapping Mapping Font Format Format Font SelectText (“text”) SelectText (“This”) OK Font 18 Format Underline OK Format Font Primitive Operator Component Operator Component Operator Primitive Operator INITIAL GOAL FormatFont (“text”, Underline) FormatFont (“This”, 18pt) SelectText (“text”) SelectText (“This”)

44. Different from HTN Planning Primitive Operator Component Operator Component Operator Primitive Operator INITIAL GOAL SelectText (“This”) FormatFont (“text”, Underline) FormatFont (“This”, 18pt) SelectText (“text”) Planner Planner FormatFont Underline OK OK FormatFont 18 No Interactions

45. Alternative Test Case Primitive Operator Component Operator Component Operator Primitive Operator INITIAL GOAL FormatFont (“text”, Underline) FormatFont (“This”, 18pt) SelectText (“text”) SelectText (“This”) Primitive Operator Component Operator Primitive Operator Primitive Operator SelectText (“This”) FormatFont (“text”, Underline) SelectFromList (18) SelectText (“text”) SelectFromList (18) SelectText (“text”) SelectText (“This”) OK Format Font Underline

46. Experiments • Purpose • To determine whether planning is a feasible approach for GUI test case generation • Execution time • Human effort • Experimental design • GUI: our version of MS Wordpad (36 modal windows, 362 events) • Tasks: 50 tasks (initial & goal states) • Test cases: generated 290 test cases (6-56 events) using the IPP planner • Hardware platform: 350 MHz Pentium based machine, 256 MB RAM

47. Test Case Generation

48. Automated Execution

49. Test Oracles GUI Implementation: Tools (Languages/Toolkits) GUI Specifications GUI Implementer GUI Model Test Designer Executing GUI Oracle GUI State (run-time) Regression Tester Expected-state Generator Tasks Expected-state sequence Test-Case Generator Execution Monitor Test Coverage Evaluator Verifier Test Case Actual State Information Verdict Test Executor

50. Determine Correct Behavior Actual State Expected State Automated Verifier Verdict • To check the GUI’s state after each event • Approaches • Manual • Automated • Challenges • Generating expected state • Extracting actual state • Comparing expected & actual states