A Comprehensive Framework for Testing Graphical User Interfaces

1. A Comprehensive Framework for Testing Graphical User Interfaces Atif M. Memon atif@cs.pitt.edu Dept. of Computer Science University of Pittsburgh Advisors: Mary Lou Soffa & Martha E. Pollack

2. Research focus 50% of code GUI Interactionsbetween the GUI and the underlying code Underlying Code

3. Challenges of GUI Testing • What is the test input to the GUI? • (Test case generation) • How much testing is enough? • (Test coverage) • Is the GUI executing correctly during testing? • (Test oracles) • What test results can be salvaged from the previous test runs to test a new version? • (Regression testing) • How to represent the GUI to handle all the above? • (Representation)

4. Current Techniques • Capture/replay tools • GUI test case generation • FSM [Esmelioglu and Apfelbaum] and VFSM [Shahady and Siewiorek] models. • Genetic algorithm technique [Kasik and George] • Visual TDE for GUIs [Foster, Goradia, Ostrand, and Szermer]

5. Goals of My Research • Develop a comprehensive environment that integrates techniques (tools) to address various aspects of GUI testing • Goals of developing the techniques • Automated: avoid manual repetition of tasks • General: apply to a wide variety of GUIs • Efficient: to meet tight testing deadlines • Robust: to recognize a GUI error as soon as it occurs and report the problem • Portable: apply to multiple platforms

6. Framework * * * * • New representation for GUIs • Test case generation based on AI planning • New event-based coverage criteria • Unique test oracles • New automated regression testing algorithms • Automated test executor • A comprehensive framework that integrates all the above components

7. Overview of the Framework GUI Implementation: Tools (Languages/Toolkits) GUI Specifications GUI Representation GUI Implementer Executing GUI Regression Tester Test Oracle Test Case Generator Test Coverage Evaluator Test Executor

8. GUI Representation GUI Implementation: Tools (Languages/Toolkits) GUI Specifications GUI Implementer GUI Model Test Designer Executing GUI Regression Tester Test Oracle Test Case Generator Test Coverage Evaluator Test Executor

9. Creating the GUI Model • Modeling • GUI’s state in terms of objects & their properties • Events as state transducers • GUI’s hierarchical structure • Classifying events

10. Modeling the GUI’s State Optional Objects Caption(Button, “Cancel”) • A GUI at time T is modeled using: • Objects O = {o1, o2, o3, …, om} • Properties P = {p1, p2, p3, …, pl}, where pi is an ni-ary (ni>= 1) Boolean relation of the form: True/False Property(oj, oa, ob, …,ox, value) Property Name Value of Property Object GUI’s state: S = p1^p2^p3^… ^pn

11. Example: Modeling the GUI’s State Form1 WState(Form1, Width(Form1, Scroll(Form1, wsNormal) 1088) TRUE) Button1 Label1 Caption(Button1, Enabled(Button1, Visible(Button1, Height(Button1, Cancel) TRUE) TRUE) 65) Align(Label1, Caption(Label1, Color(Label1, Font(Label1, alNone) “Files of type:”) clBtnFace) (tfont))

12. Caption Cancel TabOrder 0 All Properties of Button1

13. Determining Objects & Properties • Specifications (reduced set) • GUI being tested • Toolkit/language (complete set) • All available properties Now we know how to represent the GUI’s state

14. GUI Events • GUI’s state is not static • Events change the GUI’s state • Events E = {e1, e2, e3, …, en}, associated with a GUI are functions from one GUI state Si to another state Sj • Notation: Sj = e1(Si)

15. Example: An Event This is the text. This is the text. This State: Si SelectText (“This”) Event: e State: Sj Sj = e(Si)

16. Representing Events • Infeasible to give exhaustive specifications of the state mapping for each event • No set limit to the number of objects a GUI can contain at any point in time • There can be infinitely many states of the GUI • Model the GUI events using operators, which specify their preconditions and effects

17. Operators • An Operator is a 3-tuple • <Name, Precondition, Effects> • Name identifies an event and its parameters • Precondition is a boolean expression • Effects is a sequence of ADD and DEL commands on properties • Operator Op is applicable in any state Si in which: • The Precondition(Op) holds • The resulting state Sj is determined by using Effects(Op) • ADD commands add their properties, and • DEL commands delete properties, • in the specified order

18. Operator Example Operator :: CUT Precondition: isCurrent(Menu2). Effects: FORALL Obj in Objects Selected(Obj)  ADD inClipboard(Obj) DEL onScreen(Obj) DEL Selected(Obj) ADD isCurrent(Menu1) DEL isCurrent(Menu2). File Edit View Ins Menu1 Cut Menu2 Primitive Operators Now we know how to represent the events of the GUI

19. Model GUI Hierarchically • Hierarchy • GUIs are designed as a hierarchy of objects • Objects are reused from libraries • Hierarchical model makes testing efficient • Classification • A new classification of events aids in creating the hierarchical model of the GUI

20. Classifying Events • Opening menus • Menu-open events • Opening modal windows • Restricted-focus events • Opening modeless windows • Unrestricted-focus events • Interacting with underlying software • System-interaction events

21. GUI Modeling Steps • From the GUI’s specifications (formal/informal), • Identify objects and properties • Create operators for GUI events • Using the event classification, create hierarchical and system-interaction operators • Details later

22. Test Case Generator GUI Implementation: Tools (Languages/Toolkits) GUI Specifications GUI Implementer GUI Model Test Designer Executing GUI Regression Tester Test Oracle Tasks Test-Case Generator Test Coverage Evaluator Test Executor Test Case

23. GUI Test Cases • Individual user events • Not enough ! • Sequences of user events lead to different states • Test case: sequence of user events

24. Definition: GUI Test Case • Event is defined as: • Sj = e(Si) • legal event sequence • e1;e2;e3;…;en is a legal event sequence • for state S0 • iff ExistsS0;S1;S2;…;Sn such that Si = ei(Si-1), 1 <= i <= n • A GUI test case is a pair • (S0, e1;e2;e3;…;en) • S0 is any state, and • e1;e2;e3;…;en is a legal event sequence

25. A Test Case for WordPad SelectText (“text”) SelectText (“This”) OK Font 18 Format This is the text. This is the text. Underline OK Format Font S0

26. Selecting Test Sequences • Infinitely many • Randomly choose sequences • Expert chooses sequences • Automatically generate events for COMMONLY USED TASKS Multiple Event Sequences This is the text. This is the text. Goal State Initial State

27. A Plan for a GUI Task SelectText(“This”) This is the text. SetFontSize(18) SelectText(“text”) MouseClick(U) This is the text. Initial State Goal State This is the text. This is the text.

28. Overview of Test Generation

29. Using Primitive Operators • One operator for each event Operator :: CUT Precondition: isCurrent(Menu2). Effects: FORALL Obj in Objects Selected(Obj)  ADD inClipboard(Obj) DEL onScreen(Obj) DEL Selected(Obj) ADD isCurrent(Menu1) DEL isCurrent(Menu2). File Edit View Ins Menu1 Cut Menu2

30. Exploit the GUI’s Structure • Reduce the number of operators • System more efficient • Easier for the test designer

31. Operator Abstractions Two types of abstractions • Combine buttons  create system-interaction operators • Decompose GUI hierarchically  create hierarchical operators

32. Create System-Interaction Operators File Mail Recipient Send To Sys-Interaction Operator: File_SendTo_MailRecipient = <File + SendTo + MailRecipient >

33. Create Hierarchical Operators Set Language English (United States) Using Primitive Operators Only Default... OK Cancel Main GUI’s Operator Set Using Abstraction … Set Language SelectFromList() Default OK Cancel ... Language Window’s Operator Set Main GUI’s Operator Set SelectFromList() Default OK Cancel … Set Language ...

34. Using Hierarchical Operators Language Window’s Operator Set SelectFromList() Default OK Cancel High Level Plan ... SetLanguage() ... Planner SelectFromList (“English(US)”) OK Sub Plan Define Abstraction SetLanguage() Hierarchical Operator

35. Effects of Exploiting the GUI’s Structure • Reduction in planning operators • 362 operators  32 operators • Ratio 10:1 for MS Wordpad • 20:1 for MS Word • Efficiency

36. Initial State This is the text. Goal State This is the text.

37. Test Case Planner Planner FormatFont Underline OK OK FormatFont 18 Mapping Mapping Font Format Format Font SelectText (“text”) SelectText (“This”) OK Font 18 Format Underline OK Format Font Primitive Operator Hierarchical Operator Hierarchical Operator Primitive Operator INITIAL GOAL FormatFont (“text”, Underline) FormatFont (“This”, 18pt) SelectText (“text”) SelectText (“This”)

38. Different from HTN Planning Primitive Operator Hierarchical Operator Hierarchical Operator Primitive Operator INITIAL GOAL SelectText (“This”) FormatFont (“text”, Underline) FormatFont (“This”, 18pt) SelectText (“text”) Planner Planner FormatFont Underline OK OK FormatFont 18 No Interactions

39. Alternative Test Case Primitive Operator Hierarchical Operator Hierarchical Operator Primitive Operator INITIAL GOAL FormatFont (“text”, Underline) FormatFont (“This”, 18pt) SelectText (“text”) SelectText (“This”) Primitive Operator Hierarchical Operator Primitive Operator Primitive Operator SelectText (“This”) FormatFont (“text”, Underline) SelectFromList (18) SelectText (“text”) SelectFromList (18) SelectText (“text”) SelectText (“This”) OK Format Font Underline

40. Methods to Generate Alternative Test Cases • Different results from planner • Hierarchical operator decompositions • Linearizations of the partial-order plan

41. Experiments • Purpose • To determine whether planning is a feasible approach for GUI test case generation • Execution time • Human effort • Experimental design • GUI: our version of MS Wordpad (36 modal windows, 362 events) • Tasks: 50 tasks (initial & goal states) • Test cases: generated 290 test cases (6-56 events) using the IPP AI planner • Hardware platform: 350 MHz Pentium based machine, 256 MB RAM

42. Test Case Generation

43. Automated Execution

44. Test Oracles GUI Implementation: Tools (Languages/Toolkits) GUI Specifications GUI Implementer GUI Model Test Designer Executing GUI Oracle GUI State (run-time) Regression Tester Expected-state Generator Tasks Expected-state sequence Test-Case Generator Execution Monitor Test Coverage Evaluator Verifier Test Case Actual State Information Verdict Test Executor

45. What Is Correct Behavior ?! SelectText (“This”) Font OK Format 18 This is the text. This is the text. This is the text. SelectText (“text”) Underline OK Format Font Check State, not only Output !!

46. Determine Correct Behavior Actual State Expected State Automated Verifier Verdict • To check the GUI’s state after each event • Approaches • Manual • Automated • Challenges • Generating expected state • Extracting actual state • Comparing expected & actual states

47. Overview of GUI Oracle Test Case Run-time information from executing GUI Expected-state Generator GUI Model Expected State Execution Monitor Actual State Verifier Oracle Verdict

48. Expected State e2 e3 en e1 • Obtaining Next State • Given a test case T with S0, the initial state, and • A sequence of events • The next state Sy is obtained from the current state Sx and operator Op • Delete property P from Sx if Effects(Op) contains the command “DEL P” • Add property P in Sy if Effects(Op) contains the command “ADD P”

49. Deriving Expected State S1 S2 S3 Sn • Obtain S1 = e1(S0) • And Si = ei(Si-1) en e2 e3 e1 S0 • Expected state is a conjunction of properties of the form • Property(Objects…, Value)

50. Obtaining Actual GUI’s State • Execution monitor • Compatible with expected state • Returns property(objects…, value) e.g., Caption(button1, “cancel”) • Actual state can be obtained by • Screen scraping • Queries