1 / 62

Software Testing for Non-traditional Software

Software Testing for Non-traditional Software . Mary Lou Soffa University of Pittsburgh soffa@cs.pitt.edu Currently on sabbatical at IBM T.J. Watson Research. Shanghai, China December 4, 2003. Research Interests . Software Engineering Testing Regression testing - maintenance

mckile
Download Presentation

Software Testing for Non-traditional Software

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Software Testing for Non-traditional Software Mary Lou Soffa University of Pittsburgh soffa@cs.pitt.edu Currently on sabbatical at IBM T.J. Watson Research Shanghai, China December 4, 2003

  2. Research Interests • Software Engineering • Testing • Regression testing - maintenance • Debugging • Optimizing compilers • Particular code optimizations • Properties of optimizations • Specification/profitability • Connection: program analysis

  3. Non Traditional Software • Traditional software • Input • Program executes without user interaction and is self contained • Program produces results • Non Traditional • Event driven – user interacts with code as it executes – Graphical User Interfaces • Application operates in an environment • Database, physical environment

  4. Outline • Testing Graphical User Interfaces • Challenges – how currently done • Test case generation • Oracles – expected results • Regression testing

  5. Testing of Graphical Unit Interfaces (GUI) 50% of code GUI Interactionsbetween the GUI and the underlying code Underlying Code

  6. Challenges of GUI Testing • How much testing is enough? • (Test coverage) • Is the GUI executing correctly during testing? • (Test oracles) • What test results can be salvaged from the previous test runs to test a new version? • (Regression testing) • How to represent the GUI to handle all the above? • (Representation) • How to do the testing automatically without user’s involvement?

  7. GUI Framework • New representation for GUIs – event driven software • Test case generation based on AI planning • Unique test oracles • New event-based coverage criteria • New automated regression testing algorithms • Automated test executor • A comprehensive framework that integrates all the above components

  8. Overview of the Framework GUI Implementation: Tools (Languages/Toolkits) GUI Specifications GUI Representation GUI Implementer Executing GUI Regression Tester Test Oracle Test Case Generator Test Coverage Evaluator Test Executor

  9. GUI Representation GUI Implementation: Tools (Languages/Toolkits) GUI Specifications GUI Implementer GUI Model Test Designer Executing GUI Regression Tester Test Oracle Test Case Generator Test Coverage Evaluator Test Executor

  10. Creating the GUI Model • Modeling • GUI’s state in terms of objects & their properties • Events as state transducers • GUI’s hierarchical structure

  11. Optional Objects Caption(Button, “Cancel”) Modeling the GUI’s State • A GUI at time T is modeled using: • Objects O = {o1, o2, o3, …, om} • Properties P = {p1, p2, p3, …, pl}, where pi is an ni-ary (ni>= 1) Boolean relation of the form: Property(oj, oa, ob, …,ox, value) True/False Property Name Object

  12. Example: Modeling the GUI’s State Form1 WState(Form1, Width(Form1, Scroll(Form1, wsNormal) 1088) TRUE) Button1 Label1 Caption(Button1, Enabled(Button1, Visible(Button1, Height(Button1, Cancel) TRUE) TRUE) 65) Align(Label1, Caption(Label1, Color(Label1, Font(Label1, alNone) “Files of type:”) clBtnFace) (tfont))

  13. GUI Events • GUI’s state is not static • Events change the GUI’s state • Events E = {e1, e2, e3, …, en}, associated with a GUI are functions from one GUI state Si to another state Sj • Notation: Sj = e1(Si)

  14. This is the text. This is the text. This Example: An Event State: Si SelectText (“This”) Event: e State: Sj Sj = e(Si)

  15. Operator Example Operator :: CUT Precondition: isCurrent(Menu2). Effects: FORALL Obj in Objects Selected(Obj)  ADD inClipboard(Obj) DEL onScreen(Obj) DEL Selected(Obj) ADD isCurrent(Menu1) DEL isCurrent(Menu2). Menu1 Cut Menu2 Primitive Operators Now we know how to represent the events of the GUI

  16. GUI Modeling Steps • From the GUI’s specifications (formal/informal), • Identify objects and properties • Create operators for GUI events • Using the event classification, create hierarchical and system-interaction operators

  17. Test Case Generator GUI Implementation: Tools (Languages/Toolkits) GUI Specifications GUI Implementer GUI Model Test Designer Executing GUI Regression Tester Test Oracle Tasks Test-Case Generator Test Coverage Evaluator Test Executor Test Case

  18. GUI Test Cases • Individual user events • Not enough ! • Sequences of user events lead to different states • Test case: sequence of user events

  19. SelectText (“text”) SelectText (“This”) OK Font 18 Format This is the text. This is the text. Underline OK Format Font A Test Case for WordPad S0

  20. Selecting Test Sequences • Infinitely many • Randomly choose sequences • Expert chooses sequences • Automatically generate events for COMMONLY USED TASKS Multiple Event Sequences This is the text. This is the text. Initial State Goal State

  21. SelectText(“This”) This is the text. SetFontSize(18) SelectText(“text”) MouseClick(U) This is the text. A Plan for a GUI Task Initial State Goal State This is the text. This is the text.

  22. Overview of Test Generation

  23. Using Primitive Operators • One operator for each event Operator :: CUT Precondition: isCurrent(Menu2). Effects: FORALL Obj in Objects Selected(Obj)  ADD inClipboard(Obj) DEL onScreen(Obj) DEL Selected(Obj) ADD isCurrent(Menu1) DEL isCurrent(Menu2). File Edit View Ins Menu1 Cut Menu2

  24. File Mail Recipient Send To Create System-Interaction Operators Sys-Interaction Operator: File_SendTo_MailRecipient = <File + SendTo + MailRecipient >

  25. Effects of Exploiting the GUI’s Structure • Reduction in planning operators • 362 operators  32 operators • Ratio 10:1 for MS Wordpad • 20:1 for MS Word • Efficiency

  26. Initial State This is the text. Goal State This is the text.

  27. Planner Planner FormatFont Underline OK OK FormatFont 18 Mapping Mapping Font Format Format Font SelectText (“text”) SelectText (“This”) OK Font 18 Format Underline OK Format Font Test Case Primitive Operator Hierarchical Operator Hierarchical Operator Primitive Operator INITIAL GOAL FormatFont (“text”, Underline) FormatFont (“This”, 18pt) SelectText (“text”) SelectText (“This”)

  28. Test Case Generation

  29. Automated Execution

  30. Test Oracles GUI Implementation: Tools (Languages/Toolkits) GUI Specifications GUI Implementer GUI Model Test Designer Executing GUI Oracle GUI State (run-time) Regression Tester Expected-state Generator Tasks Expected-state sequence Test-Case Generator Execution Monitor Test Coverage Evaluator Verifier Test Case Actual State Information Verdict Test Executor

  31. ?! SelectText (“This”) Font OK Format 18 This is the text. This is the text. This is the text. SelectText (“text”) Underline OK Format Font What Is Correct Behavior Check State, not only Output !!

  32. Actual State Expected State Automated Verifier Verdict Determine Correct Behavior • To check the GUI’s state after each event • Approaches • Manual • Automated • Challenges • Generating expected state • Extracting actual state • Comparing expected & actual states

  33. Overview of GUI Oracle Test Case Run-time information from executing GUI Expected-state Generator GUI Model Expected State Execution Monitor Actual State Verifier Oracle Verdict

  34. e2 e3 en e1 Expected State • Obtaining Next State • Given a test case T with S0, the initial state, and • A sequence of events • The next state Sy is obtained from the current state Sx and operator Op • Delete property P from Sx if Effects(Op) contains the command “DEL P” • Add property P in Sy if Effects(Op) contains the command “ADD P”

  35. S1 S2 S3 Sn Deriving Expected State • Obtain S1 = e1(S0) • And Si = ei(Si-1) en e2 e3 e1 S0 • Expected state is a conjunction of properties of the form • Property(Objects…, Value)

  36. Experiments • Purpose: determine • Time to derive expected state • Time to execute monitor and verifier • Experimental design • GUI: our version of MS Wordpad (36 modal windows, 362 events) • Test cases: generated 290 test cases (6-56 events) using an AI planner • Hardware platform: 350 mhz pentium based machine, 256 MB RAM • Properties: reduced set • Level of checking: GUI relevant property set

  37. Deriving Expected State Total CPU time (all 290 test cases and expected states) 75.84 sec.

  38. Automated Execution GUI Relevant-properties checking Total running time < 10 minutes

  39. Regression Testing GUI Implementation: Tools (Languages/Toolkits) GUI Specifications GUI Specifications GUI Implementer GUI Model Test Designer Executing GUI Oracle GUI State (run-time) Regression Testing Algorithms Expected-state Generator Tasks Expected-state sequence Test-Case Generator Coverage Criteria Coverage Evaluation Algorithms Execution Monitor Verifier Test Case Coverage Report Actual State Information Verdict Test Executor

  40. A test case of length 3 Adobe Acrobat 5.0 File Document Security OK GUI Regression Testing Problem

  41. A test case of length 3 File Document Security OK Adobe Acrobat 6.0 GUI Regression Testing Problem Event “Document Security” no longer in menu Test case cannot be executed – unusable!!!

  42. A New Test Case File Security Document Properties Repair the old test case to obtain this new one OK

  43. Overview of our Regression testing • Traditional regression testing • Event-flow/component model • Repairing test cases • Case studies

  44. Traditional Regression Testing original test suite usable test cases unusable (obsolete) test cases discard select Selected usable test cases new test cases regression test suite • Regression test-all • Selective regression testing process

  45. Regression Testing for GUIs • Test case generation is manual – capture/replay tools • Small number of test cases • Each test case is valuable • Changes result in a large number of obsolete test cases • Structural and layout changes • Frequent modifications • Driven by constant user feedback – prototyping • Need frequent testing

  46. Our approach GUI Regression Testing original test suite usable test cases unusable test cases select repair not repairable Selected usable test cases new test cases discard repaired regression test suite

  47. GUI regression model • Model represent all legal sequences of events and event interactions • Two levels • Component-call-graph – built using the concept of modal window • Forms GUI component • GUI event flow graph – event structure of a component • Helps to isolate changes and repairs

  48. follows Cut Copy Paste About … Contents … Open … Save Open … Save Definition: Event ex follows ey iff ex can be performed immediately after ey. Representing a Component File Edit Help Forms event flow graph

  49. Component-Call Graph Main FileNew FileSave FileOpen PageSetup Print ViewOptions FormatFont Properties • Definition: Component-Call-Graph • Nodes are components in the GUI • Main is distinguished component • Edges represent the invokes relation between components, i.e., (Cx, Cy) є{edges} iff Cx invokes Cy.

  50. Event Flow Graph File Help Edit Open Save About Contents Cut Copy Paste To File, Edit and Help To File, Edit and Help • Definition: Event Flow FG • Vertices represent events • Edges show the follows relationship • Events first available (shown in red) • Events that invoke other components (dotted lines).

More Related