An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing

1. An Exploration of Errors in Web-based Applications in the Contextof Web-based Application Testing PhD Proposal Kinga Dobolyi May 2009

2. The shopping cart

3. The shopping cart

4. The shopping cart

5. What is going on • Problem: faults in web-based applications cause losses of revenue, and they are hard to test • Approach: study errors in web-based applications in the context of web testing • Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing

6. Outline • Introduction and motivation • Thesis statement • Background • Goals and approaches • Preliminary work • Expected contributions

7. Motivation • Testing of web-based applications in particular deserves further examination due to economic considerations: • Monetary throughput: Backbone of e-commerce and communications businesses • Customers: low customer loyalty • Development: Companies are choosing not to test due to resource constraints

8. Motivation: e-commerce • Internet usage: 73% of people in the US in 2008 • Browsers are dominant application • $204 billion in Internet retail sales annually • Global online B2B transactions total several $trillions annually • One hour of downtime at Amazon.com cost $1.5 million dollars • 70% of major online sites exhibit user-visible failures

9. Motivation: customers • Customer loyalty is notoriously low • Determined by the usability of the application [Offutt 2002] • Freedom and options

10. Motivation: customers • Lesson learned: web-based applications need to be well-designed and well tested • Are they?

11. Motivation: development • Technology challenges: • Heterogeneous, opaque components • Dynamic page content generation • Persistent state operated upon by concurrent, global users around the clock

12. Motivation: development • Web-based applications are often not tested • Enormous pressure to change • Short delivery times, high developer turnover rates, and quickly evolving user needs • No formal process model

13. Motivation: summary • Problem: faults in web-based applications cause losses of revenue, and they are hard to test • Approach: study errors in web-based applications in the context of web application testing • Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing

14. Thesis statement • Hypothesis: web-based applications have special properties that can be exploited to build tools and models that improve the current state of web application testing and development: • Tend to fail in predictable and similar ways • Human centric definition of acceptability

15. Thesis statement • Problem: faults in web-based applications cause losses of revenue, and they are hard to test • Approach: study errors in web-based applications in the context of web testing • Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing

16. Background: testing techniques • Non-functional (static) validation • Server load testing • Link testing • HTML/spelling validation • Modeling approaches • Capture-replay • User session-based testing

17. Background: oracles • Oracles (oracle-comparator) 1 <<P>The same table could be indented. 2 <<TABLEborder="1"> 3 --- 4 ><p>The same table could be indented.</p> 5 ><tableborder="1"summary=""> • False positives from diff-like tools • Want precise comparators

18. Background: automation • Automation • Test case generation: VeriWeb, PHP • Test case replay • URL + post-data • Failure detection

19. Background: metrics • How do we measure success? • Code coverage • Fault seeding • Human • Automatic • Cost • How do we know these are indicative of the real world?

20. Background: fault definition • Defining an error: • “the inability to obtain and deliver information, such as documents or computational results, requested by web users.” [Ma & Tian 2003] • Fault taxonomies • Figure from [Marchetto et al 2007]

21. Background: challenges • Functional validation remains a challenge • Regression testing should be more precise and automatic • We do not know if test suite efficacy metrics are indicative of the real world • We should examine the severity of uncovered faults

22. Goals and approaches • Problem: faults in web-based applications cause losses of revenue, and they are hard to test • Approach: study errors in web-based applications in the context of web testing • Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing

23. Goals and approaches • I propose to: • Model errors in web-based applications • Identify them more accurately • Automate the oracle-comparator process • Make web testing more cost-effective • Devise a model of fault severity that will guide test case design, selection, and prioritization • Validate or refute the current underlying assumption that all faults are equally severe in fault-based testing

24. Goals and approaches: Goals • Reduce the cost of regression testing web-based applications • Use special structure of web-based application output to precisely identify errors • Automate web-based application regression testing • Unrelated web-based applications tend to fail in similar ways • Understand customer-perceived severities of web application errors.

25. Goals and approaches: Goals • Formally ground the current state of industrial practice • Validate or refute fault injection as a standard for measuring web application test suite quality • Understand how to avoid high-severity faults during web application design and development • Reduce the cost of regression testing web applications by exposing high-severity faults • Test case design, selection, and prioritization (test suite reduction)

26. Goals and approaches: Outline

27. Goals and approaches: Step 1 – oracle-comparator • Construct a precise oracle-comparator that uses the tree-structured nature of XML/HTML output and other features • Model errors on a per-project basis • Semantic distance metric to reduce false positives

28. Goals and approaches: Step 2 – automated oracle-comparator • Exploit the similar way in which web applications fail to avoid the need for human annotations in training a precise oracle-comparator • Train a precise oracle-comparator on data from other, unrelated web applications • Use fault injection to improve the results when necessary

29. Goals and approaches: Step 3 – human study • Conduct a human study of real-world fault severity to identify a model of fault severity • Severities different than self-reported in bug repositories • Screenshots of current-next idiom • Also survey developers

30. Goals and approaches: Step 4 – fault seeding validation • Compare the severities of real-world faults to seeded faults using human data (validate fault seeding) • The severities of seeded errors have uniform distributions? • The severity distribution of seeded errors matches the distribution of real-world errors, according to the results of the survey from Step 3?

31. Goals and approaches: Step 5 – software engineering guidelines • Identify underlying technologies and methodologies that correlate with high-severity faults • As an alternative to testing • Tie high severity errors to underlying code, components, programming languages, and software engineering practices

32. Goals and approaches: Step 6 – testing techniques • Identify testing techniques to maximize return on investment by targeting high-severity faults • Introduce a new metric for the (web application) test suite reduction research community

33. Preliminary Work: Step 1 • Step 1: Construct a precise oracle-comparator using tree structured XML/HTML output and other features • Multiple versions of 10 open-source benchmarks • 7154 pairs of oracle- testcase output, 919 of which labeled as “possibly an error”

34. Preliminary Work: Step 1 • Evaluation: F-measure (precision and recall) using our model

35. Preliminary Work: Step 1 • Longitudinal study to measure effort saved • Calculate cost of looking : cost of missing • Low ratio means we are saving effort

36. Preliminary Work: Step 2 • Step 2: Exploit similarities in web application failures to avoid human annotations when training a precise oracle-comparator • Same setup as Step 1 • Use existing, annotated pairs of test-oracle output from unrelated applications to train a comparator for the application at test

37. Preliminary Work: Step 2 • Evaluation: measure precision and recall

38. Preliminary Work: Step 2 • Use fault seeding to introduce project-specific faults into the training data set

39. Preliminary Work: Step 3 • Step 3: Model real-world fault severity based on a human study. • Collect 400 real-world faults • Evaluation: have subjects use the model to classify faults

40. Preliminary Work: Step 4 • Step 4: Compare the severities of real-world faults to seeded faults using human data. • Test same human subjects with 200 + 200 manually-injected and automatically-injected faults • Conduct a survey of developers for fault severity

41. Preliminary Work: Step 5 • Step 5: Identify technologies and methodologies that correlate with high-severity faults • Do high severities correlate with: • Programming Language • Level in three-tier architecture • COTS component • User error (usability issue) • Type of error (business logic, resource allocation, syntax error, etc) • Fault taxonomies (existing) • Surface features of visible output: white screens, stack traces, misspellings, formatting errors • Evaluation: developer survey (time permitting)

42. Preliminary Work: Step 6 • Step 6: Identify testing techniques to target high-severity faults • Targets testing • Assign a testcase a severity rating a priori • Evaluation: compare the severity of faults exposed with our technique versus other test suite reduction approaches

43. Expected Contributions • Problem: faults in web-based applications cause losses of revenue, and they are hard to test • Approach: study errors in web-based applications in the context of web testing • Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing

44. Expected Contributions • Reduce the cost of regression testing by constructing a precise oracle-comparator • Develop a model of customer-perceived severities of web application faults • Validate or refute fault injection as a standard for measuring web application test suite quality • Propose new software engineering guidelines for web application development and testing

45. Questions?

46. Original Contributions • Fault Severity Model • Severity has not been studied in this domain, and customers are an integral part of these applications • Providing a new metric to the research community • Validate/refute fault seeding • Precise-oracle comparators • First to use different versions of benchmarks • Can be completely automated • XML and HTML

47. Expected Impact • Fault Severity Model • Can be applied to testing techniques in this field to make them financially feasible for developers • Change the way in which test suite efficacy is measured • Potentially impact web site design as usability issues may become more evident • Precise oracle-comparator • Automation makes it much more feasible for adoption than existing techniques • Potentially allow companies to conduct regression testing if they were not testing beforehand

48. Timeline • Steps 1 and 2: precise comparators • Completed, 2 papers under submission • Steps 3 and 4: human study • Data collection completed, analysis under way for submission of 1 paper • Expected completion by September • Step 5: software engineering guidelines • Expected completion by October • Expected 0.5 paper • Step 6: testing according to fault model • Expected completion by February • Expected 1 paper