User-session based Testing of Web Applications

1. User-session based Testing of Web Applications

2. Two Papers • A Scalable Approach to User-session based Testing of Web Applications through Concept Analysis • Uses concept analysis to reduce test suite size • An Empirical Comparison of Test Suite Reduction Techniques for User-session-based Testing of Web Applications • Compares concept analysis to other test suite reduction techniques

3. Talk Outline • Introduction • Background • User-session Testing • Concept Analysis • Applying Concept Analysis • Incremental Reduced Test Suite Update • Empirical Evaluation (Incremental vs. Batch) • Empirical Comparison of Concept Analysis to other Test Suite Reduction Techniques • Conclusions

4. Characteristics of Web-based Applications • Short time to market • Integration of numerous technologies • Dynamic generation of content • May contain millions of LOC • Extensive use • Need for high reliability, continuous availability • Significant interaction with users • Changing user profiles • Frequent small maintenance changes

5. User-session Testing • User session • A collection of user requests in the form of URL and name-value pairs • User sessions are transformed into test cases • Each logged request in a user session is changed into an HTTP request that can be sent to a web server • Previous studies of user-session testing • Previous results showed fault detection capabilities and cost effectiveness • Will not uncover faults associated with rarely entered data • Effectiveness improves as the number of sessions increases (downside: cost increases as well)

6. Contributions • View user sessions as use cases • Apply concept analysis for test suite reduction • Perform incremental test suite update • Automate testing framework • Evaluate cost effectiveness • Test suite size • Program coverage • Fault detection

7. Concept Analysis • Technique for clustering objects that have common discrete attributes • Input: • Set of objects O • Set of attributes A • Binary relation R • Relates objects to attributes • Implemented as a Boolean-valued table • A row for each object in O • A column for each attribute in A • Table entry [o, a] is true if object o has attribute a, otherwise false

8. Concept Analysis (2) • Identifies concepts given (O, A, R) • Concept is a tuple (Oi, Aj) • Concepts form a partial order • Output: • Concept lattice represented by a DAG • Node represents concept • Edge denotes the partial ordering • Top element T = most general concept • Contains attributes that are shared by all objects in O • Bottom element  = most special concept • Contains objects that have all attributes in A

9. Concept Analysis for Web Testing • Binary relation table • User session s = object • URL u = attribute • A pair (s, u) is in the relation table if s requests u

10. Concept Lattice Explained • Top node T • Most general concept • Contains URLs that are requested by all user sessions • Bottom node  • Most special concept • Contains user sessions that requests all URLs • Examples: • Identification of common URLs requested by 2 user sessions • us3 and us4 • Identification of user sessions that jointly request 2 URLs • PL and GS

11. Concept Analysis for Test Suite Reduction • Exploit lattice’s hierarchical use-case clustering • Heuristic • Identify smallest set of user sessions that will cover all URLs executed by original suite

12. Incremental Test Suite Update

13. Incremental Test Suite Update (2) • Incremental algorithm by Godin et al. • Create new nodes/edges • Modify existing nodes/edges • Next-to-bottom nodes may rise up in the lattice • Existing internal nodes never sink to the bottom • Test cases are not maintained for internal nodes • Set of next-to-bottom nodes (user sessions) form the test suite

14. Web Testing Framework

15. Empirical Evaluation • Test suite reduction • Test suite size • Replay time • Oracle time • Cost-effectiveness of incremental vs. batch concept analysis • Program coverage • Fault detection capabilities

16. Experimental Setup • Bookstore Application • 9748 LOC • 385 methods • 11 classes • JSP front-end, MySQL backend • 123 user sessions • 40 seeded faults

17. Test Suite Reduction • Metrics • Test suite size • Replay time • Oracle time

18. Incremental vs. Batch Analysis • Metric • Space costs • Relative sizes of files required by incremental and batch techniques • Methodology • Batch: 123 user sessions processed • Incremental: 100 processed first, then 23 incrementally

19. Program Coverage • Metrics • Statement coverage • Method coverage • Methodology • Instrumented Java classes using Clover • Restored database state before replay • Wget for replaying user sessions

20. Fault Detection Capability • Metric • Number of faults detected • Methodology • Manually seeded 40 faults into separate copies of the application • Replayed user sessions through • Correct version to generate expected output • Faulty version to generate actual output • Diff expected and actual outputs

21. Empirical Comparison ofTest Suite Reduction Techniques

22. Empirical Comparison of Test Suite Reduction Techniques • Compared 3 variations of Concept with 3 requirements-based reduction techniques • Random • Greedy • Harrold, Gupta, and Soffa’s reduction (HGS) • Each requirements-based reduction technique satisfies program or URL coverage • Statement, method, conditional, URL

23. Random and Greedy Reduction • Random • Selection process continues until reduced test suite satisfies some coverage criterion • Greedy • Each subsequent test case selected provides maximum coverage of some criterion • Example: • Select us6 – maximum URL coverage • Then, select us2 – most marginal improvement for all-URL coverage criterion

24. HGS Reduction • Selects a representative set from the original by approximating the optimal reduced set • Requirement cardinality = # of test cases covering that requirement • Select most frequently occurring test case with lowest requirement cardinality • Example: • Consider requirement with cardinality 1 – GM • Select us2 • Consider requirement with cardinality 2 – PL and GB • Select test case that occurs most frequently in the union • us6 occurs twice, us3 and us4 once • Select us6

25. Empirical Evaluation • Test suite size • Program coverage • Fault detection effectiveness • Time cost • Space cost

26. Experimental Setup • Bookstore application • Course Project Manager (CPM) • Create grader/group accounts • Assign grades, create schedules for demo time • Send notification emails about account creation, grade postings

27. Test Suite Size • Suite Size Hypothesis • Larger suites than: • HGS and Greedy • Smaller suites than: • Random • More diverse in terms of use case representation • Results • Bookstore application: • HGS-S, HGS-C, GRD-S, GRD-C created larger suites • CPM • Larger suites than HGS and Greedy • Smaller than Random

28. Test Suite Size (2)

29. Program Coverage • Coverage Hypothesis • Similar coverage to: • Original suite • Less coverage than: • Suites that satisfy program-based requirements • Higher URL coverage than: • Greedy and HGS with URL criterion • Results • Program coverage comparable to (within 2% of) PRG_REQ techniques • Slightly less program coverage than original suite and Random • More program coverage than URL_REQ techniques, Greedy and HGS

30. Program Coverage (2)

31. Fault Detection Effectiveness • Fault Detection Hypothesis • Greater fault detection effectiveness than: • Requirements-based techniques with URL criterion • Similar fault detection effectiveness to: • Original suite • Requirements-based techniques with program-based criteria • Results • Best fault detection but low number of faults detected per test case - Random PRG_REQ • Similar fault detection to the best PRG_REQ techniques • Detected more faults than HGS-U

32. Fault Detection Effectiveness (2)

33. Time and Space Costs • Costs Hypothesis • Less space and time than: • HGS, Greedy, Random • Space for Concept Lattice vs. space for requirement mappings • Results • Costs considerably less than PRG_REQ techniques • Collecting coverage information for each session is the clear bottleneck of requirements-based approaches

34. Conclusions • Problems with Greedy and Random reduction • Non-determinism • Generated suites with wide range in size, coverage, fault detection effectiveness • Test suite reduction based on concept-analysis clustering of user sessions • Achieves large reduction in test suite size • Saves oracle and replay time • Preserves program coverage • Preserves fault detection effectiveness • Chooses test cases based on use case representation • Incremental test suite reduction/update • Scalable approach to user-session-based testing of web applications • Necessary for web applications that undergoes constant maintenance, evolution, and usage changes

35. References • Sreedevi Sampath, Valentin Mihaylov, Amie Souter, Lori Pollock "A Scalable Approach to User-session based Testing of Web Applications through Concept Analysis," Automated Software Engineering Conference (ASE), September 2004. • Sara Sprenkle, Sreedevi Sampath, Emily Gibson, Amie Souter, Lori Pollock, "An Empirical Comparison of Test Suite Reduction Techniques for User-session-based Testing of Web Applications,"International Conference on Software Maintenance (ICSM), September 2005.