1 / 37

Software Testing: A Selection of Papers by Elaine Weyuker

Software Testing: A Selection of Papers by Elaine Weyuker. Peter Greenwood for EEE 599. Elaine Weyuker. PhD (CS) Rutgers U ‘77 Software Testing Researcher at AT&T Bell Labs, Murray Hill, NJ CS Professor Courant Institute of NYU 30+ papers, primarily on Software Testing. The Papers.

connie
Download Presentation

Software Testing: A Selection of Papers by Elaine Weyuker

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Software Testing:A Selection of Papers by Elaine Weyuker Peter Greenwood for EEE 599

  2. Elaine Weyuker • PhD (CS) Rutgers U ‘77 • Software Testing Researcher at AT&T Bell Labs, Murray Hill, NJ • CS Professor Courant Institute of NYU • 30+ papers, primarily on Software Testing EEE 599 - Weyuker Presentation

  3. The Papers • Axiomatizing Software Test Data Adequacy (TSE Dec 86) • The Automatic Generation of Load Test Suites and the Assessment of the Resulting Software (TSE Sep 95, with Avitzer) • Reliability Testing of Rule-Based Systems (Software Jun 96, with Avritzer and Ros) EEE 599 - Weyuker Presentation

  4. Goal of Presentation • highlight some of Weyuker’s prolific work in software testing • contrast the general and theoretical character of her early work with the specific and practical nature of her current work EEE 599 - Weyuker Presentation

  5. Axiomizing Software Test Data Adequacy (Dec 86) • How much testing is enough to determine if a program is acceptable? • very important question • relatively little research activity (1986) • no industry standards • test until testing time expired • test until all cases in current test set pass EEE 599 - Weyuker Presentation

  6. When Should Testing Stop? • Terminate when an adequacy criterion is met • a measure of how well the testing process has been performed • relates a test set to the program, the specification, or both • could relate a test set to the program’s operational profile EEE 599 - Weyuker Presentation

  7. Constraints of this Approach • deals only with dynamic testing (running the program on test cases) • does not involve any static testing (analysis of code, structure, etc.) EEE 599 - Weyuker Presentation

  8. Constraints (con’t) • Weyuker’s paper deals only with program-based criteria • ignores specifications • most adequacy criteria are of this type • easier to automate EEE 599 - Weyuker Presentation

  9. Weyuker on Testing “The purpose of testing is to uncover errors, not to certify correctness.” EEE 599 - Weyuker Presentation

  10. The Axioms (1 & 2) • Axiom 1: (Applicability) For every program, there exists a finite adequate test set. • Axiom 2: (Nonexhaustive Applicability) There is a program P and a test set T such that P is adequately tested by T, and T is not an exhaustive test set. EEE 599 - Weyuker Presentation

  11. Axioms 3 & 4 • Axiom 3: (Monotonicity) If T is adequate for P, and T is a subset of T’, then T’ is adequate for P. • Axiom 4: (Inadequate Empty Set) The empty set is not adequate for any program. EEE 599 - Weyuker Presentation

  12. Axioms 5 & 6 • Axiom 5: (Anti-extensionality) There are programs P and Q such that P is equivalent to Q, T is adequate for P, but T is not adequate for Q. • Axiom 6: (General Multiple Change) There are programs P and Q which are the same shape, and a test set T such that T is adequate for P, but T is not adequate for Q. EEE 599 - Weyuker Presentation

  13. Axiom 7 • Axiom 7: (Anti-decomposition) There exists a program P and a component Q such that T is adequate for P, T’ is the set of vectors of values that variables can assume on entrance to Q for some t of T, and T’ is not adequate for Q. EEE 599 - Weyuker Presentation

  14. Axiom 8 • Axiom 8: (Anti-composition) There exist programs P and Q such that T is adequate for P and P(T) is adequate for Q, but P is not adequate for P;Q. EEE 599 - Weyuker Presentation

  15. Summary of Axiomatizing • axioms are applied to 5 adequacy criteria • path, branch, statement, mutations, modified size • path, branch, statement fail application and antidecompositon • branch and statement also fail anticomposition EEE 599 - Weyuker Presentation

  16. Summary of Axiomatizing EEE 599 - Weyuker Presentation

  17. Criticism • Zweben & Gourlay, On the Adequacy of Weyuker’s Test Data Adequacy Axioms (IEEE TSE Apr 89) • “The properties proposed by Weyuker are neither fundamental enough nor precise enough to serve as a useful formal system for discussing adequacy or comparing criteria” EEE 599 - Weyuker Presentation

  18. More Criticism • Professor Norman Fenton, keynote address to 2nd QUALMS User Group Meeting, South Bank University, Jul 92 • “Elaine Weyuker … has presented a list of axioms … Measurement Theory reveals that some of these axioms are contradictory.” EEE 599 - Weyuker Presentation

  19. Dr. Weyuker’s Current Work EEE 599 - Weyuker Presentation

  20. Automatic Generation of Load Test Suites (Sep 95) • develops 3 algorithms for automatic test case generation • applies them to 5 real industrial software systems • assess the results EEE 599 - Weyuker Presentation

  21. Failure Types • Hard failure • complete loss of traffic ( ie cable break, power failure, router failure) • Soft failure • degraded performance (ie activity burst) • important that this does not occur • telecoms designed with excess capacity • avoid permanent soft failure EEE 599 - Weyuker Presentation

  22. Goals • a smoothly degrading system • graceful recovery EEE 599 - Weyuker Presentation

  23. Load Testing Strategies • system functionality has already been tested • assess system performance under a known load • black-box oriented • based on system operational profile EEE 599 - Weyuker Presentation

  24. Typical Telecom Operational Profile • number of different types of call • average service demands and probability distribution of each call type • average external arrival rate • routing probabilities for each node • this information is usually readily available EEE 599 - Weyuker Presentation

  25. Purpose of Algorithms • tests resource allocation mechanisms associated with software systems • system behavior under load is not measured by functionality testing • each variable vector is a distinct system state EEE 599 - Weyuker Presentation

  26. Basis of Algorithms • too many states to exhaustively test • calculate probability of system being in each state (from operational profile) • heuristically determine some threshold  • generate a test case for those states where P(S) >  EEE 599 - Weyuker Presentation

  27. The Algorithms • Deterministic State Testing (DST) • generate a test case for each state S where P(S) >  • DST with State Transition Checking (TDST) • as for DST, but also generate test cases for every S’ reachable from S EEE 599 - Weyuker Presentation

  28. The Algorithms (con’t) • DST with Length N Cycles (CDST) • as for DST, but also generate test cases for every S’ within a cycle of length N from S with a probability >  EEE 599 - Weyuker Presentation

  29. Target Systems • Project 1: telephone call rerouter • Project 2: 800 number call traffic monitor • Project 3: routing automator • Project 4: transaction processing system • Project 5: network interface software EEE 599 - Weyuker Presentation

  30. Testing Results • automated test generation routines were developed for all five projects • automated test results were presented for projects 1 & 2 EEE 599 - Weyuker Presentation

  31. Project 1 Results • telephone call rerouter software • test results accurately predicted system performance • testing revealed a serious architectural problem EEE 599 - Weyuker Presentation

  32. Project 2 Results • 800 number call traffic monitor • led to better requirements analysis • used automated test case generation exclusively, replacing hand-crafting • more test cases with more variety • uncovered four potentially severe faults • minimal additional cost EEE 599 - Weyuker Presentation

  33. Reliability Testing of Rule-Based Systems (Sep 96) • applies same automatic test case generation algorithm to an industrial network surveillance system • system was a reimplementation, so a large body of operational data was available from the original system • system designed to run 24 hours/day; 365 days/year EEE 599 - Weyuker Presentation

  34. Approach • system has very large state space • 355 days of operational data from 93 switches were used to identify 857 states entered • 0.999 coverage required testing 843 states EEE 599 - Weyuker Presentation

  35. State Testing Requirements EEE 599 - Weyuker Presentation

  36. Findings • historical data is useful for selecting test cases based on state probabilities • selection algorithm must consider • reliability requirements • size of the (limited) state space • comprehensiveness of individual test cases EEE 599 - Weyuker Presentation

  37. change of focus from theory to application extension from telecom to general application accomplished, respected, and prolific researcher in software testing and reliability Summary EEE 599 - Weyuker Presentation

More Related