Software Testing Techniques and Quality Assurance Practices

1. Object-Oriented and Classical Software EngineeringFifth Edition, WCB/McGraw-Hill, 2002Stephen R. Schachsrs@vuse.vanderbilt.edu

2. CHAPTER 6 TESTING

3. Overview • Quality issues • Nonexecution-based testing • Execution-based testing • What should be tested? • Testing versus correctness proofs • Who should perform execution-based testing? • When testing stops

4. Testing • Two types of testing • Execution-based testing • Nonexecution-based testing

5. Testing (contd) • “V & V” • Verification • Determine if the phase was completed correctly • Validation • Determine if the product as a whole satisfies its requirements

6. Testing (contd) • Warning • “Verify” also used for all nonexecution-based testing

7. Software Quality • Not “excellence” • Extent to which software satisfies its specifications • Software Quality Assurance (SQA) • Goes far beyond V & V • Managerial independence • development group • SQA group

8. Nonexecution-Based Testing • Underlying principles • We should not review our own work • Group synergy

9. Walkthroughs • 4–6 members, chaired by SQA • Preparation—lists of items • Inspection • Up to 2 hours • Detect, don’t correct • Document-driven, not participant-driven • Verbalization leads to fault finding • Performance appraisal

10. Inspections • Five-stage process • Overview • Preparation, aided by statistics of fault types • Inspection • Rework • Follow-up

11. Fault statistics • Recorded by severity and fault type • Compare with previous products • What if there are a disproportionate number of faults in a module? • Carry forward fault statistics to the next phase

12. Statistics on Inspections • 82% of all detected faults (IBM, 1976) • 70% of all detected faults (IBM, 1978) • 93% of all detected faults (IBM, 1986) • 90% decrease in cost of detecting fault (Switching system, 1986) • 4 major faults, 14 minor faults per 2 hours (JPL, 1990). Savings of $25,000 per inspection • Number of faults decreased exponentially by phase (JPL, 1992) • Warning • Fault statistics and performance appraisal

13. Metrics for Inspections • Fault density (e.g., faults per KLOC) • Fault detection rate (e.g., faults detected per hour) • By severity (major/minor), by phase • What does a 50% increase in the fault detection rate mean?

14. Execution-Based Testing • Definitions • Failure (incorrect behavior) • Fault (NOT “bug”) • Error (mistake made by programmer) • Nonsensical statement • “Testing is demonstration that faults are not present”

15. What is execution-based tested? • “The process of inferring certain behavioral properties of product based, in part, on results of executing product in known environment with selected inputs.” • Inference • Known environment • Selected inputs • But what should be tested?

16. Utility • Does it meet user’s needs? • Ease of use • Useful functions • Cost-effectiveness

17. Reliability • Frequency and criticality of failure • Mean time between failures • Mean time to repair • Mean time, cost to repair results of failure

18. Robustness • Range of operating conditions • Possibility of unacceptable results with valid input • Effect of invalid input

19. Performance • Extent to which space and time constraints are met • Real-time software

20. Correctness of specifications • Incorrect specification for a sort • Function trickSort which satisfies this specification:

21. Correctness of specifications (coptd) • Incorrect specification for a sort: • Corrected specification for the sort:

22. Correctness • NOT necessary • NOT sufficient

23. Correctness Proofs • Alternative to execution-based testing

24. Example of Correctness Proof • Code segment to be proven correct

25. Example of Correctness Proof (contd) • Flowchart of code segment

26. Example of Correctness Proof

27. Correctness Proof Case Study • Never prove a program correct without testing it as well

28. Episode 1 • 1969 — Naur Paper • “Naur text-processing problem” Given a text consisting of words separated byblankor bynl (new line) characters, convert it to line-by-line form in accordance with following rules: (1) line breaks must be made only where given text has blankor nl ; (2) each line is filled as far as possible, as long as (3) no line will contain more than maxpos characters • Naur constructed a procedure (25 lines of Algol 60), and informally proved its correctness

29. Episode 2 • 1970 — Reviewer in Computing Reviews • The first word of the first line is preceded by blank unless the first word is exactly maxpos characters long

30. Episode 3 • 1971 — London finds 3 more faults • Including: • The procedure does not terminate unless a word longer than maxpos characters is encountered

31. Episode 4 • 1975 — Goodenough and Gerhart find three further faults • Including: • The last word will not be output unless it is followed by blank or nl

32. Proofs and Software Engineering • Lesson: • Even if product is proved correct, it must STILL be tested.

33. Three Myths • Software engineers do not have enough math for proofs • Proving is too expensive to be practical • Proving is too hard

34. Proofs and Software Engineering (contd) • Can we trust a theorem prover ? • How to find input–output specifications, loop invariants • What if the specifications are wrong? • Can never be sure that specifications or a verification system are correct

35. Proofs and Software Engineering (contd) • Correctness proofs are a vital software engineering tool, WHERE APPROPRIATE. • If • Human lives are at stake • Indicated by cost/benefit analysis • Risk of not proving is too great • Also, informal proofs can improve the quality of the product • Assert statement

36. Who Performs Execution-Based Testing? • Testing is destructive • A successful test finds a fault • Solution • 1. The programmer does informal testing • 2. SQA does systematic testing • 3. The programmer debugs the module • All test cases must be • Planned beforehand, including expected output • Retained afterwards

37. When Testing Can Stop • Only when the product has been irrevocably retired