Software Testing

1. Software Testing Sudipto Ghosh CS 406 Fall 99(Also used later) November 2, 1999

2. Learning Objectives • Testing • What is testing? • Why test? • How testing increase our confidence in program correctness? • When to test? • What to test? • How to test? • Different types of testing • How to assess test adequacy? CS 406 Testing

3. Testing • The act of checking if a part or product performs as expected. • Why do we test? • Check if there are any errors? • Increase confidence in the “correctness” CS 406 Testing

4. Testing • When should we test? • Separate phase for testing? OR • At the end of every phase? OR • During each phase, simultaneously with all development and maintenance activities CS 406 Testing

5. What can be tested? • All products generated during lifecycle Phase Product How to test? Specification Req. doc Scenario construction, simulations Design Design doc Simulations Implementation Modules Run against test cases Integration System Run against test cases Maintenance System Regression testing CS 406 Testing

6. Our focus • Focus: Testing programs • Programs may be sub-systems or complete systems • Programs are written in a programming language • Explore techniques and tools for testing programs CS 406 Testing

7. Verification, Validation, Testing • Verification: • Process of determining whether a phase has been correctly carried out • Are we building the product right? (Boehm) • Validation: • Intensive evaluation process that takes place just before the product is delivered to the client • Are we building the right product? (Boehm) • V & V denotes testing • (Verification: mathematically prove program correctness) CS 406 Testing

8. Problems with formal verification • Adequate mathematical training • Economic viability of proofs • Proving hard for non-trivial products • Correctness of the “prover” itself • Finding input, output specifications, loop invariants • Wrong specifications CS 406 Testing

9. Terminology • Program - Sort • Collection of functions (C) or classes (Java) • Specification • Input: p - array of n integers, n>0 • Output: q - array of n integers such that • q[0]  q[1]  …  q[n] • Elements in q are a permutation of elements in p, which are unchanged • Description of requirements of a program CS 406 Testing

10. Tests • Test input (Test case) • A set of values given as input to a program • Includes environment variables • {2, 3, 6, 5, 4} • Test set • A set of test cases • { {0}, {9, 8, 7, 6, 5}, {1, 3, 4, 5}, {2, 1, 2, 3} } CS 406 Testing

11. Oracle • Function that determines whether or not the results of executing a program under test is as per the program’s specifications • Problems • Correctness of Oracle • Correctness of specs • Generation of Oracle • Need more formal • specs Software Under Test output Compare Failure? Success? Test Cases Oracle output CS 406 Testing

12. Correctness • Program Correctness • A program P is considered with respect to a specification S, if and only if: • For each valid input, the output of P is in accordance with the specification S • What if the specifications are themselves incorrect? CS 406 Testing

13. Errors, defects, faults • Often used interchangeably. • Error: • Mistake made by programmer. • Human action that results in the software containing a fault/defect. • Defect / Fault: • Manifestation of the error in a program. • Condition that causes the system to fail. • Synonymous with bug CS 406 Testing

14. Failure • Incorrect program behavior due to a fault in the program. • Failure can be determined only with respect to a set of requirement specs. • For failure to occur, the testing of the program should force the erroneous portion of the program to be executed. CS 406 Testing

15. Errors and failure Error revealing inputs cause failures Inputs Program Erroneous outputs indicate failures Outputs CS 406 Testing

16. Debugging • Suppose that a failure is detected during the testing of P. • The process of finding and removing the cause of this failure is known as debugging. • Testing usually leads to debugging. • Testing and debugging usually happen in a cycle. CS 406 Testing

17. Test-debug cycle Test Yes Failure? No Testing complete? No Yes Debug Done CS 406 Testing

18. Software quality • Extent to which the product satisfies its specifications. • Get the software to function correctly. • SQA group is in charge of ensuring correctness. • Testing • Develop various standards to which the software must conform • Establish monitoring procedures for assuring compliance with those standards CS 406 Testing

19. Nonexecution-Based Testing • Person creating a product should not be the only one responsible for reviewing it. • A document is is checked by a team of software professionals with a range of skills. • Increases chances of finding a fault. • Types of reviews • Walkthroughs • Inspections CS 406 Testing

20. Walkthroughs • Reviewer prepares two lists: • Items that the reviewer does not understand • Items that the reviewer believes are incorrect TEAM Rep from Specs team Manager from Specs team Client representative Rep of team from next phase Rep from SQA group CS 406 Testing

21. Managing walkthroughs • Distribute material for walkthrough in advance. • Includes senior technical staff. • Chaired by the SQA representative. • Task is to record fault for later correction • Not much time to fix it during walkthrough • Other individuals are trained to fix it better • Cost of 1 team vs cost of 1 person • Not all faults need to be fixed as not all “faults” flagged are incorrect CS 406 Testing

22. Managing walkthroughs (contd.) • Two ways of doing walkthroughs • Participant driven • Present lists of unclear items and incorrect items • Rep from specs team responds to each query • Document driven • Person responsible for document walks the participants through the document • Reviewers interrupt with prepared comments or comments triggered by the presentation • Interactive process • Not to be used for the evaluation of participants CS 406 Testing

23. Inspections • Proposed by Fagan for testing • designs • code • An inspection goes beyond a walkthrough • Five formal stages • Stage 1 - Overview • Overview document (specs/design/code/ plan) to be prepared by person responsible for producing the product. • Document is distributed to participants. CS 406 Testing

24. Inspections (contd.) • Stage 2 - Preparation • Understand the document in detail. • List of fault types found in inspections ranked by frequency used for concentrating efforts. • Stage 3 - Inspection • Walk through the document and ensure that • Each item is covered • Every branch is taken at least once • Find faults and document them (don’t correct) • Leader (moderator) produces a written report CS 406 Testing

25. Inspections (contd.) • Stage 4 - Rework • Resolve all faults and problems • Stage 5 - Follow-up • Moderator must ensure that every issue has been resolved in some way • Team • Moderator-manager, leader of inspection team • Designer - team responsible for current phase • Implementer - team responsible for next phase • Tester - preferably from SQA team CS 406 Testing

26. What to look for in inspections • Is each item in a specs doc adequately and correctly addressed? • Do actual and formal parameters match? • Error handling mechanisms identified? • Design compatible with hardware resources? • What about with software resources? CS 406 Testing

27. What to record in inspections • Record fault statistics • Categorize by severity, fault type • Compare # faults with average # faults in same stage of development • Find disproportionate # in some modules, then begin checking other modules • Too many faults => redesign the module • Information on fault types will help in code inspection in the same module CS 406 Testing

28. Pros and cons of inspection • High number of faults found even before testing (design and code inspections) • Higher programmer productivity, less time on module testing • Fewer faults found in product that was inspected before • If faults are detected early in the process there is a huge savings • What if walkthroughs are used for performance appraisal? CS 406 Testing

29. Comparison of inspection and walkthroughs • Inspection • Checklist used to find faults • Five step process • Formalized procedure in each step • Longer time • Walkthrough • No checklist used • Two step process • No formalized procedure in the steps • Shorter time CS 406 Testing

30. Strengths and weaknesses of reviews • Effective way of detecting faults • Faults detected early in development • Effectiveness reduced if software process is inadequate • Large scale software hard to review if it consists of smaller largely independent units (OO) • Need access to complete, updated documents from previous phase CS 406 Testing

31. Testing and code inspection • Code inspection is generally considered complimentary to testing • Neither is more important than the other • One is not likely to replace testing by code inspection or by formal verification CS 406 Testing

32. Program testing • Testing is a demonstration that faults are not present. • Program testing can be an effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence. - Dijkstra CS 406 Testing

33. Execution-based testing • Execution-based testing is a process of inferring certain behavioral properties of a product, based, in part, on the results of executing the product in a known environment with selected inputs. • Depends on environment and inputs • How well do we know the environment? • How much control do we have over test inputs? • Real time systems CS 406 Testing

34. Utility • Extent to which a user’s needs are met when a correct product is used under conditions permitted by its specs. • How easy is the product to use? • Does it perform useful functions? • Is it cost-effective? CS 406 Testing

35. Reliability • Probability of failure-free operation of a product for a given time duration • How often does the product fail? • mean time between failures • How bad are the effects? • How long does it take to repair it? • mean time to repair • Failure behavior controlled by • Number of faults • Operational profile of execution CS 406 Testing

36. Robustness • How well does the product behave with • range of operating conditions • possibility of unacceptable results with valid input? • possibility of unacceptable results with invalid input? • Should not crash even when not used under permissible conditions CS 406 Testing

37. Performance • To what extent does the product meet its requirements with regard to • response time • space requirements • What if there are too many clients/ processes, etc... CS 406 Testing

38. Levels of testing • Unit testing • Integration testing • System testing • Acceptance testing CS 406 Testing

39. Test plan • Test unit specification • Features to be tested • Approach for testing • Test deliverables • Schedule • Personnel allocation CS 406 Testing

40. References • Textbook • Scach - Classical and Object-Oriented Software Engineering • Other books • Beizer - Software Testing Techniques • Beizer - Black-Box Testing: Techniques for Functional Testing of Software and Systems • Marick - The Craft of Software Testing • Myers - The Art of Software Testing CS 406 Testing