Software Testing

1. Software Testing CS 470

2. Testing • Goal is to find faults • What kind of faults? • Algorithmic • Computation and Precision • Documentation • Overload • Capacity • Timing • Performance • Recovery • System • Standards

3. Algorithmic Fault • Code’s logic does not produce the proper output for a given input • Some examples • Branch under wrong conditions • Fail to set loop invariant • Fail to initialize variables • Comparing inappropriate data types • Syntax faults may help find algorithmic faults • E.g. missing { } around a loop, require even for loops with one statement

4. Computation and Precision Faults • Formula is implemented incorrectly • Computation not performed to the required accuracy • E.g. precision errors we saw with the Patriot Missile System

5. Documentation Faults • Documentation typically derived from the program design • Implementation may be performed differently than specified in the documentation • Improper documentation of an API can lead to further faults as other developers use the documentation to write code

6. Stress or Overload Faults • Data structures filled past their capacity • Buffer overflow • Dimension of tables larger than code can handle • Length of queue too great

7. Capacity Faults • Performance becomes unacceptable as activity reaches system limit • E.g., system capable of handling 1,000,000 records becomes slower and slower as more records are added • May be examined in relation to disk access, number of processes, etc.

8. Timing or Coordination Faults • Race conditions • Specific inter-process communication • May be particularly difficult to detect • Hard to replicate a fault • Many states usually possible among each process; even higher number of states when considered in conjunction

9. Throughput or Performance Faults • System does not perform at the speed prescribed by the requirements

10. Recovery Faults • System does not recover from a failure as specified

11. Hardware and System Faults • Problem with underlying hardware or operating system • Disk failure • Out of memory • Process table full • Software doesn’t adequately handle the system fault

12. Standards and Procedures Faults • Organizational standards and procedures not followed, e.g. • E.g., always add header to each method with comments • Always use the letter i preceding an integer variable, s preceding a String, etc. • May not affect the running of programs, but may foster an environment where faults are created

13. Some other Faults used at IBM • Interface • Fault in interacting with another component or driver • Checking • Fault in logic that fails to validate data properly before it is used • Build/Package/Merge • Fault that occurs because of problems in repositories and version control

14. Back to Testing… • Goal is to find the faults • Different types of testing • Unit testing • Integration testing • Functional testing • Performance testing • Acceptance testing • Installation testing

15. Unit Testing • The first opportunity to test whether a particular module or component is complete • Often performed by the developer • Can choose test cases in many ways. Some sample to get broad test coverage: • Statement testing : Every statement is executed at least once • Branch testing : Every decision point is seen at least once. • Path testing : Variation of above, each path through the code tested at least once (visualize code as a flow chart) • Definition-use path testing : Every path from every definition of every variable to every use of that definition is exercised

16. Integration Testing • Integration • Piecing together individual components to make a working system • Individual components should already have passed their unit tests • Integration Testing • Tests to ensure the system functions when components pieced together

17. Integration Testing • Bottom-up • Low-level first, with test harness • Top-Down • Stubs • Big-Bang • Put them together and see if it works • Sandwich • System viewed as three layers • Top-down in top layer, bottom-up in lower, converge in the middle

18. Comparison of Integration Strategies

19. Acceptance Testing • Enable customers and users to determine if the system really meets their needs and expectations • Benchmarks • Test cases under typical conditions to examine performance • Pilot test • Users exercise system on an experimental basis • Alpha : Performed in-house • Beta : Performed on customers • Parallel testing • If replacing in old system, use both in parallel to allow users to gradually become accustomed to the new system • Longevity test • Can the software function if running for X hours?

20. Results of Acceptance Tests • Ideally: acceptance testing uncovers discrepancies in requirements • Reality: by working with the system users discover aspects of the problem which they were not aware • New requirements • New development and features • Other direction: • Some requirements may not be needed

21. Installation Testing • Installing the system at the user site • System-specific configuration • Co-existence test with other software • E.g. does it work with Office software? • Does anything break if the software is removed?

22. Test Cases • Want a broad set of test cases to cover the range of possible values and code paths • Closed Box • Apply all possible inputs, compare with expected output according to requirements • Includes “out of range” inputs • Open Box • View code’s internal structure • Generate tests based on this structure, e.g. give values on both sides of an if-then else test

23. Who Should Test? • In your case, you (the developer) will test • But the developer is often “too close” to the code to be objective and recognize some subtle faults • Difficult for developer to separate implementation structure and required function • Independent tester or test team is preferable • Familiar with how to “break” software • Familiar with testing methods and tools • Users should be involved throughout the process • Might catch missing requirements at early stages

24. Automated Testing Tools • Programs that will help test your code • Static Analysis, e.g. lint • Code Analysis • Check syntax, of a construct is fault-prone, etc. • Structure Checker • Tool to depict logic flow, check for structural flaws • Data analyzer • Review data structures, illegal data usage, improper linkage • Sequence checker • Highlight events in the wrong sequence • Dynamic Analysis, e.g. VTune • Monitor and collect statistics as program is running • How many times a statement was executed, branches covered, memory use, etc. • Useful for performance analysis to find the hotspots of the code

25. How Much Testing? • How do we know when to stop? • When all tests pass? • When ‘enough’ tests pass meeting some threshold • Myers (1979) • As the number of detected faults increases, the probability of the existence of more undetected faults increases • Suggests that if the number of detected faults is decreasing, we may not have too many left and can stop soon • Fault Seeding • Deliberately add faults to the process • Use their detection as a measure of the remaining faults

26. Fault Discovery Techniques • Jones, 1991

27. Code Inspections • Inspections can be applied to many different things by many different groups • Inspections are a “Best Known Method” (BKM) for increasing quality • Developed by Michael Fagan at IBM, paper published 1976 • Estimates: Inspections of design and code usually remove 50-90% of defects before testing • Very economical compared to testing • Formal inspections are more productive than informal reviews

28. Industry Experience • Widely used at many companies • HP, Microsoft, IBM, AT&T, Bell Labs, etc. all have an established track record with inspections • Aetna: Found 82% of errors in a program by inspections, was able to decrease development resources by 25% • AT&T : Reported 90% decrease in defects after introducing inspections • Finding defects early can save money and time • TRW : Changes in requirements stage cost 50-200 times less than changes during coding or maintenance • JPL : Estimates saved $25,000 per inspection by finding/fixing defects early on

29. Cost of Fixing a Defect Req Design Coding Test Accept Prod

30. Formal Inspections • By formalizing the process, inspections become systematic and repeatable • Each person in the inspection process must understand their role • Use of checklists focus concentration on detection of defects that have been problematic • Metrics • Feedback and data collection metrics are quantifiable • Feed into future inspections to improve them • Designers and developers learn to improve their work through inspection participation

31. More reasons to use inspections • Inspections are measurable • Ability to track progress • Reduces rework and debug time • Cannot guarantee that a deadline will be met but can give early warning of impending problems • Information sharing with other developers, testers

32. Definition • What is an inspection? • A formal review of a work product by peers. A standard process is followed with the purpose of detecting defects early in the development lifecycle. • Examples of work products • Code, Specs, Web Pages • Presentations, Guides, Requirements, • Specifications, Documentation

33. When are inspections used? • Possible anytime code or documents are complete • Requirements: Inspect specs, plans, schedules • Design: Inspect architecture, design doc • Implementation: Inspect technical code • Test: Inspect test procedure, test report

34. A defect is a deviation from specific or expected behavior Something wrong Missing information Common error Standards violation Ambiguity Inconsistency Perception error Design error Defects • Inspections are used to find defects

35. A defect is a defect • A defect is based on the opinion of the person doing the review • This means that any defect that is found IS a defect • Not open to debate • Not all defects are necessarily bugs • Many defects may not be “fixed” in the end • No voting or consensus process on what is a defect • How to fix a defect should be debated later, not when the defects are logged

36. Other Review Methods

37. Other Defect Detection Methods

38. Why a formal review? • Provides a well-defined process • Repeatability, measurement • Avoids some scenarios with less formal processes • “My work is perfect” • Point is not to criticize the author • “I don’t have time” • Formal process proceeds only when all are prepared, have inspected code in advance • We won’t do inspections this semester but you may encounter them in the future if you work on a large software project

39. Walkthrough vs. Inspection

40. What should be inspected? • For existing code or documentation, select • The most critical piece to the program’s operation • Most used section • Most costly if defects were to exist • Most error-prone • Least well-known • Most frequently changed • For new code or documentation • 20% <= inspect <= 100%

41. Inspection Process

42. Typical Inspection Process Planning 45 mins Prep 15-120 mins Log Defects 60-120 mins Rework Follow-Up

43. Roles Moderator Inspectors Work Owner Scribe

44. Owner Planning • Owner decides what code/documents to review • Should include requirements • Common-errors list • One will also be provided by the moderator • Owner can include more specific common errors • Paper copy of code listing for everyone • See previous slides for what code to inspect • If possible, add line numbers so code is easier to reference • Up to owner’s discretion as to what/how much, but generally one does not continue more than two hours at a time

45. Preparation • Ideally, each inspector has materials to inspect in advance • Identify defects on their own to ensure independent thought • Note defects and questions • Complete a defect log • High/Medium/Low • Without this preparation, group review might find only 10% of defects that could otherwise be found (Fagan) • Rules of thumb • 2 hours for 10 full pages of text

46. Preparation – Our Exercise • Too many projects/would take too much time to perform detailed inspection for all our projects • Compromise – Hopefully enough to give you a flavor of inspections, still be useful • Everyone should prepare by becoming familiar with common defects • Owner will provide brief walkthrough • Inspectors will log defects in real-time as walkthrough is made

47. Common Defects • See web page links • C++ • Java • Similar issues apply to other languages

48. Walkthrough • Prior to walkthrough • Owner distributes selected code, requirements docs • Inspectors have prepared • Purpose • Provide context, explanation for selected code to inspectors • Relate code back to requirements • Process • Owner provides walkthrough for one “chunk” of code that logically belongs together (e.g., a method) • Inspectors search for defects • Repeat for next chunk

49. Walkthrough Example • Requirement: Support authentication based upon user@host using regular expressions Open file Containing operators 1 /********************************************************* 2 * Returns a 1 if the user is on the ops list, and 3 * returns a 0 if the user is not on the ops list. 4 *********************************************************/ 5 int Authorized(char *user) 6 { 7 FILE *f; 8 9 f=fopen(OPSPATH,"r"); /* open authorized file */ 10 while (fgets(tempstr,80,f)!=NULL) 11 { 12 tempstr[strlen(tempstr)-1]='\0'; /* annoying \r at end */ 13 if (!fnmatch(tempstr,user,FNM_CASEFOLD)) { fclose(f); return(1); } 14 } 15 fclose(f); 16 return(0); 17 } Returns true if wildcards match

50. Defect Logging • Performed by the scribe; leaves work owner free to concentrate on other tasks • Moderator leads meeting and facilitates process • Keeps discussions to a minimum • Defect understanding only • No criticisms • No “rat holes” • Limited discussion • Moderator has the right to stop discussion at any time • May use round-robin for each inspector • Focus is on finding defects • A defect is a defect