Testing Overview

1. Testing Overview References: Pressman, Software Engineering: a Practitioner’s Approach, McGraw Hill Pfleeger, Software Engineering, Theory and Practice, Prentice Hall J. McGregor and D. Sykes. A Practical Guide to Testing Object-Oriented Software, Addison-Wesley, 2001. I. Burnstein. Practical Software Testing, Springer-Verlag, 2003. 1209

2. Question • How do you know your software works correctly?

3. Question • How do you know your software works correctly? • Answer: Try it.

4. Question • How do you know your software works correctly? • Answer: Try it. • Example: I have a function of one integer input. I try f(6). It returns 35. • Is my program correct? Groups of 3 1 minute

5. Question • How do you know your software works correctly? • Answer: Try it. • Example: I have a function of one integer input. I try f(6). It returns 35. • My function is supposed to compute n*6-1. Is it correct? • Is my program correct? Groups of 3 1 minute

6. Goals of testing: I want to show that my program is correct: it generates the right answer for every input. Can we write tests to show this? Groups of 3 1 minute

7. Goals of testing: Can we prove a program is correct by testing? Yes, if we can test it exhaustively: every combination of inputs in every environment.

8. How long will it take? Consider X+Y for 32-bit integers. How many test cases are required? How long will it take? 1 test per second: 1,000 tests per second: 1,000,000 per second: Groups of 3

9. How Long? Consider X+Y for 32-bit integers. How many test cases are required? 232 * 232 = 264 =1019 (The universe is 4*10^17 seconds old) How long will it take? 1 test per second: 580,000,000,000 years 1,000 tests per second: 580,000,000 years 1,000,000 per second: 580,000 years

10. Example 2 A A loop returns to A We want to count the number of paths The maximum number of iterations of the loop is 20 How many? B C

11. Example 2 A Suppose the loop does not repeat: Only one pass executes 5 distinct paths B C

12. Example 2 A Suppose the loop repeats exactly once 5*5=25 distinct paths B If it repeats at most once, 5 + 5*5 C

13. Example 2 A What if it repeats exactly n times? B C

14. Example 2 A What if it repeats exactly n times? 5n paths B C

15. Example 2 A What if it repeats at most n times? ∑5n = 5n + 5n-1+ … + 5 N=20, ∑5n = 1015 32 years at 1,000,000 tests per second B C

16. Example 3 Consider testing the following function: The function fn shall take as input an integer value and return the integer part of the one number look ahead divided by 30,000. (The one number look ahead is simply the number one greater than the input.) How many test cases do you need to ensure this function works correctly? List several.

17. Example 3 The function fn shall take as input an integer value and return the integer part of the one number look ahead divided by 30,000. Consider testing the following code with an error. (The first line should be j = j + 1;) public static int fn (int j) { j = j - 1; j = j / 30000; return j; } Would any of your tests uncover this error?

18. Example 3 For 16-bit integers, errors at: -30,000 -29,000 29,000 30,000 4 cases out of 65,536 possible The function fn shall take as input an integer value and return the integer part of the one number look ahead divided by 30,000. Consider testing the following code with an error. (The first line should be j = j + 1;) public static int fn (int j) { j = j - 1; j = j / 30000; return j; } Would any of your tests uncover this error?

19. Example 4 • Consider testing a Java compiler? • How many inputs are needed to test every input?

20. Limits of testing: You can’t test it completely. You can’t test all valid inputs. You can’t test all invalid inputs. You really can’t test edited inputs. You can’t test in every environment. You can’t test all variations on timing. You can’t even test every path. (path, set of lines executed, start to finish)

21. Why Bother?

22. Goals of testing: Identify errors. Make errors repeatable (when do they occur) Localize errors (where are they) The purpose of testing is to find problems in programs so they can be fixed.

23. Costs of Testing • What costs are associated with testing software? Groups 2 min

24. In class discussion: • Read Software Error Costs • Be prepared to discuss main points Groups: 2 min

25. Cost • Testing accounts for between 30 and 90% of the total cost of software • Microsoft employs one tester for each developer • We want to reduce the cost • Increase test efficiency: #defects found/test • Reduce the number of tests • Find more defects • How? • Organize!

26. Levels of Software Testing • Unit/Component testing • Integration testing • System Testing • Acceptance Testing • Installation Testing

27. Levels of Software Testing • Unit/Component testing • Verify implementation of each software element • Trace each test to detailed design • Integration testing • System Testing • Acceptance Testing • Installation Testing

28. Levels of Software Testing • Unit/Component testing • Integration testing • Combine software units and test until the entire system has been integrated • Trace each test to high-level design • System Testing • Acceptance Testing • Installation Testing

29. Levels of Software Testing • Unit/Component testing • Integration testing • System Testing • Test integration of hardware and software • Ensure software as a complete entity complies with operational requirements • Trace test to system requirements • Acceptance Testing • Installation Testing

30. Levels of Software Testing • Unit/Component testing • Integration testing • System Testing • Acceptance Testing • Determine if test results satisfy acceptance criteria of project stakeholder • Trace each test to stakeholder requirements • Installation Testing

31. Levels of Software Testing • Unit/Component testing • Integration testing • System Testing • Acceptance Testing • Installation Testing • Perform testing with application installed on its target platform

32. Testing Phases: V-Model Requirements Specification System Specification System Design Detailed Design Acceptance Test Plan System Integration Test Plan Sub-system Integration Test Plan Unit code and Test Acceptance Test System Integration test Sub-system Integration test Service

33. Hierarchy of Testing Testing Ad hoc Program Testing System Testing Acceptance Testing Unit Testing Integration Testing Function Benchmark Properties Black Box Top Down Pilot Performance Equivalence Bottom Up Reliability Alpha Boundary Big Bang Availability Decision Table Beta Sandwich Security State Transition Usability Use Case Documentation Domain Analysis Portability White Box Capacity Control Flow Data Flow

34. Who Tests? • Professional testers: Organize and run tests • Analysts: involved in system requirements definition and specification • System designers: understand proposed solution and solution’s constraints • Implementers: Understand constraints associated with implementation • Configuration management representative: arranges for changes to be made consistently across all artifacts

35. A good test: • Has a reasonable probability of catching an error • Is not redundant • Is neither too simple nor complex • Reveals a problem • Is a failure if it doesn’t reveal a problem

36. Competent Programmer Hypothesis “We assume, as an article of faith, that the programmers are well trained, well supplied with the proper tools, and competent.”

37. Ad hoc Testing • Most popular approach • Sometimes called • “exploratory” • “unstructured” • “random” • “1401 testing”

38. Ad hoc Testing It is likely that there is some structure, but the structure is informal and based on experience, thus difficult to describe. • Most popular approach • Sometimes called • “exploratory” • “unstructured” • “random” • “1401 testing” Random testing is actually different: ad hoc is rarely random.

39. Ad Hoc TestingSimple example (Kaner): “The program is designed to add two numbers, which you enter. Each number should be one or two digits. The program will echo your entries, then print the sum. Press <Enter> after each number. To start the program, type ADDER.”

40. Simple example 2: Step 1. start with simple, obvious test. Type ADDER, 2, 3 and see what happens. There might be lots of things to note. Do things line up? Is there adequate advice? Step 2. Make notes about what else can be tested. After the obvious ones, start constructing a formal test series.

41. Simple example 3: Step 3. Look for boundary conditions: Cases: 99 + 99, -99 + -99, 99 + -14, -38 + 99, 56 + 99, 9+9, 0+0, 0+23, -78 + 0 Use each digit at least once. Test things likely to fail (boundaries) largest single digit, large second number, large first number if it passes 2+3, you expect it to pass 3+4. In some sense, they are the same.

42. Simple example 4: Step 4. Do some “on the fly” testing. Run with whatever feels right. Keep track of these tests. How about the invalid side of boundaries: 100+100, <> + <>, 1.2 + 5, control keys, etc. Step 5. summarize what you know about it. This is for your own use. Write a problem report for each bug.

43. Simple example 5: Step 6. Think of hidden boundaries. What about single byte signed numbers: -127-+127. Look at sums greater than 127 (which we already did, but look for strange results here.) Testing for valid or invalid characters: test “0”(48), “/”(47), “9”(57), “:”(58) and so on. Next cycle, review responses to problem reports and see what’s been done.

44. Automate testing • Do whenever possible. Design it for this. • “A century ago the steam locomotive reached it’s peak. Fifty years before that intercity stagecoaches pulled by teams of horses had an outrider on the first horse to stabilize the team. Manual testing to me is like having such a rider at the front of a speeding locomotive.” Beizer.

45. Advice • View testing as part of the development process • Buy a tool and use it • Testing is the last line of defense: Errors indicate there is a problem with the development process

46. Closing words “Testing is our last line of defense against bugs, not the first or only line of defense. When a bug is found by testing, it means that earlier phases of our software development process are wanting.” “I don’t see testing actually disappearing because the remaining bugs are always subtler and nastier.” Bezier