CSE 7314 Software Testing and Reliability Robert Oshana Trip 3 class notes

1. CSE 7314 Software Testing and Reliability Robert Oshana Trip 3 class notes oshana@airmail.net

2. Agenda • Administrative • Famous software failures • Overview of testing techniques • Black box testing techniques • Black box science • Black box art • White box testing techniques • Static white box techniques • Dynamic white box techniques

3. Infamous software failures • Disney’s “Lion King” • 1st multimedia CD-ROM • Just in time for Christmas – sales expectations very huge • CS phones began to ring • Failure to properly test SW on many different PC models • Worked well on the models that Disney programmers used to create the game !! “Software Testing” by Ron Patton

4. Intel bug • Try this on your PC: • (4195835 / 3145727) * 3145727 – 4195835 • If answer is 0, your computer is fine • Anything else => you have the Pentium bug !! “Software Testing” by Ron Patton

5. NASA Mars Polar Lander 1999 “Software Testing” by Ron Patton • Lander disappeared Dec 3rd 1999 • Failure to set a single data bit • NASA tried to save $ by replacing radar with contact switch • During testing vibration caused switch to trip (like it had landed) • Lander tested by multiple teams • Leg fold down procedure (never looked to see if touch down bit set) • Landing process (always reset computer, clearing bit, before starting testing

6. What goes into a SW product? “Software Testing” by Ron Patton

7. What parts make up a SW product? “Software Testing” by Ron Patton

8. Optimal test effort “Software Testing” by Ron Patton

9. Precision vs accuracy • Testing a simulation game like a flight simulator.. • Should you test precision or accuracy? • How about a calculator? • Depends on what the product is and what the development team is aimingat !!

10. Neither accurate nor precise “Software Testing” by Ron Patton

11. Precise, but not accurate “Software Testing” by Ron Patton

12. Accurate, but not precise “Software Testing” by Ron Patton

13. Accurate and precise “Software Testing” by Ron Patton

14. Try this • Start the Windows Calculator program • Type 5,000-5 (comma is important) • Look at result • Is it a bug or not??

15. Quiz • Q: Given that its impossible to test a program completely, what information do you think should be considered when declaring whether its time to stop testing? • A: There is no correct answer when to stop testing. Each project is different. Things to consider would be, Are lots of bugs being found?, Is the team satisfied with the number and types of tests that have been run?, Has the product been validated against the users requirements?

16. Quiz • Q: If you were testing a flight simulator, what would be more important, accuracy or precision? • A: Simulator should look and feel like flying a real plane. What’s more important is how accurately the simulator reflects reality, precision can follow. This is exactly what has happened with simulation games over the years.

17. Quiz • Q: Is it possible to have a high-quality and low reliability product? What might an example be? • A: Yes, but it depends on the customers expectations for quality. Example is a high performance sports car that has high quality (fast acceleration, style, etc) but are notoriously unreliable (often breaking down and expensive to repair)

18. Quiz • Q: Why is it impossible to test a program completely? • A: There are too many inputs, too many outputs, and too many path combinations to fully test. Also, software specs can be subjective and be interpreted in different ways (bug is in the eye of the beholder!!)

19. Quiz • Q: If you were testing a feature of your software on Monday and finding a new bug every hour, at what rate would you expect to find bugs on Tuesday? • A: Two axioms: The number of bugs remaining is proportional to the number of bugs already found (won’t come in Tuesday and find the SW perfect!). Pesticide paradox; continuing to run the same tests over and over again means you won’t find more bugs until you start adding more tests!! Result: you’ll continue to find bugs at the same rate or slightly less

20. Part 2

21. Black box vs White box

22. BB and WB testing • White box or black box testing improves quality by 40%. Together they improve quality by 60%

23. Black box science techniques • Equivalence partitioning • Data testing • Orthogonal arrays • Decision tables • Other techniques • Quiz • Black box art

24. Test to pass and test to fail • Test-to-pass; assume SW minimally works • Don’t push capabilities • Don’t try to break it • Simple and straightforward test cases • Would you drive a brand new car model at top speed right off the line? • Run these first

25. Test to pass vs Test to Fail “Software Testing” by Ron Patton

26. Equivalence partitioning

27. Equivalence partitioning • A group of tests forms an equivalence class if you believe that: • They all test the same thing • Of one catches a bug, the others probably will too • If one doesn’t catch a bug, the others probably won’t either

28. Example • How would you test the “Save as” function? • Valid characters except \ / : * ? “ < > and | • 1 to 255 characters

29. Example • Equivalence partitions • Valid characters • Invalid characters • Valid length names • Names that are too short • Names that are too long

30. Data testing

31. Data testing • Simple view of software is to divide into two domains • Data (or its domain) • Program • Examples of data • Keyboard input • Mouse clicks • Disk files • printouts Divide testing up along the same lines!!

32. Data testing • The amount of data can be overwhelming even for a simple program • Must intelligently reduce test cases by equivalence partitioning • Boundary conditions • Sub-boundary conditions • Nulls • Bad data

33. Boundary conditions

34. Boundary value analysis • Boundaries are often prone to failure • Does it make sense to also test in the middle? • Procedure • Test exact boundaries • Value immediately above upper boundary • Value immediately below lower boundary

35. Simple program with bug 1: Rem Create a 10 element integer array 2: Rem Initialize each element to –1 3: Dim data(10) As Integer 4: Dim i As Integer 5: For i = 1 To 10 6: data(i) = -1 7: Next i 8: End “Software Testing” by Ron Patton

36. Simple program with bug data(0) = 0 data(1) = -1 data(2) = -1 data(3) = -1 data(4) = -1 data(5) = -1 data(6) = -1 data(7) = -1 data(8) = -1 data(9) = -1 data(10) = -1 “Software Testing” by Ron Patton

37. Types of boundary conditions • Numeric • Character • Position • Quantity • Speed • Location • Size

38. First/last Start/finish Empty/full Slowest/fastest Largest/smallest Next to/farthest from Min/max Over/under Shortest/longest Soonest/latest Highest/lowest Characteristics of those types

39. First-1/Last+1 Start-1/Finish+1 Less then empty/More than full Even slower/Even faster Largest+1/Smallest-1 Min-1/Max+1 Just over/Just under Even shorter/Longer Even sooner/later Highest+1/Lowest-1 Testing the boundary conditions

40. Example • Test entry field that allows 1 to 255 • Try entering 1 character • Enter 255 characters • Try 254 • Enter 0 • Enter 256

41. Program that reads and writes a floppy • Try saving a file with one entry • Try saving a file at the limits of what a floppy disk holds • Try saving an empty file • Try saving a file that is too big for the floppy

42. Program to print multiple pages onto a single page • Try printing one page (the standard case) • Try printing the most pages allowed • Try printing zero pages • Try printing more than it allows

43. Flight simulator • Try flying at ground level • Try flying at maximum height • Try flying below ground level • Below sea level • Outer space

44. Sub-boundary conditions • Some boundary conditions are internal to the software and not user apparent • Powers of two • Bit (0-1), nibble (0-15), byte (0-255), word, kilo, mega, giga, etc • ASCII table • Not nice and contiguous

45. Default, empty, blank, null, zero, and none • Rather than type wrong information for example, no data is entered (just press return) • Forgotten in spec and overlooked by programmer • Happens in real life! • SW should default to lowest value or some reasonable value

46. Default, empty, blank, null, zero, and none • Always have an equivalence partition for these values

47. Invalid, wrong, incorrect, and garbage data • Test-to-fail • See if the SW can handle whatever a user can do • Reasonable to think that some percentage of people will handle SW incorrectly • If any data lost, for whatever reason, user will blame SW (bug!)

48. Invalid, wrong, incorrect, and garbage data • If SW wants numbers, give it letters • If SW expects positive numbers, give it negative numbers • Fat fingers • How many windows can you have open • No real rules, be creative, have fun!

49. State testing • A software state is a condition or mode that the SW is in • Other side of testing looks at programs logic flow • States • Transitions between them • Same type of complexity required partitioning this space as well

50. Create a state transition map “Software Testing” by Ron Patton