Software Testing and Reliability

1. Software Testingand Reliability Southern Methodist University CSE 7314

2. Functional and structural approaches to testing

3. A few words on combinatorics • Based on the Cartesian product of sets, we can count the number of possible inputs that a program has, i.e. | I |

4. Example • Assume a program has a single input, Customer ID (CID) • May be any value in the domain {00000-99999} • What is | I | ?

5. Example • Now assume we add a second input, the Order ID (OID) • This may be any value in the domain {00000-99999} as well • Now what is | I |?

6. Example • Finally, add a credit card number to the input • This is a 12 digit number • | I | has now reached 10**22 • If we can execute 1 million tests per second, it will take 1016 seconds, or about 300 million years!!

7. Example • Since we cannot know what data may exercise a given statement/path in general, we may attempt to resort to exhaustive testing • This attempt is doomed to fail due to the combinatorial explosion

8. Engineering the testing process • Any engineered product (and most other things) can be tested in one of two ways • Knowing the specified function that a product has been designed to perform, tests can be conducted that demonstrate each function is fully operational while at the same time searching for errors in each function

9. Engineering the testing process • Knowing the internal workings of a product, tests can be conducted to ensure that “all gears mesh” (internal operations performed according to specifications)

10. Structural testing • Uses knowledge of the internal workings • Also known as Clear box/glass box • Code based • Can be useful for finding interesting inputs • Misses an entire class of faults, missing code

11. Behavioral • Uses knowledge of the specific function that is to be performed • Based solely on the specification without regard for the internals • Also known as Black box • More user oriented • Misses an entire class of faults, extra code (surprises) except by accident

12. Passing criteria • How do we know when • 1. a single test has passed • 2. when we are done testing

13. Passing criteria • A single test passes when its output is correct • This requires a specific definition of correct and ties into the automated oracle problem

14. When are we done? • Conway Criteria: • No syntactic errors (it compiles) • No compile errors or immediate execution failures • There exists Some set of data for which the program gives the correct output • A typical set of data produces the correct output

15. When are we done? • Difficult sets of data produce the correct output. • All possible data sets in the problem specification produce the correct output • All possible data sets and likely erroneous input succeeds. • All inputs produce the correct output

16. Nature of software defects • Logic errors and incorrect assumptions are inversely proportional to the probability that a program path will be executed

17. Nature of software defects • We often believe that a logical path is not likely to be executed when, in fact, it may be executed on a regular basis • Typographical errors are random More of a case for WHITE box testing……

18. Overview of Coverage testing

19. Logical coverage • What does statement coverage imply ?

20. Example • 1 IF (( X > 1) AND ( Y == 1)) • 2 Z = Z / X • 3 END • 4 IF ((X == 2) OR ( Z >1)) • 5 Z = Z + 1 • 6 END What is required for statement coverage ?

21. Example • To achieve statement coverage, one can choose x = __, y = __ • Many possible errors are missed: • 1 IF ((X > 1) OR (Y ==1)) • 1 IF ((X >=1) AND (Y ==1)) • 1 IF ((X > 1) AND (Y >=1))

22. Blindness • Statement coverage is blind to several classes of error when choosing values to achieve coverage

23. Assignment blindness • Due to assignment of a particular value to a variable, the error does not propagate

24. Equality blindness • Due to an Equality check of a particular variable, the error does not propagate

25. Self blindness • Conditional itself covers up the error

26. Assignment Blindness example • 1 IF (<conditional>) • 2 X = 1 • 3 END • 4 IF (X + Y > 1) //Should have been (Y > 0) • 5 .... • 6 END

27. Assignment Blindness example • In this example, values are chosen to make the conditional true, the statement X=1 is executed and the error in line 4 is not seen

28. Assignment Blindness example • If every path forces the same assignment, then the 'error' doesn't really matter, (does it exist??) • For instance, if the conditional in statement 1 always evaluated to true

29. Equality Blindness Example • 1 IF (A == 2) • 2 ..... • 3 IF (B > 1) // Should have been (A + B > 3)

30. Equality Blindness Example • In this example, the value of 2 is chosen for A to force execution of the body • The error in statement 3 is missed

31. Self Blindness • 1 IF (X < 1) // Should have been (2X < 1) • In this example, the value of 0 is chosen for X

32. Observation • Statement coverage is the weakest form of coverage • Many classes of errors can escape this testing • Typical projects have less than 80% statement coverage!!

33. Branch coverage • Each branch of a decision must be executed at least once • This is stronger that statement coverage, but it is still weak • Note: A decision is a logical combination of one or more conditions

34. Branch coverage example • 1 IF (( X > 1) AND ( Y == 1)) • 2 Z = Z / X • 3 END • 4 IF ((X == 2) OR ( Z >1)) • 5 Z = Z + 1 • 6 END

35. Branch coverage example • To achieve branch coverage, one option is to choose • x = __, y = __ • x = __, y = __

36. Branch coverage example • Many possible errors are still missed: • 1 IF ((X > 1) OR ( Y ==1) • 1 IF ((X >=1) AND (Y ==1)) • 1 IF ((X > 1) AND (Y <=1))

37. Blindness in branch coverage • Branch coverage is blind to several classes of error when choosing values to achieve coverage • Compound decisions (more than one condition) are weakly tested • Boundaries of conditions (within a decision) are not explicitly tested

38. Condition coverage • Each condition within a decision must assume all possible values • This is 'potentially' stronger than branch coverage, but not always • It may, in fact, be weaker

39. Example of condition coverage • 1 IF (( X > 1) AND ( Y == 1)) • 2 Z = Z / X • 3 END • 4 IF ((X == 2) OR ( Z >1)) • 5 Z = Z + 1 • 6 END

40. Example of condition coverage • To achieve condition coverage for statement 1, one must choose values such that: • X > 1 ~(Y == 1) • ~(X > 1) Y == 1

41. Example of condition coverage • Both of these vectors miss the execution of statement 2 • A better choice may be: • ~(X > 1) ~(Y == 1) • X > 1 Y == 1

42. Multi-condition coverage • Every combination of possible condition values within a decision must be chosen • This is strictly stronger than branch or condition coverage • still has weaknesses!

43. Multi-condition coverage example • 1 IF (( X > 1) AND ( Y == 1)) • 2 Z = Z / X • 3 END • 4 IF ((X == 2) OR ( Z >1)) • 5 Z = Z + 1 • 6 END

44. Multi-condition coverage example • To achieve multi-condition coverage for statement 1, one must choose values such that: • 1. X > 1 Y == 1 • 2. X > 1 ~(Y == 1) • 3. ~(X > 1) Y == 1 • 4. ~(X > 1) ~(Y == 1)

45. Multi-condition coverage example • To achieve multi-condition coverage for statement 2, one must choose values such that: • 5. X == 2 Z > 1 • 6. X == 2 ~(Z > 1) • 7. ~(X == 2) Z > 1 • 8. ~(X == 2) ~(Z > 1)

46. Coverage

47. Multi-condition coverage example • There are 9 paths through this code and this test set only covers 4 of those

48. Multi-condition coverage example • Consider this error • Line 2 is incorrectly written as Z = Z –X • Only the second vector will execute this line but for the values chosen, Z-X = Z/X

49. Multi-condition coverage example • Also consider that no vector will execute both line 2 and line 5 • In a more complex example, it is easy to imagine variables being set in line 2 that are then incorrectly used in line 5

50. Multi-condition coverage example • For instance: • 2. Z = Z/X; A = 0 • .... • 5. Z = Z + 1; B = 1/A;