Software Testing

1. Software Testing Sudipto Ghosh CS 406 Fall 99 November 16, 1999

2. Learning Objectives • Data flow-based testing • Test adequacy assessment CS 406 Testing

3. Problems with path testing • With each predicate, there is a combinatorial explosion in the number of paths. • Potentially infinite number of paths because of loops. • Not all paths a feasible, i.e., there may not be inputs for which the path is executed. • Suppose that somehow, we satisfied this criterion. Would we find all the faults? CS 406 Testing

4. More problems • A path is tested only if it is present!! • Consider the example: double logBase19 (double x) { return ln(x)/ln(19); } • Missing condition is: if(x > 0) CS 406 Testing

5. More problems • It is possible to test every path without detecting the fault in the product. • Function to test the equality of 3 integers: • (Fallacious) assumption: • If the average of the 3 numbers is equal to the first, then they are equal. boolean areEqual(int x,int y,int z){ if ((x+y+z)/3 == x) return TRUE; else return FALSE; } • Exercise: Think of test cases. CS 406 Testing

6. Reducing the number of paths to be covered • Find something between branch and path coverage. • Select a set of paths that ensure branch coverage and try some other paths that help reveal errors. • Consider two paths same if they differ only in their sub-paths that are caused due to loops. • Restrict test cases to linear code sequences • Identify set of points L, from which control flow may jump; includes entry, exit, branch statements • Paths begin and end at elements in L. CS 406 Testing

7. Data Flow-Based Testing • Select paths to be executed based on data flow analysis • Information about where • variables are defined • variables are used • Idea is to make sure that the definitions of variables and their subsequent use is tested CS 406 Testing

8. Variables in statements • Variables occur in statements as • Definition – def • Variables on the left side of a statement ( c ) • c := 10; • Variable is used in a computation – c-use • Variables on the right-hand side of a statement ( c ) • temp := c + 5; • Variables used in a write statement • Variable is used in a predicate – p-use • if( c!= MAXLENGTH ) { CS 406 Testing

9. An example (xy) • scanf(x, y); if(y < 0) • pow = 0 – y; • else pow = y; • z = 1.0; • while(pow != 0) • { z = z * x; pow = pow – 1; } • if ( y < 0 ) • z = 1.0/z; • printf(z); CS 406 Testing

10. Def/use graph of example def = {x, y}: c-use = {} 1 y y def = {pow}: c-use = {y} 3 2 def = {pow}: c-use = {y} 4 def = {z}: c-use = {} 5 def = {}: c-use = {} pow pow def = {z, pow}: c-use = {x, z, pow} 6 7 def = {}: c-use = {} y y def = {z}: c-use = {z} 9 def = {}: c-use = {z} 8 CS 406 Testing

11. More on def/use graphs • With each node, we associate the c-uses of a variable • With each edge, we associate the p-uses of a variable CS 406 Testing

12. Def-clear paths • Def-clear path with respect to a variable x: • Path from node i to node j, such that there is no def of x in the nodes in the path from i to j. Nodes i and j may have a def. • Find out examples from the previous graph. • Path from node i to edge (j, k), such that there is no def of x in the nodes in the path. • Find out examples from the previous graph. CS 406 Testing

13. Global c-use • A c-use of a variable is considered global if: • there is not def of x within the block preceding the c-use • Example? CS 406 Testing

14. Global def • A def is global if it can be used outside the block in which it is defined. • A def of a variable x, in a node i, is a global def, if: • it is the last def of x in the block (node) i • a def-clear path exists from i to another node which has a global c-use of x • Example? CS 406 Testing

15. Set def(i) • Set of variables for which there is a global def in the node i. • Examples: CS 406 Testing

16. Set c-use(i) • Set of variables for which there is a global def in the node i. • Examples: CS 406 Testing

17. Set p-use(i, j) • For an edge (i, j), the set p-use(i, j), is the set of variables for which there is a p-use for the edge (i, j). • Examples: CS 406 Testing

18. Set dcu(x, i) • dcu(x, i) represents all those nodes in which the global c-use of x uses the value assigned by the def of x in i. • Suppose a variable x is in def(i) of node i. • Set of nodes, such that : • Each node has x in its c-use • There is a def-clear path from i to j • Examples: CS 406 Testing

19. Set dpu(x, i) • dpu(x, i) represents all those edges in which the p-use of x uses the value assigned by the def of x in i. • Suppose a variable x is in def(i) of node i. • Set of edges, such that : • each edge has x in its p-use • There is a def-clear path from i to (j, k) CS 406 Testing

20. Test case selection criteria • Let G be the def-use graph for a program • Let P be a set of all the complete paths of G that were executed during the test. • Examples: CS 406 Testing

21. All-defs criterion • Make sure that the definitions of all variables are tested. • For every def of every variable, one of its uses (either p-use or c-use) must be included in a path. • For every node i in G and every x in def(i), P includes a def-clear path w.r.t. x to some member of dcu(x, i) CS 406 Testing

22. All-p-uses criterion • All p-uses of all definitions should be tested. • For every x  def (i), P includes a def-clear path w.r.t. x from i to all members of dpu(x, i). • Note that a c-use may not be tested • Can require • all p-uses, some c-uses criterion CS 406 Testing

23. All c-uses criterion • All c-uses of all definitions should be tested. • For every x  def (i), P includes a def-clear path w.r.t. x from i to all members of dcu(x, i). • Note that a p-use may not be tested • Can require • all c-uses, some p-uses criterion CS 406 Testing

24. All-uses criterion • All p-uses and all c-uses of a definition must be exercised. • The set P must include, • for every node i and every x def(i) • a def-clear path w.r.t. x from i to all elements of dcu(x, i) and to all elements of dpu(x, i). CS 406 Testing

25. Number of test cases • Can be shown theoretically that the limit of the size of test set is up to quadratic in the number of two-way decision statements in the program. • Empirical studies have shown that the actual number of test cases that satisfy a criterion is quite small and increases linearly with the number of two-way decisions. CS 406 Testing

26. Inclusion relationship between criteria all-paths (path coverage) all-uses all-c-uses/ some-p-uses all-p-uses/ some-c-uses all-defs all-p-uses all-edges (branch coverage) all-nodes (statement coverage) CS 406 Testing

27. Implication of inclusion • Suppose C1 includes C2 (C1 C2), I.e test cases satisfying C1 also satisfy C2 • Does it mean that C1 is always better than C2? • Statistically, the set of test cases satisfying C1 will be better than those satisfying C2. • Testing done using these criteria performs better than randomly selected test cases. CS 406 Testing

28. Test assessment • Develop • a test set T • a collection of test inputs • Question: • How good is T? • Test assessment is the measurement of the goodness of T. • Test assessment is carried out based on one or more criteria. CS 406 Testing

29. Test adequacy assessment • These criteria are known as test adequacy criteria. • Test assessment is also known as test adequacy assessment. CS 406 Testing

30. Test adequacy assessment (contd) • Test adequacy assessment provides the following information: • A metric, also known as adequacy score, or coverage, usually between 0 and 1. • A list of all the weaknesses found in T, which when removed, will raise the score to 1. • The weaknesses depend on the criteria used for assessment. CS 406 Testing

31. Test adequacy assessment (contd) • Once the coverage has been computed, and the weaknesses identified, one can improve T. • Improvement of T is done by examining one or more weaknesses and constructing new test requirements designed to overcome the weaknesses. CS 406 Testing

32. Test adequacy assessment (contd) • This is continued until all weaknesses are overcome, i.e., the adequacy criterion is satisfied (coverage = 1) • In some instances, it may not be possible to satisfy the adequacy criteria for one or more of the following reasons: • Lack of sufficient resources (people, time) • Weaknesses that cannot be removed because they are infeasible • The cost of removing the weakness is not justified CS 406 Testing

33. Test adequacy assessment (contd) • While improving T by removing its weaknesses, one usually tests the program more thoroughly than it has been tested so far. • This additional testing is likely to result in the discovery of remaining errors. • Hence, we say that test assessment and improvement helps in the improvement of software reliability. CS 406 Testing

34. Test adequacy assessment - summary procedure 0. Develop T 1. Select an adequacy criterion C 2. Measure adequacy criterion of T w.r.t C 3. Is T adequate? Yes No Yes 4. Improve T 5. More testing is warranted? No 6. DONE!! CS 406 Testing