Basic Concepts of Software Testing (2.1 – 2.3)

1. Basic Concepts of Software Testing(2.1 – 2.3) Course Software Testing & Verification 2013/14 WishnuPrasetya

2. Software error, how costly can it be ? • Some errors in the software of UK Inland Revenue is rumored to cause tax-credit over-payment of 3.45 billions USD.(Charette, Why Software Fails. IEEE Spectrum, 2005)

3. Testing in large projects By users/customers (or 3rd party hired to represent them) is usually organized in stages like this (also called V-model ) : Ch. 7 provides you more background on “practical aspect”, e.g. concrete work out of the V-model, outlines of “test plan”  read the chapter yourself! By developers

4. Do you test your programs ? • People do not do it, or do not do it enough. Why !? • Not enough money, not enough time, not enough... • At least, do it systematically • set a meaningful and quantifiable adequacy criterion

5. When have you tested enough? • In principle you don’t know. • Pragmatically, you will have to decide what “enough” is, for you/your stake holders to be confident to use the software. • This is called coverage criterion. The are many existing proposals for these; and there is nothing that prevents you from proposing your own. But importantly: You want to be able to articulate your testing criterion, and preferably with a countable expression.

6. Challenges in Testing • When have tested enough? • How to produce a test-set that would be good enough? • Can we automate the execution of the tests? • Can we automatically generate tests? • If a test fails, how to find the source of the failure? (debugging) • Can we automate debugging?

7. Terms used • A test-set (Def 1.18): is just a set of test-cases. • (Deviate from Def. 1.17) A test-case specifies a series of interactions with the target program and the input needed by those interactions. Some interactions may trigger responses from the program. The test-case also specify what kind of responses it expects from the program. • In the simple case: just one interaction foo(x), and one response, namely return value. • Test inputs : the inputs meant above. • Test oracle: specify the expectation on responses as meant above.

8. Now, how do AO formulate this?(I reformulate it a bit) • Def. 1.21. A coverage criterion is used to specify a set of test requirements. (the set is denoted by TR). • Def. 1.20. A test requirement specifies a single goal (e.g. a position in your program) that your test-set must cover. • Ex. CC : every line in P must be visited. TR: the set of all P’s lines.

9. “line coverage” is too imprecise? • Abstractly, a program is a set of branches and branching points. An execution flows through these branches, to form a path through the program. • Such an abstraction is called Control Flow Graph (CFG, Legard 1970), such as the one above. • AO gives you a set of more precise concept of coverage, based on graph. foo(x,y) { while (x>y) x=x-y ; if (x<y) return -1 else return x }

10. Graph terminologies... • (Sec 2.1) A graph is described by (N,N0,Nf,E). E is a subset of N×N. So: • directed, no labels, no multi-edges. • A pathp : [n0, n1, ... , nk ] • non-empty! • each consecutive pair is an edge in E • length of p = #edges in p = natural length of p - 1 • Test-path (Def. 2.31)

11. Graph terminologies • (OA forget to define this!) A path p is a subpath of q if it is not longer than q and: (i: i0 : (k : 0klengthp : pk = qi+k )) • [1,2] is a subpath of [0,1,2,3] • [1,2] is nota subpath of [1,0,2] • OA also use the term “qtoursp”

12. We use TR to specify how to tour • Consider a test-requirement that specify a path q to cover. A test-path p tours q if q is a subpath of p. • (C2.8) The Coverage Criterion is to tour a given set of paths as TR. • Example, TR = { [0,0], [2], [3] }which test-set is good enough to satisfy this? 0 1 2 3

13. OA’s graph coverage criteria • (C2.1) Node coverage: TR is the set of paths of length 0 in G. • (C2.2) TR is the set of paths of length at most 1 in G. • So, what does this criterion tell? • Why “at most” ? • (Def 1.24) A coverage criterion C1subsumes (stronger than) C2 iff every test-set that satisfies C1 also satisfies C2.

14. 1 0 3 0 1 4 2 2 5 6 Does C2.2 subsume C2.1 ? • Obviously. How about the other way around? • Are we happy now!? How about C2.3 (Edge-pair coverage): TR contains all paths of length at most 2.

15. 0 3 1 4 8 2 5 7 Potential weakness: diamonds row 9 6 A row of k diamond has 2k different paths. However, suggesting that TR should include allpaths in G is not realistic either (think of loops!).

16. 0 1 2 AO propose prime paths coverage • A simple path p is a path where every node in p appears only once, except the first and the last which are allowed to be the same. • Such a path is a prime pathif it is not a proper subpath of another simple path. 3

17. 0 Prime Path Coverage (PPC) • (C2.4) TR is the set of all prime paths in G. • Strong in a diamond row (force you to cover all 2k paths). • Still finite. TR = { 012, 010, 101 } Give a test-set that can tour this. 1 2 The TR may appear to suggest that you need at least two test-cases: one with no iteration, and one with at least one iteration, but this is not the case!

18. 0 0 0 1 1 1 2 2 2 Diamond Loop 3 3 Prime paths : { 0130 , 0230 , 1301, 2302, 3013, 3023, 1302, 2301 } 3 The loop behaves like a diamond row of length 2, PPC will (also) force you to cover all four paths in this row, effectively forcing you to also iterate 2x. Perhaps a bit surprisingly, a single test-case (if it is feasible) can tour all the prime paths above.

19. Calculating PPC’s TR • How long can a prime path be? • It follows that they can be systematically calculated, e.g. : • “invariant” : maintain S to be a set of simple and still extensible (at the end-side) paths, and T of non-extensible (at the end-side) simple paths • “extend” S by extending the paths in it; we move non-extensibles to T  keep the invariant maintained. • This will terminate!  T contains non-extensible simple paths.

20. 0 3 1 4 2 5 6 Example [0, 1, 2] [0, 2, 3] [0, 2, 4] [1, 2, 3] [1, 2, 4] [2, 3, 6] ! [2, 4, 6] ! [2, 4, 5] ! [4, 5, 4] * [5, 4, 6] ! [5, 4, 5] * [0, 1] [0, 2] [1, 2] [2, 3] [2, 4] [3, 6] ! [4, 6] ! [4, 5] [5, 4] S : [0] [1] [2] [3] [4] [5] [6] ! We mark simple paths that cannot be extended. We also mark cycles. !  cannot be extended *  cycle

21. 0 3 1 4 2 5 6 Example [0, 1, 2] [0, 2, 3] [0, 2, 4] [1, 2, 3] [1, 2, 4] [2, 3, 6] ! [2, 4, 6] ! [2, 4, 5] ! [4, 5, 4] * [5, 4, 6] ! [5, 4, 5] * [0, 1, 2, 3] [0, 1, 2, 4] [0, 2, 3, 6] ! [0, 2, 4, 6] ! [0, 2, 4, 5] ! [1, 2, 3, 6] ! [1, 2, 4, 5] ! [1, 2, 4, 6] ! [0, 1, 2, 3, 6] ! [0, 1, 2, 4, 6] ! [0, 1, 2, 4, 5] ! S is now empty; terminate. • Prime paths = • all the (*) paths, plus • the non-cyclic simple paths (!) which are not a subpath of another (!)

22. Discussion • PPC is strong, but: • a series of if-then leads to exponential number of goal paths • the same with loops with if-thens in the body; though it does not necessarily mean we need equal number of test-cases • Problematic in all our CRs: if a node/edge/path turns out to be “unfeasible”. In principle this is a limitation we have to accept, however in loops this may happen more structurally.

23. 3 1 4 2 Typical unfeasibility in loops • Some loops always iterate at least once. E.g. above if a is never empty  prime path 124 is unfeasible. • (Def 2.37) A test-path p tours q with sidetripsif all edges in q appear in p, and they appear in the same order. i = a.length-1 ; while (i0) {if (a[i]==0) break ; i-- } • does not tour 124, but with sidetrips it does.

24. Generalizing Graph-based Coverage Criterion (CC) • Obviously, sidetrip is weaker that strict touring. • Every CC we have so far (C2.1, 2.2, 2.3, 2.4, 2.7) can be thought to be parameterized by the used concept of “tour”. We can opt to use a weaker “tour”, suggesting this strategy: • First try to meet your CC with the strongest concept of tour. • If you still have some paths uncovered which you suspect to be unfeasible, try again with a weaker touring.

25. 3 1 4 2 Oh, another scenario... • Often, loops always break in the middle. E.g. above if a always contain a 0. Problem: e.g. 1234 does not tour 124, not even with sidetrip! • (Def 2.38) A test-path p tours q with detours if all nodes in q appear in q, and they appear in the same order. • Weaker than sidetrip.

26. Pre-calculating the needed test-paths • Given a set of paths as your TR (e.g. set of all prime paths), can we pre-calculate a set T of test-paths that would tour TR ? • Let q = [n0,...,nk] be the target path to tour: • calculate a path p from N0 to n0. • calculate a path r from nkto Nf. • The path front(p)++q++tail(r) will tour q. • Path from a to b can be calculated using e.g. depth-first algorithm, breath-first algorithm, shortest-path algorithm.

27. Breath/depth first algorithm, data structures • queue : a list with these operations • isEmpty • add(x), adding x at the endof the list (after its last element) • retrieve() : retrieving the first element of the list. • stack: a list with the above operators • but add(x) will now add x at the front of the list. • add is also called push, and retrieve is called pop. • We will also maintain a set of already visited nodes.

28. 0 3 1 4 2 5 6 Breath/Depth First Algorithm boolreachable(a,b) { visited = { } queue.add(a) while (queue is not empty) { x = queue.retrieve() if (x==b) return true visited.add(x) for each y in { y | (x,y) in E && y not in visited } queue.add(S) } return false } If you replace queue with stack, you get a depth first algorithm.

29. 0 3 1 4 2 5 6 Extending the BFS • Extend the breath-first algorithm (BFS) so that it also construct a tree from a to all nodes reachable from a. Once you have this tree, calculating a path from a to b is easy. • Data structure: tree, with these operations: • root(a) constructs a tree with a single node a. • add(<x,y>) adds an edge <x,y> into the tree (x must already be in the tree, and y is not already in the tree)

30. 0 3 1 4 2 5 6 Overlying it with tree-construction reachable(a,b) { visited = { } ; tree = root(a) queue.add(a) while (q is not empty) { x = queue.retrieve() if (x==a) return tree visited.add(x) for each y in { y | (x,y) in E && y not in visited } queue.add(S) tree.add(<x,y>) } return empty }

31. Reconstructing the path(imperative alg) path(tree,b) { if tree is empty then return Nothing ; x = tree.root ; if x == b then return Just [x] ; for each child u of x in tree do case path(u,b) of Nothing  continue Just s  return Just (x:s) return Nothing } Because the tree was obtained from BFS, the reconstructed path will be the shortest one.

32. Mapping your program to its CFGmany ways, depending on purpose P1(x) { if (x==0) { x++ ; return 0 } else return x } • See Sec. 2.3.1. AO use left; I usually use middle. • (Sec 2.3.1) Define a block a maximum sequence of “instructions” so that if one instruction is executed, all the others are always executed as well. Implies that a block • a block does not contain a jump or branching instruction, except if it is the last one • no instruction except the first one, can be the target of a jump/branching instruction in the program. (x==0) dummy (x==0) x==0 x!=0 x++ return x x++ ; return 0 x++ ; return 0 return x return x return 0

33. Mapping your program to its CFG P2(a) { for (inti=0; i<a.length ; i++) { if (a[i]==0) break ; if (odd a[i]) continue ; a[i] = 0 } } 0 1 2 5 3 4 Add “end-of-program” as a virtual node.

34. Discussion: exception P3(...) { try { File f = openFile(“input”) x = f.readInt() y = (Double) 1/x } catch (Exception e) { y = -1 } return y } 0 1 1 Every instruction can potentially throw an exception. Yet representing each instruction as its own node in our CFG will blow up the size of the graph, and thus also the size of the TR to tour. A minimalistic approach is to ignore the fact different instructions in the same block may throw different exceptions, and thus may be responsible for their own kind of error. Ignoring this, we can pretend that the jump to an exception handler always happen at the end of the block, so giving us the above CFG. But you need to be aware of the above ignorance.

35. Discussion: exception P3(...) { try { File f = openFile(“input”) x = f.readInt() y = (Double) 1/x } catch (IOException e) { y = -1 }catch (Exception e) { y = -1 } return y } 1a 0 1b 1 Had we programmed as above, we can now at least distinguish between an execution that throws an IOException and other kind of exceptions (because they are represented by different nodes in the CFG), but of course we are still blind to the distinction between e.g. an execution that throws a ParseException and that the throws DivisionByZeroException.

36. More discussion • What to do with recursion? • What to do with dynamic binding ? f(x) {if(x0) return a else f(....) } You don’t know ahead which “add” will be called; it depends on the run-time type of c. E.g. it may refuse to add certain type of students. Graph-based coverage is not strong enough to express this aspect. register(Course c, Student s) { if (! c.full()) c.add(s) ;}