Control flow & Data flow Testing

Control flow & Data flow Testing • Control flow Testing (CFT) addresses the test case coverage of “execution” or “control”paths. • Data flow Testing (DFT) addresses the test case coverage of “interactions among data items.”

Some Definitions for Control Flow Testing • Control flow Testing requires the conversion of • requirement’s work flow and/or functional flow(black-box testing) or • design and/or code (white-box testing) to a “control flow graph” • The following definitions apply to a control flow graph: • Node: each node represents some unit of processing • Initial/Final nodes: Initial node is where the sequence of processing starts and final node is where the sequence of processing ends • Link: each link (line between nodes x and y) represents the relation between node x and node y. • Directed link: a link showing the direction of flow with an arrow (e.g. a directed link from node x to node y implies node y is processed after the node x processing.) • Out-links/In-links: a directed link that originates from a node is an out-link ; a directed link that ends up in a node is an in-link • Decision node: A node associated with multiple out-links (e.g. binary predicate); also called a “branching” node • Junction node: A node associated with multiple in-links • Process node: A node that is neither Decision nor Junction node

Control Flow Graph Example entry G1 P1 p – process node d– decision node J– junction node d1 P2 P3 j1 d2 G2 Path : starts with the entry node, follows through a number of links and nodes, and ends up in an exit node; we are interested “logical” paths. Segment : a portion of a path Loop : A path or a segment contains a loop if some node or segment is revisited d3 P4 P5 P6 j2 j3 P7 exit

Linearly Independent Path • A path through the system is Linearly Independent** from other paths only if it includes some segment or edge that is not covered in the other path. S1 - The statements are represented by the rectangular and diamond blocks. - The segments between the blocks are the eedges labeled with numbered circles. 1 2 C1 Path1 : S1 – C1 – S3 Path2 : S1 – C1 – S2 – S3 OR Path1: edges (1,4) Path2: edges (1,2,3) need to run both paths? 4 S2 3 S3 - Path1 and Path2 are linearly independent because each includes some edge that is not included in the other. (note: not necessarily nodes) - Later --- Cyclomatic # gives you the number of linearly independent paths

Control Flow Testing (Path Testing) • The “Basic idea”behind Control Flow Testing is to “select paths” as test cases and come up with the inputs (input values) to execute through those paths. It includes the following 4 steps: • Build the Control Flow Diagram (CFD) • Select the paths in the (CFD) to cover for the testing • Pick or develop the input values for testing the chosen paths and • Execute the test • Analyze the test results

(1) BuildControl Flow Diagram • Consider the problem of deciding how many “real” roots a quadratic equation of the form ax2 + bx + c =0 has: • b2 -4ac > 0 -> 2 roots • b2 -4ac = 0 -> one root 3. b2 -4ac < 0 -> imaginary root • Pseudo code may be: • Input (a, b, c); • d = b2 -4ac ; • If (d>0) then #root = 2 ; • else if (d =0) then #root =1; • else #root = 0; • output (#root); process steps 1 & 2 d>0 T F d=0 #root = 2 T F #root = 1 #root = 0 output (#root)

(2) Selecting Paths in CFD for CFT • Most of the time we would be interested in some % of path coverage. • For 100 % path coverage, we need to consider for two CFG’s: • sequential concatenation of CFG1 and CFG2 with m and n paths respectively will yield a total of (m x n) paths • nesting CFG2 in CFG1 will yield (m + n-1) paths process 1 & 2 CFG1 d>0 F T CFG2 d=0 #root = 2 T F #root = 0 #root = 1 output (#root)

(3)Picking input valuesin CFD for CFT • If we were only interested in the path that generated 2 roots , then we would need to input a, b, and c such that b2 - 4ac > 0 or the (d > 0) is truepath: • (a= 1; b=3; c =1) • For the path (d > 0) is false and (d = 0) is true path, b2 - 4ac = 0: • (a =1; b=2; c=1) • For the path (d > 0) is false and (d = 0) is false path, b2 - 4ac < 0: • (a=1, b=1, c=1) process 1 & 2 d>0 T F d=0 #root = 2 T F #root = 1 #root = 0 output (#root)

(3) Picking input valuesin CFD for CFT (cont.) p1 • If we want to cover 100% paths, then we need to cover (m x n) pathsfor sequential concatenation. • In this case, it is 2 x 2 = 4 paths of (TT, TF, FT, and FF). • Suppose: c1 is (x> 0) and c2 is (x < 100)then: 1. X= 5 => c1 is T & c2 is T 2. X = -1 => c1 is F and c2 is T 3. X = 105 => c1 is T and c2 is F 4. But it is impossible to get c1 and c2 to be F and F because c1 is false => (x ≤ 0) & c2 is false => (x ≥100). (x ≤ 0) ∩ (x ≥100) = θ. (p5 is “extraneous” function?) c1 T F p2 p3 c2 T F p4 p5 p6

# of Paths in CFT • Note that with (m x n) paths as the result of sequential concatenation of two graphs, g1 with m paths and g2 with n paths, there may be a lot of paths to test: e.g. (medium sized software program) concatenating: g1 with m = 8 paths (i.e. 8 “cases”) and g2 with n = 9 paths (i.e. 9 “cases”)will result in 72 potential paths. That is a lot of potential paths to test! • If we want to do a little less --- cover 75% of the paths ---- would that be ok? • Is there any way to do a little less and still feel “comfortable”?

Statement Coverage Testing in Path Testing p1 • In Statement Testing we are interested in the coverage of statements. • Note that we can cover all the statements with only 2 paths out of a total of 4 paths: • Path1: p1-c1-p2-c2-p5-p6 • Path2: p1-c1-p3-c2-p4-p6 • In general, we can cover all statementswith less than or equal to the total number of paths. c1 T F p2 p3 c2 T F p4 p5 p6

A Little “Less” Coverage with Statement Coverage • We are a “quality” conscious organization. While we can not test all the possible paths in our software, we cover 100% of the statements in our tests. Trust us! --- Every statement in our software is executed and tested before we release it!!

Branch Coverage Testing in Path Testing p1 • In Branch Coverage Testing we are interested in the coverage of all the “branches” coming out of the decision control. • Again Notethat we can cover all the Branches with only 2 paths out of a total of 4 paths: • Path1: p1-c1-p2-c2-p5-p6 • Path2: p1-c1-p3-c2-p4-p6 • In general, we can cover all branches with less than or equal to the total number of paths. c1 T F p2 p3 c2 T F p4 p5 p6

BothStatement Coverage & Branch Coverage • We are a “quality” conscious organization. While we can not test all the possible paths in our software, we cover 100% of the statementsand branches in our tests. Trust us! --- Every statement and branch in our software is executed and tested before we release it!!

McCabe’s Cyclomatic Complexity # • Recall McCabe’s complexity number used to measure structural complexity --- • Given a structure with n binary decisions, the cyclomatic complexity number is (n + 1) • It turn out the “maximum” number of paths you need to cover (a)all the statements and/or (b)all branches in the software structure with n binary decisions is n+1, which is the cyclomatic complexity number.

More on Linearly Independent Paths • In discussing dimensionality, we talks about orthogonal vectors (or “basis” of vector space) . • (e.g.) Two dimensional space has two orthogonal vector from which all the other vectors in two dimension can be obtained via “linearcombination” of these 2 vectors: • [1,0] • [0,1] e.g. [2,4] = 2[1,0] + 4[0,1] [2,4] [1,0] [0,1]

More on Linearly Independent Paths • A set of paths is considered to be a Linearly Independent Set if every path may be constructed as a “linear combination” of paths from the linearly independent set. For example: We already know: a) there are a total of 22=4 logical paths. b) 1 path (path4) will cover all statements c) 2 paths will cover all branches. d) 2 branches +1 = 3linearly independent paths. C1 2 1 S1 1 2 3 4 5 6 3 We pick: path1, path2 and path3 as the Linearly Independent Set 1 path1 1 1 C2 5 path2 1 1 4 path3 1 1 1 S2 path4 1 1 1 1 6 path 4 = path3 + path1 – path2 = (0,1,1,1,0,0)+(1,0,0,0,1,1)- (1,0,0,1,0,0) = (1,1,1,1,1,1) - (1,0,0,1,0,0) = (0,1,1,0,1,1)

Other Sets of Linearly Independent Paths 1 2 3 4 5 6 1 path1 1 1 path2 1 1 path3 1 1 1 path4 1 1 1 1 • Consider the set of linearly independent paths: 1, 2 & 4 instead. • Can we get Path3 = (0,1,1,1,0,0)? • Consider path3 = path2 + path4 – path1 • = (1,0,0,1,0,0,)+(0,1,1,0,1,1) – (1,0,0,0,1,1) = (0,1,1,1,0,0) • Consider another set of linearly independent paths: 2, 3 & 4 instead. • Can we get Path 1 = (1,0,0,0,1,1))? • Consider path1 = path2 + path4 – path3 • = (1,0,0,1,0,0,)+(0,1,1,0,1,1) – (0,1,1,1,0,0) = (1,0,0,0,1,1)

More on Linearly Independent Paths We already know: a) there are a total of 22=4 logical paths. b) 1 path (path4) will cover all statements c) 2 paths will cover all branches. d) 2 branches +1 = 3 linearly independent paths. C1 2 1 1 2 3 4 5 6 S1 1 path1 1 1 3 path2 1 1 C2 5 path3 1 1 1 4 path4 1 1 1 1 S2 6 Note : Although path1 and path3 are linearly independent, they do NOT form a Linearly Independent Set because no linear combination of path1 and path3 can get , say, path4.

How many “combinations” are there ? • For this example of: • 4 total paths • 3 linearly independent paths • How many combinations of linearly independent paths are there? • From combinatorics we know that if we want to pick 3 items at a time from 4 items, there are 4!/[3!*(4-3)!] = 4 combinatorics. • We tried 3 of the 4 potential linearly independent paths combinations. You try the fourth combination which is ---?

An Algorithm to Find the “Basis” Set • Select a baseline path, an arbitrary, normal execution path that contains as many decisions as possible. • Retrace the baseline path and “flip” each of the decision encountered; each flip creates a new path. Continue until all the decisions are flipped • The basis set is composed of all the paths generated from steps 1 and 2 above Cyclomatic # = 2+ 1 = 3; So there are 3 linearly independent paths C1 2 1 S1 1. pick baseline path P1: C1 –S1- C2 – S2: <0,1,1,0,1,1> 2. flip C1 P2 : C1 – C2 – S2 : <1,0,0,0.1,1> 3. flip C2 P3: C1 - C2 : <1,0,0,1,0,0> 3 C2 5 4 Can we get the 4th path : C1 – S1 – C2 : <0,1,1,1,0,0> from the above basis set? How about : (P1 + P3) – P2 ? (P1 + P3) – P2 = (<0,1,1,0,1,1> + <1,0,0,1,0,0>) - <1,0,0,0,1,1> = <1,1,1,1,1,1> - <1,0,0,0,1,1> = <0,1,1,1,0,0> = P4 <------ the 4th path S2 6

CFT Testing with “loop” Structure p1 • CFT testing with a loop structure has 2 main components: • A body of statements (P2) that is executed repeatedly • A control statement (c1) that determine whether if the body of statements should be executed and thus serve as entrance and exit criteria. c1 p3 p2

Loop Structure Testing • For each loop, we would want to test the following 7 situations: • Bypass the loop (never enter the loop) • Enter the loop once • Enter the loop twice • Enter the loop some “typical” number • Enter the loop (max-1) times • Enter the loop (max) times • Enter the loop (max + 1) times <----- may not be able to do We can shrink the test cases to 5 if we skip cases 3 and 5 above.

CFT Path Testing with Loops • Number of paths in CFT testing was a concern as the program logic increased. • (m x n) paths increased as either m and/or n increased in concatenation • (m + n -1) paths increased as either m and/or n increased in nesting • If we include the “loop” structure and count each execution through the loop as a path, then the number of paths can increase dramatically, especially with nested loops.

Nested “loop” Structure Example p1 • For nested loop, for each iteration of the outer loop, we have some iterations of the inner loop: • Inner loop may have m iteration • Outer loop may have n iterations • Thus there may be as much as [ mx (n2 +n)/2] possible execution paths! • This is much larger than (m x n) or (m + n -1) as m and n increases. c1 p3 Outer loop p2 p 21 C21 Inner loop p 22

“Extreme” Loop Structure Situation • For a nested loop structure, consider that the inner loop may have m iterations. The outer loop may have 0, ----, n iterations depending on the outer “control” statement. Thus we have : 0xm + 1xm------- + nxmpossible number of paths OR m x ∑ i = m x [(n2 + n )/2]possible paths i= n i=0 This is a HUGE number to deal with if the outer iterations, n, is large!

More “reasonable” loop testing • Clearly we can not test nested loop structures thinking about all possible paths! • If we use the 7 possible test cases for a loop structure as a guideline, then 2 nested loops would need 7 x 7 = 49 test cases. • In general, this would still be big for multiple nested loops: 7n for (n) nested loops.

4. Analyze Test resultsin CFD for CFT (cont.) • Analysis of test results for CFT is basically a case of analyzing the satisfaction of test “coverage” of control paths. • As the previous cases show, 100% of “possible paths” coverage may not be attainable ---- nor reasonable • Otherwise, analysis is similar in all types of testing.

Test Case Coverage • For us, test case coverage is a key issue in determining when to stop testing. We stop testing when our tests have covered all that wewant to cover. Ask: • Are there gapsand redundancies? • Have we covered all the relevant situations? We will use the Triangle Problem as an example to look at these questions

Previous Sample Triangle Psuedo-code 1. Program Triangle 2. Dim a, b,c As Integer 3. Dim IsTriangle As Boolean 4. Output ( “enter a,b, and c integers”) 5. Input (a,b,c) 6. Output (“side 1 is”, a) 7. Output (“side 2 is”, b) 8. Output (”side 3 is”, c) 9. If (a<b+c) AND (b<a+c) And (c<b+a) 10. then Is_Triangle = True 11. else Is_Triangle = False 12. endif 13. If Is_Triangle 14. then if (a=b) AND (b=c) 15. then Output (“equilateral”) 16. else if (a NE b) AND (a NE b) AND (b NE c) 17. then Output ( “Scalene”) 18. else Output (“Isosceles”) 19. endif 20. endif 21. else Output (“not a triangle”) 22. endif 23. end Triangle2

Condensation Graph from pseudo code Statements coverage - 4 paths Branch (DD-path) coverage - 4 paths Cyclomatic # = 4+1 = 5 - 5 lin. Ind paths All combinations - 8 paths first 1- 8 9 10 11 Is_Triangle= True Is_Triangle = False 12 Triangle ~Triangle 13 14 21 16 15 Not triangle equilateral 17 18 scalene isosceles 19 20 22 Last

All Combination paths ? • Let’s look at the all 8 combination paths • P1: < 8,9,10,12,13,14,15,20,22> (Equilateral) • P2: <8,9,10,12,13,14,16,17,19,20,22> (Scalene) • P3: <8,9,10,12,13,14,16,18,19,20,22> (Isosceles) • P4: <8,9,10,12,13,21,22> (not possible) • P5: <8,9,11,12,13,14,15,20,22> (not possible) • P6: <8,9,11,12,13,14,16,17,19,20,22> (not possible) • P7: < 8,9,11,12,13,14,16,18,19,20,22> (not possible) • P8: <8,9,11,12,13,21,22> (Not a triangle) - So, there are 4 decision-decision (dd) paths (branch testing) that make sense. - These are P1, P2, P3, and P8. - We should at least test these four paths.

Compare against Boundary Value Test(15 test cases for Triangle problem ) Remember the boundary: 1 ≤ TriangleSide ≤ 200 Test case a b c expected output paths 1 100 100 1 Isosceles P3 2 100 100 2 Isosceles P3 3 100 100 100 Equilateral P1 4 100 100 199 Isosceles P3 5 100 100 200 Not Triangle P8 6 100 1 100 Isosceles P3 7 100 2 100 Isosceles P3 8 100 100 100 Equilateral P1 9 100 199 100 Isosceles P3 10 100 200 100 Not Triangle P8 11 1 100 100 Isosceles P3 12 2 100 100 Isosceles P3 13 100 100 100 Equilateral P1 14 199 100 100 Isosceles P3 15 200 100 100 Not Triangle P8 Let’s analyze this table in more detail --- next chart

Comparison Summary • Potential “Gap” exist in the Boundary ValueTest. When we look at the equivalence classes (or logic table) of the outputs, we see that Scalene triangle is not covered. • Path P2 is not covered with the 15 Boundary Value test cases! • There are, however, lots of “Duplications” • P3 is covered 9 times (Isosceles triangle) • P1 is covered 3 times (Equilateral) • P8 is covered 3 times (Not Triangle) Clearly, boundary value –functional testing is not enough here ; is it possible that it is also not as efficient in this case?

Control flow & Data flow Testing