Structural Testing

Structural Testing • Structural testing measures • Flowcharts • Structural testing and flowcharts • Minimal test suites

Program Flowcharts main () { int x, pos, even; printf("Enter an integer:"); scanf("%d", &x); pos = 0; even = 0; if (x > 0) {pos = 1;} if (x%2 == 0) {even = 1;} printf ("%d, %d, %d\n", x, pos, even); return 0; } • Example of code: • Intended to set pos = 1 iff x is positive, even = 1 iff x is even • Draw a graph of the statements and decisions in the code: • Statements = boxes • Decisions = diamonds • This graph called a “flowchart”

Pos=1 Pos=0 Even=0 Flowchart Example Printf() T X>0 F Scanf() T X%2=0 Even = 1 F Printf() End

Flowchart Example • A flowchart is really a labeled directed graph • Boxes and diamonds = nodes • Arrows = edges • Edges indicate parts of “paths” which may be followed through code • Labels on edges going out of diamonds indicate what happens when decision in diamond is true/false

Flowcharts: Example 2 (Code) • Code: • Changes lower-case letters in s to uppercase i = 0; while (s[i] != 0) { if (s[i] >= ‘a’) && (s[i] <= ‘z’)) { s[i]=s[i] – 32; } i++; }

s[i]>=‘a’&&s[i]<=‘z’? s[i]!=0? i=0 i++ End s[i]=s[i]-32 Flowcharts: Example 2 (Flowchart) F T F T

Structural Testing and Flowcharts • What does this have to do with testing? • If test suite executes all statements, then test suite visits every box and diamond in the flowchart • We say the test suite covers every node • We can also say the test suite covers every statement • This does not mean that every arrow (edge) in the flowchart has been followed • Example: consider the first example flowchart • We run it on one test case: x=2 • All nodes visited • However, neither F branch has ever been taken • If every arrow in the flowchart has been taken by some case in the test suite: • We say the test suite covers every edge • We can also say the test suite covers every decision

Edge vs. Node Coverage • Edge cover is more thorough than node coverage • Catches more problems • Example: consider first example flowchart • Change statements pos =0; even = 0 in the first column to pos = 1; even = 1 • Now, program always sets pos and even to 1 • But we wanted pos to indicate whether x is positive, even to indicate whether x is even • If we just test with x = 2 • If we test with both x = 2 and x = -1: • We cover every edge • We also force a failure to happen, program reports wrong results for x = -1 • Thus, edge coverage can reveal failures where node coverage cannot.

Coverage Terminology • If a test suite executes 100% of statements in program, we say it achieves • 100% node coverage or 100% statement coverage • If a test suite follows 90% of edges in flowchart, we say it achieves 90% edge coverage (etc) • “Line Coverage”: what gcov measures • Any line containing code is measured • Considered covered if any code on line is executed • Not exactly the same thing as statement coverage • Can have more than one statement on a line • However, 100% statement coverage implies 100% line coverage • Edge coverage also called decision coverage • Tests each decision in an if, while, etc. both ways

Strength of Coverage Measures • 100% node coverage implies 100% line coverage • However, 100% line coverage does not imply 100% node coverage • 100% edge coverage implies 100% node coverage • If followed every edge, must have visited every node • However, as we saw, 100% node coverage does not necessarily imply 100% edge coverage • Thus we say: • Node (statement) coverage is strong than line coverage • Edge (decision) coverage is stronger than node coverage • We can draw a graph: • Arrow from A to B: B is a stronger coverage measure than A Edge (Decision) Node (Statement) Line

Minimal Test Suites • By throwing more and more test cases into a test suite, we may achieve higher coverage • However, longer test suites take longer to run • Hence, we often want to find some minimal test suite • Important to define what minimal means i.e. minimal test cases? Minimal number of input used? Minimum time to run? • Example: first flowchart example: • Test suite: x=2, x=1, x=-2 covers all edges • However, it has 3 test cases • Test suite: x=2, x=-1 covers al edges and has only 2 test cases • Cannot have a minimal test suite which covers all edges and has only 1 test case • Have to execute each decision both ways • Hence, x=2, x=-1 is a minimal test suite for 100% edge coverage • Minimal in the sense of needing the fewest test cases

Coverage and Test Suites: Another Example • Consider again the case-conversion code flowchart (Example 2) • Find “minimal” test suite which achieves: • 100 % node coverage • 100% edge coverage • Decide what minimalmeans in this context!

Case Conversion Flowchart: Coverage • Clearly we can achieve 100% node coverage with one-char test case • Save, “a” • This test case also follows both arrows from the top diamond • This is because: • First time through loop (i==0), T branch taken • Next time condition is evaluated (i==1), s[i] is the null (end of string) character • Hence, F branch is also taken • However, F branch of other diamond not taken • To cover all edges, we need test suite like • (A) s= “a”, s=“X”; or • (B) s = “aX”

Case Conversion Flowchart: Minimality • Which of (A) and (B) is “minimal” • Depends on how we define minimality • (A) has 2 test cases, but each has just 1 character • (B) has just 1 test case, but it has 2 characters • We should always define precisely what we mean by minimality for a given test suite • Here, makes more sense to accept (B) as minimal • Don’t’ have to run program as many times, incur overhead of starting up process again • Hence, for this problem, we should say something like: • Test suite X will be considered smaller than test suite Y if either: • X has few test cases than Y; or • X has the same number of test cases as Y, but the total number of characters in X is smaller • Saves us from having to accept s = “aaaaaaXXXXXX” as minimal

Stronger Coverage Measures • So far we have been writing compound decisions • (e.g. (c >=‘a’) && (c <=‘z’)) inside a single diamond • Decision: the whole thing in parentheses after the “if”, while”, etc • Consider the code:if ((c>=‘a’) && (c<=‘z’)) {..} • (c >=‘a’) && (c<=‘z’) is the decision • Conditions: the individual terms of the condition, connected by logical operators • (c>=‘a’) && (c<=‘z’) are the two conditions • If we separate each condition with the decision into a separate diamond: • We can get a better reflection of what the program does

s[i]>=‘a’ s[i]<=‘z’ s[i]>=‘a’ && s[i]<=‘z’ Splitting Up Diamonds: Flowcharts • Example: • Instead of writing we write • Now we have two new arrows • To cover all arrows, we have to cover the two ones as well F T F T F T

Short Circuit Condition Coverage • If we split up all diamonds and then cover all arrows: • We are achieving more thorough coverage than just edge (decision) coverage can give us • This is called short circuit (SC) condition coverage • Recall definition of short circuit evaluation • Evaluation of a Boolean expression is cut short as soon as it can be determined to be true or false • Applies to language with short circuit evaluation of decisions, e.g. C, C++, Java • Similar but not identical to another notion called just “condition coverage” • Condition coverage applies to languages without short circuit evaluation of decisions, e.g. Visual Basic • We will not study this coverage measure

Achieving SC Condition Coverage • Consider the case conversion example • Test case “aX” is sufficient to achieve 100% edge coverage • Whole decision (c>=‘a’) && (c<=‘z’) is tested both ways: true for ‘a’ and false for ‘X’ • Is not sufficient to achieve 100% SC condition coverage • Have not take the F arrow from the s[i] <= ‘z’ diamond • To take that arrow we need a character that is: • Greater than or equal to ‘a’ • Greater than ‘z’ • We need one such ASCII character • The test case “aX~” will achieve 100% SC condition coverage

Achieving SC Condition Coverage in General • In general, for a decision A && B to achieve SC condition coverage, we need to design test cases to make: • A true, B true • A true, B false • A false • We will not even be able to get to the evaluation of B unless A is true as in first two cases (due to SC evaluation) • Similarly, for a decision A || B to achieve SC condition coverage, we need to design test cases to make: • A false, B true • A false, B false • A true • We will not even be able to get to the evaluation of B unless A is false as in the first two cases

SC Condition Coverage vs. Edge Coverage • If we have achieved SC condition coverage: • We must have evaluated each condition of an “if”, “while”, etc. both ways • Therefore, we must have evaluated each decision both ways • Therefore, SC condition coverage is stronger than edge (decision) coverage • Is there anything stronger than SC condition coverage?

Path Coverage • “100% path coverage”: every possible path through the flowchart of the code has been executed by some test case • Path: sequence of nodes visited during an execution • Can contain repeated elements if execution has visited a node more than once • The strongest possible coverage measure • For code with no loops, this is achievable • For code with while loops: • Going around loop a different number of times means we have taken a different path • For some code, we can go around a loop any number of times • Therefore, for some code: • it is not possible to achieve 100% path coverage

Printf() T X>0 F Pos=1 Pos=0 Even=0 Scanf() T X%2=0 Even = 1 F Printf() End Path Coverage: Example 1 • Integer classification example again: • Four paths through the code are possible • X = 2 • X = 1 • X = -2 • X = -1 • Draw the paths for each of the above examples! • Therefore we need four test cases for 100% coverage

s[i]>=‘a’&&s[i]<=‘z’? s[i]!=0? i=0 i++ End s[i]=s[i]-32 Path Coverage: Example 2 • Case conversion example again • Every string of a different length will follow a different path F T F T

Path Coverage for Loops • Path coverage for many while loops is impossible • Most programs have while loops! • Most have at least one that can be followed an arbitrary number of times: • e.g. A top level loop where we read a command and execute it • Therefore, for most whole programs, we don’t even attempt 100% path coverage • Path coverage is still useful for small sections of code

Strength of Coverage Measures Path (strongest) Edge (Decision) Node (Statement) Line (weakest) Where arrow from A to B means “B is stronger than A”

Testing Loops • If we have loops, we can’t test all possible paths • However, just decision coverage may not spot usual kinds of errors • Often code is wrong because programmer didn’t consider what would happen when: • Loop decision is false right from the start • Loop decision is true once, then false • Sometimes there is a maximum number (max) of times loop can be executed • Example: loop may stop at end of an array, or when some condition is true • In these cases, code may be wrong because programmer didn’t consider what would happen when • Loop decision is true max times • Loop decision is true max-1 times

Loop Coverage • It is therefore useful to write test cases which: • Execute loop 0 times • Execute loop 1 time • Execute loop more than 1 time • Execute loop max times (if max exists) • Execute loop max -1 times (if max exists) • These tests are often referred to as loop coverage • However, there is no real formal definition of loop coverage as with node, edge etc. coverage

Loop Coverage: Exercise • Provide adequate loop coverage for the following: (code finds the rightmost dot in a string, used to find extension for filename) • i = strlen(fname) -1; • while (Ii >0 && fname[i] != ‘.’) {i--;} • To do “loop coverage”, list some of the possible test cases: • (0 times)  • (1 time)  • (more than 1 time)  • (max – 1 times)  • (max times)  Note: max her is (length of the string –1) • Note: the code above is almost certainly correct • However, problem may not be with the loop itself, but the code that comes after • For instance, maybe the code that comes afterward: • Expects a non-empty extension • Expects a non-empty filename before the extension • Can’t handle the case of no extension • Loop testing will help find those problems too

Structural Testing