Whitebox Testing

Whitebox Testing CS 420 - Spring 2007

A Testing Life Cycle Fix Error Requirement Specs Fault Resolution Error Fault Fault Isolation Design Error Fault Coding Fault Classification Fault incident Testing

Terminology • Error • Represents mistakes made by people • Fault • Is result of error. May be categorized as • Fault of Commission – we enter something into representation that is incorrect • Fault of Omission – Designer can make error of omission, the resulting fault is that something is missing that should have been present in the representation

Cont… • Failure • Occurs when fault executes. • Incident • Behavior of fault. An incident is the symptom(s) associated with a failure that alerts user to the occurrence of a failure • Test case • Associated with program behavior. It carries set of input and list of expected output

Cont… • Verification • Process of determining whether output of one phase of development conforms to its previous phase. • Validation • Process of determining whether a fully developed system conforms to its SRS document

Verification versus Validation • Verification is concerned with phase containment of errors • Validation is concerned about the final product to be error free

Program Behaviors Specified (expected) Behavior Programmed (observed) Behavior Fault Of Commission Fault Of Omission Correct portion Relationship – program behaviors

Classification of Test • There are two levels of classification • One distinguishes at granularity level • Unit level • System level • Integration level • Other classification (mostly for unit level) is based on methodologies • Black box (Functional) Testing • White box (Structural) Testing

Specified (expected) Behavior Programmed (observed) Behavior 5 6 2 1 Test Cases (Verified behavior) 3 4 8 7 Relationship – Testing wrt Behavior Program Behaviors

Test methodologies • Functional (Black box) inspects specified behavior • Structural (White box) inspects programmed behavior

Programmed Specified Test Cases Functional Test cases

Programmed Specified Test Cases Structural Test cases

When to use what • Few set of guidelines available • A logical approach could be • Prepare functional test cases as part of specification. However they could be used only after unit and/or system is available. • Preparation of Structural test cases could be part of implementation/code phase. • Unit, Integration and System testing are performed in order.

Unit testing – essence • Applicable to modular design • Unit testing inspects individual modules • Locate error in smaller region • In an integrated system, it may not be easier to determine which module has caused fault • Reduces debugging efforts

Test cases and Test suites • Test case is a triplet [I, S, O] where • I is input data • S is state of system at which data will be input • O is the expected output • Test suite is set of all test cases • Test cases are not randomly selected. Instead even they need to be designed.

Need for designing test cases • Almost every non-trivial system has an extremely large input data domain thereby making exhaustive testing impractical • If randomly selected then test case may loose significance since it may expose an already detected error by some other test case

Design of test cases • Number of test cases do not determine the effectiveness • To detect error in following code if(x>y) max = x; else max = x; • {(x=3, y=2); (x=2, y=3)} will suffice • {(x=3, y=2); (x=4, y=3); (x=5, y = 1)} • Each test case should detect different errors

White-Box Testing • Statement coverage • Branch coverage • Path coverage • Condition coverage • Data flow-based testing

Statement Coverage • Statement coverage methodology: • design test cases so that every statement in a program is executed at least once. • The principal idea: • unless a statement is executed, we have no way of knowing if an error exists in that statement

Statement coverage criterion • Observing that a statement behaves properly for one input value: • no guarantee that it will behave correctly for all input values.

Example Euclid's GCD Algorithm • int f1(int x, int y){ • while (x != y){ • if (x>y) then • x=x-y; • else y=y-x; • } • return x; }

Euclid's GCD computation algorithm • By choosing the test set {(x=3,y=3),(x=4,y=3), (x=3,y=4)} • all statements are executed at least once.

Branch Coverage • Test cases are designed such that: • different branch conditions is given true and false values in turn. • Branch testing guarantees statement coverage: • a stronger testing compared to the statement coverage-based testing.

Example • Test cases for branch coverage can be: {(x=3,y=3), (x=4,y=3), (x=3,y=4)}

Condition Coverage • Test cases are designed such that: • each component of a composite conditional expression given both true and false values. • Example • Consider the conditional expression ((c1.and.c2).or.c3): • Each of c1, c2, and c3 are exercised at least once i.e. given true and false values.

Branch testing • Branch testing is the simplest condition testing strategy • compound conditions appearing in different branch statements are given true and false values. (note: the entire condition is given true and false values, not ALL possible sub expressions)

Branch testing • Condition testing • stronger testing than branch testing: • Branch testing • stronger than statement coverage testing.

Condition coverage • Consider a Boolean expression having n components: • for condition coverage we require 2n test cases. • practical only if n (the number of component conditions) is small.

Path Coverage • Design test cases such that: • all linearly independent paths in the program are executed at least once. • Defined in terms of • control flow graph (CFG) of a program.

Control flow graph (CFG) • A control flow graph (CFG) describes: • the sequence in which different instructions of a program get executed. • the way control flows through the program.

How to draw Control flow graph? • Number all the statements of a program. • Numbered statements: • represent nodes of the control flow graph. • An edge from one node to another node exists: • if execution of the statement representing the first node can result in transfer of control to the other node.

Example int f1(int x,int y){ • while (x != y){ • if (x>y) then • x=x-y; • else y=y-x; • } • return x; }

Example Control Flow Graph 1 2 3 4 5 6

Path • A path through a program: • A node and edge sequence from the starting node to a terminal node of the control flow graph. • There may be several terminal nodes for program.

Independent path • Any path through the program: • introducing at least one new node or one new edge that is not included in any other independent paths. • It may be straight forward to identify linearly independent paths of simple programs. However For complicated programs it is not so easy to determine the number of independent paths.

McCabe's cyclomatic metric • An upper bound: • for the number of linearly independent paths of a program • Provides a practical way of determining: • the maximum number of linearly independent paths in a program.

McCabe's cyclomatic metric • Given a control flow graph G,cyclomatic complexity V(G): • V(G)= E-N+2 • N is the number of nodes in G • E is the number of edges in G

Example • Cyclomatic complexity = 7 – 6 + 2 = 3.

Cyclomatic complexity • The cyclomatic complexity of a program provides: • a lower bound on the number of test cases to be designed to get coverage of all linearly independent paths. • only gives an indication of the minimum number of test cases required.

Path Testing - Test Cases • Draw control flow graph. (Loops can cause an explosion in the number of independent paths, so use looping values of 0 and 1 to test loops). • Determine V(G). • Determine the set of linearly independent paths. • Prepare test cases: • to force execution along each path

Example Control Flow Graph 1 2 3 4 5 6

Derivation of Test Cases • Number of independent paths: 4 • 1, 6 test case (x=1, y=1) • 1, 2, 3, 5, 1, 6 test case(x=1, y=2) • 1, 2, 4, 5, 1, 6 test case(x=2, y=1)

An interesting application of cyclomatic complexity • Relationship exists between: • McCabe's metric • the number of errors existing in the code, • the time required to find and correct the errors.

Cyclomatic complexity • Cyclomatic complexity of a program: • also indicates the psychological complexity of a program. • difficulty level of understanding the program.

Cyclomatic complexity • From maintenance perspective, • limit cyclomatic complexity • of modules to some reasonable value. • Some software development organizations: • restrict cyclomatic complexity of functions to some max number.

Data-flow-based Testing • Basic idea: test the connections between variable definitions (“write”) and variable uses (“read”) • Starting point: variation of the control flow graph • Each node represents a single statement, not a chain of statements • Set DEF(n) contains variables that are defined at node n (i.e., they are written) • Set USE(n): variables that are read

1 2 3 4 5 6 7 8 9 10 Example Assume y is already initialized 1 s:= 0;2 x:= 0;3 while (x<y) {4 x:=x+3;5 y:=y+2;6 if (x+y<10)7 s:=s+x+y; else8 s:=s+x-y; DEF(1) := {s}, USE(1) := DEF(2) := {x}, USE(2) :=DEF(3) := , USE(3) := {x,y}DEF(4) := {x}, USE(4) := {x}DEF(5) := {y}, USE(5) := {y}DEF(6) := , USE(6) := {x,y}DEF(7) := {s}, USE(7) := {s,x,y}DEF(8) := {s}, USE(8) := {s,x,y}DEF(9) := , USE(9) := DEF(10) := , USE(10) :=

1 2 3 4 5 6 7 8 9 10 Reaching Definitions A definition of variable x at node n1 reaches node n2 if and only if there is a path between n1 and n2 that does not contain a definition of x DEF(1) := {s}, USE(1) := DEF(2) := {x}, USE(2) :=DEF(3) := , USE(3) := {x,y}DEF(4) := {x}, USE(4) := {x}DEF(5) := {y}, USE(5) := {y}DEF(6) := , USE(6) := {x,y}DEF(7) := {s}, USE(7) := {s,x,y}DEF(8) := {s}, USE(8) := {s,x,y} Reaches nodes 2,3,4,5,6,7,8, but not 9 and 10.

Def-use Pairs • A def-use pair (DU) for variable x is a pair of nodes (n1,n2) such that • x is in DEF(n1) • The definition of x at n1 reaches n2 • x is in USE(n2) • In other words, the value that is assigned to x at n1 is used at n2 • Since the definition reaches n2, the value is not killed along some path n1...n2.

1 2 3 4 5 6 7 8 9 10 Examples of Def-Use Pairs Reaches nodes 2, 3, 4, 5, 6, 7, 8, but not 9,10 For this definition, two DU pairs:1-7, 1-8 DEF(1) := {s}, USE(1) := DEF(2) := {x}, USE(2) :=DEF(3) := , USE(3) := {x,y}DEF(4) := {x}, USE(4) := {x}DEF(5) := {y}, USE(5) := {y}DEF(6) := , USE(6) := {x,y}DEF(7) := {s}, USE(7) := {s,x,y}DEF(8) := {s}, USE(8) := {s,x,y}

Whitebox Testing