Verification and Validation (Part 2)

Verification and Validation(Part 2) John Morris Computer Science/Electrical and Computer Engineering Credit: Several slides in this presentation were adapted from originals written byStephen M. Thebaut, University of Florida A hard day’s work ensuring that some Japanese colleagues understand why Auckland is The City of Sails!

Validation or Verification? Validation Building the right software Make sure it’s what the user wants Verification Building the software right Make sure it works Accurate, complete specification essential!

Testing • Aim • Locate and repair defects • Axiom • No matter how much testing you do, you can’t be sure that there isn’t an error waiting to bite you! • Testing only reveals the presence of defects,it never proves their absence!!

Testing The alternative? • Formal verification • Uses formal logic to prove that software is correct • Currently: • Prohibitively expensive • Little automated support • Mainly manual techniques • Error prone • Only feasible when cost of failure is extreme • Usually when failure leads to loss of life • Air and space craft control • Medical systems • Nuclear plants

Testing - Motivation Definitely the least glamorous part of software development  • Possibly the most expensive! • If not carried out thoroughly! • Estimates of the economic cost of software failure produce astronomic numbers • US: $59.5 billion in 2002 • http://www.nist.gov/public_affairs/releases/n02-10.htm • ~10% of projects are abandoned entirely • Including some very large ones

Exhaustive Testing • Inefficient, naive? • Never forget the KISS principle • An automated test system can do a very large numbers of tests in a reasonable time • and do them while you’re designing the next test! • Analysis needed is trivial whereas • Analysis for a more efficient test regime may be quite complex and error-prone • It’s easy to convince someone that an exhaustively tested system is reliable

Equivalence classes – formal definition • A set of equivalence classes is a partition of a set such that • Each element of the set is a member of exactly one equivalence class • For a set, S, and a set of equivalence classes, Ci • U Ci = S • Ci  Cj =  (null set) unless I = j

Equivalence classes – formal definition • A set of equivalence classes is a partition of a set such that • The elements of an equivalence class, C, are classified by an equivalence relation, ~ • If a  C and b  C , then a ~ b • The equivalence relation is • Reflexive a ~ a • Transitive if a ~ b and b ~ c, then a ~ c • Symmetric if a ~ b then b ~ a • A Representative of each class is an arbitrary member of the class • They’re all ‘equivalent’ – so choose any one!

Equivalence classes – verification • Equivalence relation • In the verification context, the elements of the set are the sets of input values for a function under test eg we are verifying a function f( int a, int b ) • The 2-tuples (1,1), (1,2), (1,3) .. (and many more!)are the elements of the set of all possible inputs for f • The equivalence relation is “behaves the same way under testing” • One common interpretation of this is: • “follows the same path through the code”

Black Box and White Box Verification • Black Box • Specification is available but no code • White Box • Code is available and can be analyzed • Equivalence classes are derived from rules in the specification and the code

Generating the Equivalence Classes Outputs • Find equivalence classes that cover outputs also! • Same general rules apply as for inputs • One representative of each class plus • Boundaries • Null output eg No items in a report – does the user want a confirming report anyway? • Just one output eg Reports often have header and trailer sections - are these correctly generated for a short (<1 page) report? • Never neglect the null case! • It’s very easy to neglect at specification stage • Required behaviour may be ‘obvious’ • No need to write it down! • It will require coding • Experienced programmers know that it’s a very common source of error!

Coverage in White Box Testing • Black Box testing will not usually cover all the special cases required to test data structures • Often, the functional goals of the specification could be met by one of several data structures • Specification may deliberately not prescribe the data structure used • Allows developers to choose one meeting performance goals • Permits substitution of an alternative with better performance (vs non functional specifications) • Coverage defines the degree to which white box testing covers the code • Measurement of completeness of testing

Coverage in White Box Testing • Usually, at least some white box coverage goals will have been met by executing test cases designed using black-box strategies • How would you know if this were the case or not? • In simple modules, which don’t use internal data structures, black box classes may be adequate • This is not the general case though! • Various coverage criteria exist Every statement at least once Every branch taken in true and false directions Every path through the code

Coverage in White Box Testing • Coverage criteria • Logic coverage • Statement: each statement executed at least once • Branch: each branch traversed (and every entry point taken) at least once • Condition: each condition True at least once and False at least once • Branch/Condition: both Branch and Condition coverage • Compound Condition: all combinationsof condition values at every branch statement covered (and every entry point taken) • Path: all program paths traversed at least once

Pseudocode and Control Flow Graphs input(Y) if (Y<=0) then Y = −Y end while (Y>0) do input(X) Y = Y-1 end “nodes” “edges”

Statement Coverage • Statement Coverage requires that each statement is executed at least once • Simplest form of logic coverage • Also known as Node Coverage • What is the minimum number of test cases required to achieve statement coverage for the program segment given next?

Pseudocode and Control Flow Graphs input(Y) if (Y<=0) then Y = −Y end while (Y>0) do input(X) Y = Y-1 end “nodes” “edges”

Branch coverage • Branch Coverage requires that each branch will have been traversed, and that every program entry point will have been taken, at least once • Also known as Edge Coverage

Branch Coverage – Entry points • Why include “…and that every program entry point will have been taken, at least once.” • Not commonin HLLs (eg Java) now • Common in scripting languages • Any language that allows a goto and a statement label!

Branch Coverage (cont’d) • Relationship between Statement and Branch Coverage? Possibilities: • None. • Statement Coverage subsumes Branch Coverage (“statement => branch”). • Branch Coverage subsumes Statement Coverage (“branch => statement”).

“statement => branch” ? Minimum input sets required for Statement Coverage? Minimum input sets required for Branch Coverage?

“branch => statement” ???

“branch => statement” ??? • Normally, YES. • But not in the presence of Dead Code Dead Code is not reachable via any executable program path.

Condition Coverage • A branch predicate may have more than one condition • input(x,y); • if ((y<=0) || (x=0) ){ • y = -y; • } • while ( (y>0) && (! EOF) )do { • input(x); • y = y-1; • }

Condition Coverage • Condition Coverage requires that each condition will have been True at least once and False at least once • What is the relationship between Branch and Condition Coverage?

Condition Coverage (cont’d) if (A || B) s1; else s2; BranchCoverage + Checkeachcondition ConditionCoverage

Branch/Condition Coverage • Branch/Condition Coverage requires that both Branch AND Condition Coverage be achieved • Branch/Condition Coverage subsumes both Branch Coverage and Condition Coverage

Compound Condition Coverage • What if the compiler generates code that masks the evaluation of conditions? • That is, suppose if ((A) || (y/x=5) ) { … } is compiled so that if A is true, y/x=5 will not be evaluated

Compound Condition Coverage • Compound Condition Coverage requires that all combinations of condition values at every branch statement be covered and that every entry point be taken at least once • Also know as Multiple Condition Coverage • Subsumes Branch/Condition Coverage, regardless of the order in which conditions are evaluated

Compound Condition Coverage (cont’d) Possible condition values: TT, TF, FT, FF input(X,Y) if ((Y<=0) || (X==0) ) Y = -Y In general, how many different combinations of condition values must be considered when a branch predicate has n conditions?

Path Coverage • Path Coverage requires that all program paths be traversed at least once • Often described as the “strongest” form of logic coverage? • Is it stronger than Compound Condition Coverage? • Path Coverage is usually impossible when loops are present • How many test cases would be required to cover all paths in the example below?)

Path Coverage for (i=0;i<30;i++) { input(X,Y); if (Y<=0) { if (X<=0) Y = -X; else Y = -Y; } else Y = X+Y; } repeat 30 times

Path Coverage 3 paths 3 X 3 = 9 paths 3 paths

Path Coverage (cont’d) repeat 30 times 3 X 3 X…X 3 = 3 paths 30

Path Coverage • Various strategies have been developed for identifying useful subsets of paths for testing when Path Coverage is impractical: • Loop Coverage • Basis Paths Coverage • Dataflow Coverage

Loop Coverage • Loop Coverage requires that the body of loops be executed • 0, 1, 2, t, max and max+1 times • Rationale • 0: Is some action taken in the body that must also be taken when the body is not executed? • 1: Lower bound on number of times body may be executed • 2: Check loop re-initialization. • t: Check typical number of iterations. • max: Check upper (valid) bound on number of times body may be executed. • max+1: If the maximum can be exceeded, what behavior results?

Coverage tools • Simple statement based coverage metrics can be checked by coverage tools • Sometimes called profilers • Tool augments the code by adding additional code to collect statistics – mainly counts of individual statement executions • At the end of a run of tests, lists of unexecuted statements are readily produced input(a); if ( a < 0 ) { print(“error”); } else x = f(a); input(a); inc_count( 1002 ); if ( a < 0 ) { print(“error”); inc_count( 1004 ); } else { x = f(a); inc_count( 1006 ); }

Statement Coverage tools • Mechanical confirmation that every statement has been executed once is clearly a valuable first stepto our goal of demonstrating that a system has been thoroughly tested • If a statement hasn’t been executed, it’s a potential timebomb! • Can be augmented for compound condition coverage by parsing code and identifying multiple conditions • However, complex or subtle interactions between input parameter values will not necessarily be caught if relying on statement coverage alone! • Path coverage is needed • Correct equivalence class analysis should produce this manually (and therefore at considerable cost!)

Dataflow Coverage

Procedure – Module Verification Steps • Obtain precise specification • Should include definitions of exception or illegal input behaviour • For each input of module • Determine equivalence classes (inc special cases) • Choose representatives • Determine expected outputs • Repeat for outputs • Many output equivalence classes are probably covered by input equivalence classes • Build test table

Procedure – Module Verification Steps • Write test program • Tests in tables is usually the best approach • Easily maintained • Test programs are part of the deliverables for a system • They need to be run when any change is made • To ensure that code that worked isn’t broken by a fix of something else!! or worse • a new feature added! • Tables are easily augmented • When you discover the case that you didn’t test for!

Verification and Validation (Part 2)