Overview

1. ECE 453 – CS 447 – SE 465 Software Testing & Quality AssuranceLecture 11InstructorPaulo Alencar

2. Overview • Structural Testing • Introduction – General Concepts • Flow Graph Testing • DD-Paths • Test Coverage Metrics • Basis Path Testing • Guidelines and Observations • Data Flow Testing • Hybrid Methods • Retrospective on Structural Testing

3. An Overview of Path Testing Coverage Criteria • Basic Measures • Statement Coverage • Decision Coverage • Condition Coverage • Multiple Condition Coverage • Condition/Decision Coverage • Modified Condition/Decision Coverage • Path Coverage • Other Measures • Function Coverage • Call Coverage • Loop Coverage • Reference for this lecture: http://www.bullseye.com/coverage.html [1]

4. Coverage Analysis • The objectives of coverage analysis are: • To determine “gaps” in test cases (i.e. parts of the program that are not tested) • To establish a quantitative measure for the testing efficiency • To identify additional tests that may be needed to eliminate “gaps” • Coverage analysis assists on evaluating the efficiency of the test cases, and not the quality product itself. Of course the objective is through testing you can eliminate faults and increase the quality of the product at the end. • As we have seen there are several coverage criteria to identify paths in the CFG and consequently generate test cases that cover these paths.

5. Coverage Analysis and Test Plans • A good test plan utilizing coverage analysis must: • Choose the appropriate coverage methodology or criterion (i.e. statement coverage, decision coverage, condition coverage etc.) • Establish minimum acceptable percentage level of coverage • Utilize more than one coverage criterion • Eventually, it is not a good idea to rely only on code coverage. Other techniques are useful as well (i.e. BVA, ECT, DTT).

6. Statement Coverage • The measure reports on the percentage of executable statements of the code exercised by a set of test cases. Also known as basic bloc coverage or segment coverage. • The major advantage of this measure is that it can be applied directly to object code and does not require processing source code. Performance profilers commonly implement this measure. • The major disadvantage of statement coverage is that it is insensitive to some control structures

7. Statement Coverage Examples int* ptr = NULL; if (B) ptr = &variable; *ptr = 10; Test Case: B=True Statement Coverage Achieved Problem : if B is false the code will fail with a null pointer exception. if(A) then F1(); F2(); Test Case: A=True Statement Coverage achieved

8. Statement Coverage Comments • Statement coverage does not report whether loops reach their termination condition - only whether the loop body was executed. • Since do-while loops always execute at least once, statement coverage considers them the same rank as non-branching statements. • Statement coverage is completely insensitive to the logical decisions. • One argument in favor of statement coverage over other measures is that faults are evenly distributed through code; therefore the percentage of executable statements covered reflects the percentage of faults discovered. • Statement coverage requires in most cases very few test cases to achieve. Not acceptable to release code based on statement coverage alone.

9. Decision Coverage • Decision coverage (also known as branch coverage, all-edges coverage, basis path coverage, decision-decision-path testing) reports whether boolean expressions in control structures are evaluated to both true and false values by the test cases. • Note in decision coverage we are not interested in all combinations of True/False values of all program predicates, just that we have exercised all the different values in our test cases. • The entire boolean expression is considered one true-or-false predicate. • Additionally, this measure includes coverage of switch-statement cases, exception handlers, and interrupt handlers.

10. Decision Coverage Examples if(A) F1(); else F2(); if(B) F3() else F4(); Test Cases for Decision Coverage: A=T, B=T A=F, B=F if (A && (B || F1())) F2(); else F3(); Test Cases for Decision Coverage: A=T, B=T A=F Problem: F1() never gets called. This problem occurs in languages with short circuiting boolean operators

11. Decision Coverage Comments • This measure has the advantage of simplicity without the problems of statement coverage. • A disadvantage may produce gaps in tested code in programs written in languages that have short-circuit logic operators (C, C++, Java, etc.)

12. Condition Coverage • Condition coverage reports the true or false outcome of each boolean sub-expression. • Condition coverage measures the sub-expressions independently of each other. • This measure is similar to decision coverage but has better sensitivity to the control flow. • An extension of Condition Coverage is the Condition/Decision Coverage which is a hybrid measure composed by the union of condition coverage and decision coverage. It has the advantage of simplicity but without the shortcomings of its component measures.

13. Condition Coverage Examples if(A && B) F1(); else F2(); if(C) F3() else F4(); Test Cases for Condition Coverage: A=T, B=T, C=F A=F, B=F, C=T if(A && B) F1(); else F2(); if(C) F3() else F4(); Test Cases for Condition Coverage: A=T, B=F, C=F A=F, B=T, C=T Problem: decision coverage not achieved

14. Multiple Condition Coverage • Multiple condition coverage reports whether every possible combination of boolean sub-expressions occurs. • The test cases required for full multiple condition coverage of a condition are essentially given by the logical operator truth table for the condition.

15. Multiple Condition Coverage Examples if(A && B) // condition 1 F1(); else F2(); if(C || D)// condition 2 F3() else F4(); Test Cases for MCC: For condition 1 For condition 2 A=T, B=T C=T, D=T A=T, B=F C=T, D=F A=F, B=T C=F, D=T A=F, B=F C=F, D=F

16. Multiple Condition Coverage Comments • For languages with short circuit operators multiple condition coverage is very similar to condition coverage (but with more test cases). • For languages without short circuit operators multiple condition coverage is effectively path coverage. • A disadvantage of this measure is that it can be tedious to determine the minimum set of test cases required, especially for very complex boolean expressions. • An additional disadvantage of this measure is that the number of test cases required could vary substantially among conditions that have similar complexity. Consider a && b && (c || (d && e)) and the ((a || b) && (c || d)) && e [1] • As with condition coverage, multiple condition coverage does not include decision coverage.

17. Modified MCC • Also known as MC/DC and MCDC. • This measure also known as MC/DC and MCDC requires enough test cases to verify every condition can affect the result of its encompassing decision. • For C, C++ and Java, this measure requires exactly the same test cases as condition/decision coverage. • Modified condition/decision coverage was designed for languages containing logical operators that do not short-circuit. • The short circuit logical operators in C, C++ and Java only evaluate conditions when their result can affect the encompassing decision.

18. Modified MCC Examples • To test if (A or B) A: T F F B: F T F • To test if (A and B) A: F T T B: T F T • To test if (A xor B) A: T T F B: T F T

19. Path Coverage • This measure reports whether each of the possible paths in each function have been followed. A path is a unique sequence of branches from the function entry to the exit. This measure is also known as predicate coverage. • Since loops introduce an unbounded number of paths, this measure considers only a limited number of looping possibilities. • A large number of variations of this measure exist to cope with loops. Boundary-interior path testing considers two possibilities for loops: zero repetitions and more than zero repetitions. For do-while loops, the two possibilities are one iteration and more than one iteration.

20. Path Coverage Examples A if (A) F1(); F2(); if (A) F3(); F4(); Paths: A-F1-F2-A-F3-F4 A-F2-A-F3-F4 A-F1-F2-A-F4 A-F2-A-F4 Problem: only two are feasible A=T A=F F1 F2 A F3 F4

21. Path Coverage Comments • Path coverage has the advantage of requiring very thorough testing. • Path coverage has two severe disadvantages. • The first is that the number of paths is exponential to the number of branches. • The second disadvantage is that many paths are impossible to exercise due to relationships of data. • There are many variations of path coverage to deal with the large number of paths. Examples include, n-length sub-path coverage reports whether we have exercised each path of length n branches. Other examples include linear code sequence and jump (LCSAJ) coverage and data flow coverage.

22. Loop Coverage • This measure reports whether each loop body in the program has been executed zero times, exactly once, and more than once (consecutively). • For do-while loops, loop coverage reports whether the body has been executed exactly once, and more than once. • The valuable aspect of this measure is determining whether while-loops and for-loops execute more than once, information not reported by others measure [1].

23. Function Coverage • This measure reports whether each function or procedure in the program has been invoked. • It is useful during preliminary testing to assure at least some coverage in all areas of the software.

24. Call Coverage • This measure reports whether each function call in the program has been executed. The hypothesis is that faults commonly occur in interfaces between modules. • Also known as call pair coverage.

25. Coverage Measures Comparison • Relative strengths when a stronger measure includes a weaker measure. • Decision coverage includes statement coverage since exercising every branch must lead to exercising every statement. • Condition/decision coverage includes decision coverage and condition coverage (by definition). • Path coverage includes decision coverage. • Predicate coverage (paths as possible combinations of logical conditions) includes path coverage and multiple condition coverage, as well as most other measures.

26. Testing Strategy (1) • Your highest level of testing productivity occurs when we identify the most failures with the least effort. • Effort is measured by the time • Required to create test cases, • Add the test cases to your test suite and • Run the test suites. • It follows that we should use a coverage analysis strategy that increases coverage as fast as possible. This provides the greatest probability of finding failures sooner rather than later.

27. Testing Strategy (2) • One strategy that usually increases coverage quickly is to first attain some coverage throughout the entire test program before striving for high coverage in any particular area [1] • The sequence of coverage goals listed below illustrates a possible implementation of this strategy [1] • Invoke at least one function in 90% of the source files (or classes). • Invoke 90% of the functions. • Attain 90% condition/decision coverage in each function. • Attain 100% condition/decision coverage. • Notice we do not require 100% coverage in any of the initial goals. This allows us to defer testing the most difficult areas and maintain high testing productivity; by achieving maximum results with minimum effort [1] Source: http://www.bullseye.com/coverage.html [1]

28. Testing Productivity Source: http://www.bullseye.com/coverage.html [1]