Overview

1. ECE 453 – CS 447 – SE 465 Software Testing & Quality AssuranceLecture 10InstructorPaulo Alencar

2. Overview • Structural Testing • Introduction – General Concepts • Flow Graph Testing • DD-Paths • Test Coverage Metrics • Basis Path Testing • Guidelines and Observations • Data Flow Testing • Hybrid Methods • Retrospective on Structural Testing

3. Path Testing • Path Testing is focusing on test techniques that are based on the selection of test paths though a program graph. If the set of paths is properly chosen, then we can claim that we have achieved a measure of test thoroughness. • The fault assumption for path testing techniques is that something has gone wrong with the software that makes it take a different path than the one intended. • Structurally, a path is a sequence of statements in a program unit. Semantically, a path is an execution instance of the program unit. For software testing we are interested in entry-exit paths.

4. Path Testing Process • Input: • Source code and a path selection criterion • Process: • Generation of a CFG • Selection of Paths • Generation of Test Input Data • Feasibility Test of a Path • Evaluation of Program’s Output for the Selected Test Cases

5. V( G) = 3 1 Basis set: x = z+5 1, 2, 3, 4, 6, 7 z = 4*3 - y 1, 2, 3, 4, 5, 4, 6, 7 1, 2, 6, 7 x > z 2 t Flow Graphs – Use in determining Paths for Testing - Revisited x = z+5 6 z = 4*3 - y f R1 R2 if(x > z) goto A; for( u=0; u < x; u++) { 3 z = z+1; y = z+k }; 5 t f u = 0 A: y = z + k z = z+1 R3 7 u++ 4 u < x Example of a simple control flowgraph.

6. 4 5 6 7 8 9 11 10 12 13 14 21 15 16 17 18 20 22 19 23 Program Graph to DD-Path Graph first B D C E F H G I J K L N O M last

7. Topological Paths • We can identify eight possible paths through the previous diagram 1. first – A – B – C – E – F – G – O – last 2. first – A – B – D – E – F – G – O – last 3. first – A – B – C – E – F – H – I – N – O – last 4. first – A – B – D – E – F – H – I – N – O – last 5. first – A – B – C – E – F – H – J – K – M – N – O – last 6. first – A – B – D – E – F – H – J – K – M – N – O – last 7. first – A – B – C – E – F – H – J – L – M – N – O – last 8. first – A – B – D – E – F – H – J – L – M – N – O – last

8. Feasible Paths • If we assume that the logic of the program dictates that “if node C is traversed, then we must traverse node H, and if node D is traversed, then we must traverse Node G” • These constraints will eliminate Paths 1, 4, 6 and 8 • We are left to consider four feasible paths: 2. first – A – B – D – E – F – G – O – last 3. first – A – B – C – E – F – H – I – N – O – last 5. first – A – B – C – E – F – H – J – K – M – N – O – last 7. first – A – B – C – E – F – H – J – L – M – N – O – last

9. first A A E B C G F D I H L K J last • Let’s consider the Triangle program, its CFG, and its DD-Path graph. [Jorgensen: Software Testing A Craftsman’s Approach] 1. program triangle (input, output); 2. VAR a,b,c: integer; 3. IsATriangle: boolean; 4. BEGIN 5. writeln(“Enter integers: “); 6. readln(a,b,c); 7. Writeln(“Side A is: “, a, “Side B is: “, b, “Side C is: “, c); 8. IF(a <b+c) AND (b < a+c) AND (c<a+b) THEN 9. IsATriangle = True; 10. ELSE IsATriangle = False; 11. IF IsATriangle) 12. THEN 13. BEGIN 14. IF (a=b) XOR (a=c) XOR (b=c) AND NOT((a=b) AND (a=c) 15. THEN writeln(“Equilateral”); 16. IF (a=b) AND (b=c) 17. THEN writeln(“Equilateral”); 18. IF(a <> b) AND (a <> c) AND (a<>c) 19. THEN writeln(“Scalene”); 20. END 21. ELSE Writeln(“Not a Triangle”); 22. END P1: A-B-D-E-G-I-J-K-Last P2: A-C-D-E-G-I-J-K-Last P3: A-B-D-L-Last P4: A-B-D-E-F-G-I-J-K-Last P5: A-B-D-E-F-G-H-I-J-K-Last P6: A-B-D-E-F-G-H-I-K-Last

10. first A A E B C G F D I H L K J last • Let’s consider the Triangle program, its CFG, and its DD-Path graph. [Jorgensen: Software Testing A Craftsman’s Approach] 1. program triangle (input, output); 2. VAR a,b,c: integer; 3. IsATriangle: boolean; 4. BEGIN A5. writeln(“Enter integers: “); A6. readln(a,b,c); A7. Writeln(“Side A is: “, a, “Side B is: “, b, “Side C is: “, c); A8. IF(a < b+c) AND (b < a+c) AND (c < a+b) THEN B9. IsATriangle = True; C10. ELSE IsATriangle = False; D11. IF IsATriangle) D12. THEN E13. BEGIN E14. IF (a=b) XOR (a=c) XOR (b=c) AND NOT((a=b) AND (a=c) F15. THEN writeln(“Equilateral”); G16. IF (a=b) AND (b=c) H17. THEN writeln(“Equilateral”); I18. IF(a <> b) AND (a <> c) AND (a<>c) J19. THEN writeln(“Scalene”); K20. END L21. ELSE Writeln(“Not a Triangle”); 22. END Node B is traversed  D, E are traversed Node C is traversed  D, L are traversed Node E is traversed  F, H, J are traversed Node F is traversed  H, J are traversed Node H is traversed  F, J are traversed Node J is traversed  F, I are traversed Logically feasible paths: P1: A-C-D-L-Last P2: A-B-D-E-F-G-I-K-Last P3: A-B-D-E-G-H-I-K-Last P4: A-B-D-E-G-I-J-K-Last

11. Structured Programming Constructs s1 s1 s1 s1 s1 s1 s2 s3 s4 s2 s3 s2 s2 s2 s2 s3 s5 s4 s3 Sequence if-then-else case if-then loop-pre-test loop-post-test

12. Violations of Structured Programming Constructs s1 s1 s1 s1 s0 s0 s2 s2 s2 s2 s0 s1 s3 s3 s3 s3 Branching into loop Branching out of loop Branching into decision Branching out of decision

13. Essential Complexity • Essential complexity. This measures how much unstructured logic exists in a module (e.g., a loop with an exiting GOTO statement) • The essential complexity can be computed on a program graph that has been condensed by collapsing structured programming constructs to a single node, until no more structured constructs can be found

14. Condensing A Program Graph first first a a d A b A b E B C H c G F D L c I H last L K J last e first a first L d V(G) = 1 e last

15. Test Coverage Metrics - Revisited

16. Some Fundamental Path Selection Criteria • Exercise every statement or instruction at least once (statement coverage) C0 • Exercise every branch and case statement, in each direction, at least once (branch coverage) C1 • Exercise each condition in a decision (condition coverage) • Exercise each condition in a decision with all possible outcomes C1P • Exercise every compound predicate outcome (MCC) CMCC • Exercise every path from entry to exit C∞

17. Example 1 2 Statement Coverage C0: SCPath1: 1-2-3(F)-10(F)-11-13 SCPath2: 1-2-3(T)-4(T)-5-6(T)- 7(T)-8-9-3(F)-10(T)-12-13 F T 3 F 10 4 F T T 5 12 11 Branch or Decision Coverage C1: BCPath1: 1-2-3(F)-10(F)-11-13 BCPath2: 1-2-3(T)-4(T)-5-6(T)- 7(T)-8-9-3(F)-10(T)-12-13 BCPath3: 1-2-3(T)-4(F)-10(F)-11-13 BCPath4: 1-2-3(T)-4(T)-5-6(F)-9-3(F)- 10(F)-11-13 BCPath5: 1-2-3(T)-4(T)-5-6(T)-7(F)-9- 3(F)-10(F)-11-13 6 F T 13 7 F T 8 9

18. Condition Coverage • Requires that each condition in a compound decision takes on all possible outcomes at least once • Condition Coverage does not require that each decision takes on all possible outcomes • Example: if Not (A or B) then C Setting A = True, B = False, then A = False and B = True satisfies Condition Coverage, but statement C will never be executed

19. Multiple Condition Coverage • Requires that each possible combination of inputs be tested for each decision. • Example: “if (A and B)” requires 4 test cases: A = True, B = True A = True, B = False A = False, B = True A = False, B = False • For n conditions, 2n test cases are needed, and this grows exponentially with n

20. Modified MCC • Requires that each condition be shown to independently affect the outcome of a decision • Generally, for n conditions, MC/DC requires only n + 1 test cases • Example: if (A or B) then C 3 test cases: A = True, B = False (A dominates) A = False, B = True (B dominates) A = False, B = False (false outcome)

21. Examples • To test if (A or B) A: T F F B: F T F • To test if (A and B) A: F T T B: T F T • To test if (A xor B) A: T T F B: T F T

22. Guidelines and Observations • Functional testing focuses on properties that are “too far” or disassociated from the source code being tested • Path testing focuses too much on the graph and not the logic of the code • Path testing very useful to measure the quality of the testing (coverage criteria) • Basis path testing is giving us a lower bound of how much testing is necessary • Path testing is providing a set of metrics that act as cross checks to functional testing • When a program path is traversed by several functional test cases we suspect redundancy • When we fail to attain DD-Path graph coverage, we know that there are gaps is the functional test cases • Coverage metrics can be used as • Setting minimum testing standards and as • Mechanism to selectively test portions of the code more rigorously than others

23. S P T Guidelines and Observations • We can distinguish between • Specified program behavior (S), • Programmed behavior (feasible paths) (P), and • Topologically feasible paths (T)

24. S P T Guidelines and Observations • Region 1 contains specified behavior that is implemented by feasible paths – most desirable • Every feasible path is topologically possible so the parts 2,6 of set P should be empty • Region 3 contains feasible paths that corresponds to unspecified behavior. Such extra functionality must be inspected • Regions 4 and 7 contain the infeasible paths. Region 4 is problematic as it corresponds to specified behavior that has almost been implemented (topologically possible, yet infeasible program paths) – coding errors • Region 7 refers to unspecified, infeasible, yet topologically possible paths – potential maintenance problem – could become Region3 2 6 5 1 4 3 7

25. Software Testing Standards • IEEE 1008 Software Unit Testing • IEEE 1044 Classification for Software Anomalies • ISO/IEC 12207 Information Technology--Software Life Cycle Processes • ISO/IEC TR 15271 Guide for ISO/IEC 12207 (Software Life Cycle Processes) • AECL CE-1001-STD REV.1 Standard for Software Engineering of Safety Critical Software • BSI BS-7738 Specification for Information Systems Products Using SSADM • (Structured Systems Analysis and Design Method) • BSI BS-7925-1 Software Testing - Vocabulary • BSI BS-7925-2 Standard for Software Component Testing • DEF 00-55 (Part 1)/1 The Procurement of Safety Critical Software in Defence Equipment-Requirements • DIN VDE 0801 Principles for Computers in Safety-Related Systems • ESA PSS-05-01 Guide to the Software Engineering Standards, Issue 1 • German Process-Model (V-Model) Software Life-Cycle Process Model • IAEA TRS-372 Development and Implementation of Computerized Operator Support Systems in Nuclear Installations • IEC 60601-1-4 Medical Electrical Equipment--Part 1: General Requirements for Safety-4.Collateral Standard: Programmable Electrical Medical Systems • IEC 60880 Software for Computers in the Safety Systems of Nuclear Power Stations • IEE 5 Software Inspection Handbook • NCC STARTS Purchasers' Handbook--Procuring Software-Based Systems • NRC NUREG/BR-0167 Software Quality Assurance Program and Guidelines • RTCA DO-178B/ED-12B Software Considerations in Airborne Systems and Equipment Certification