1 / 98

Testing Techniques

Testing Techniques. Mary Jean Harrold Georgia Institute of Technology. Outline. Introduction Black-box coverage measures equivalence partitioning boundary-value analysis White-box coverage measures control flow condition data-flow fault-based Integration techniques

aben
Download Presentation

Testing Techniques

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Testing Techniques Mary Jean Harrold Georgia Institute of Technology

  2. Outline • Introduction • Black-box coverage measures • equivalence partitioning • boundary-value analysis • White-box coverage measures • control flow • condition • data-flow • fault-based • Integration techniques • Regression testing techniques

  3. Verification and Validation (V&V) • Verification answers the question: Are we building the product right? Thus, it refers to the testing we do to assure that that the product resulting from one phase is a correct translation from the product resulting from the previous phase. For example, we attempt to verify that the implementation is consistent with the design. • Validation answers the question: Are we building the right product? Thus, it refers to the testing we do to assure that the resulting software is traceable to customer requirements.

  4. Requirements Analysis Phase: Design Phase: Implementation Phase: Integration Phase: Maintenance Phase: Develop test plan and system tests; perform technical review Develop integration tests; perform technical review Develop and run unit tests; perform technical review Run integration tests Run system tests Run regression tests Software Development Phases and Testing

  5. Software Development Phases and Testing (Graphical View) Requirements Analysis System High-Level Design Integration Low-Level Design Unit Coding Unit Delivery Acceptance Maintenance Regression

  6. Some Important Facts Fact 1: It’s cheaper to find and fix faults at the unit level -- most faults are incorrect computations within subroutines Fact 2: It’s good to do as much testing early as you can -- you can’t test quality in, you have to build it in

  7. Testing Terminology P is a program, S is the specification for P P is correct for d in D if P(d) = S(d) A d in D is called a test case A finite subset of D is called a test suite or test set P D d R S

  8. Testing Terminology* Failure:an erroneous result • incorrect outputs/response for given inputs/stimuli • fails to meet real-time constraints Error:incorrect concept • may cause failures if not corrected Fault:the cause of one or more failures • discovered after release *Additional definitions in “Testing Definitions” handout

  9. Testing Terminology IMPORTANT: Fault doesn’t imply failure - a program can be coincidentally correct in that it executes the fault but does not fail SQUARE(z) // finds square of z y = 2 * z print y Fault: line “y = 2 * z” should be “y = z * z” Failure: for input y = 2 Coincidentally correct: for other inputs

  10. Coverage Criteria Test Selection (Coverage) Criterion C: a rule for selecting d in D to place in T We want a C that gives test suites that guarantee correctness We settle for a C that gives test suites that improve confidence Limitations of coverage criteria: achieving coverage doesn’t show correctness Types of criteria: black-box: specification, state white-box: control flow, condition, data-flow, fault-based

  11. Coverage Criteria Coverage criteria C is used in two ways: 1. To measure the quality of a test suite T (i.e., help us evaluate T) the adequacy score is the percentage of “coverable” items identified using C that are covered by T 2. To guide the selection of an C-adequate test suite T(i.e., tell us when to stop testing): the adequacy requires that some percentage of “coverable” items are covered by tests in T

  12. Black-Box Testing • Selection of test suite is based on some aspect (such as requirements, specification, function) other than the code • We won’t discuss further

  13. Selection of test suite is based on some aspect of the code We’ll consider several examples control flow statement branch basis path path condition simple multiple loop dataflow all-uses all-du-paths fault based mutation White-Box Testing

  14. Some Terminology Test Requirements: those aspects of the program that must be covered according to the coverage criterion Test Specification: constraints on the input that will cause a test requirement to be satisfied Test Case (Test Suite): a set of inputs that satisfies a test specification

  15. 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < = 10) do 5. if (j >0) 6. sum = sum + j endif 7. i= i + 1 8. read j endwhile 9. print sum Sum: Example Program

  16. White-Box Testing Statement Coverage • test requirements: statements in the program • test specifications: constraints on inputs that cause statements to be executed • test cases: inputs that satisfy the constraints in the test specifications For Sum • test requirements: statements 1-9 • test specifications: (0<i<=10 and j>0 at least once) • test suite: i = 10, j1 = 4, j2 = 10; expect sum = 4

  17. White-Box Testing Branch Coverage • test requirements: branches in program • test specifications: constraints on inputs that cause branches to be executed • test cases: inputs that satisfy the constraints in the test specifications For branch coverage, use control-flow graph (CFG): nodes represent statements (or sequences of statements) in the program; edges represent flow of control between statements

  18. 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < = 10) do 5. if (j >0) 6. sum = sum + j endif 7. i= i + 1 8. read j endwhile 9. print sum Sum: Control-Flow Graph 1,2,3 T 4 5 T F F 6 9 7,8 Some CFG edges are labeled (e.g., (4,5) and (5,7)) Some CFG edges are backedges (e.g., (7-8,4))

  19. White-Box Testing Branch Coverage For Sum test requirements: 4T, 4F, 5T, 5F test specifications: for example, for 4T (0<i<=10, j’s can be any integer) for 5F (0<i<=10, at least one of the (used) j’s <=0) test suite: i = 10, j1 = 4, j2 = 10; expect sum = 4 i = 10, j1 = -3, j2 =5; expect sum = 0

  20. White-Box Testing Basis-Path Coverage • test requirements: a set of basis paths in the program • test specifications: constraints on inputs that cause these basis paths to be executed • test cases: inputs that satisfy the constraints in the test specifications For basis-path coverage, need cyclomatic complexity: defines the maximum number of independent paths in the program.

  21. Cyclomatic Complexity; Basis Paths Cyclomatic complexity (CC) can be computed in three ways: 1. e - n + 2p, where e is the number of edges in the CFG, n is the number of nodes in the CFG, p is the number of connected components in the CFG 2. number of predicates + 1 for structured programs 3. number of regions in the program CC is an upper bound on the number of basis paths

  22. CC (by three methods) 1. e = 7, n = 6, p = 1 7 - 6 + 2 * 1 = 3 2. 2 predicates (4 and 5) 2 + 1 = 3 3. 3 regions: one inside 5,6,7-8 one inside 4,5,7-8 one outside graph Sum: Cyclomatic Complexity and Basis Paths 1,2,3 T 4 5 T F F 6 9 7,8

  23. Basis Paths there can be many sets but no set can be bigger than CC Two example sets are: 1. 1-2-3, 4, 9 1-2-3, 4, 5, 6, 7-8, 4, 9 1-2-3, 4, 5, 7-8, 4, 9 2. 1-2-3, 4, 5, 6, 7-8, 4, 9 1-2-3, 4, 5, 7-8, 4, 9 1,2,3 T 4 5 T F F 6 9 7,8 Sum: Cyclomatic Complexity and Basis Paths

  24. main( ) { 1. sum = 0 2. i = 1 3. while (i <= 5) { 4. scanf(“%d”, &j); 5. If (j < 0) 6. continue; 7. sum = sum + j; 8. if (sum > 10) 9. break; 10. i = i + 1; } 11.printf(“sum is %d”, sum); } Sum2: Example Program 1,2 T 3 4,5 T F F 6 11 7,8 T F 9 10

  25. White-Box Testing Basis Path Coverage For Sum test requirements: a set of basis paths test specifications: for example, (you complete) test suite: (you complete)

  26. White-Box Testing Condition Coverage • test requirements: each condition must be T, F under test • test specifications: constraints on inputs that cause each condition to be true, each condition to be false • test cases: inputs that satisfy the constraints in the test specifications

  27. 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < = 10) do 5. if (j >0) 6. sum = sum + j endif 7. i= i + 1 8. read j endwhile 9. print sum For Sum test requirements: in 4, I>0 must be T, F in f4, I<=10 must be T, F 5 must be T, F test specifications: (you complete) test suite: (you complete) Sum: Example Program

  28. White-Box Testing Multiple Condition Coverage • test requirements: each condition possible combination of truth values must be assigned • test specifications: constraints on inputs that cause the requirements to hold • test cases: inputs that satisfy the constraints in the test specifications

  29. 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < = 10) do 5. if (j >0) 6. sum = sum + j endif 7. i= i + 1 8. read j endwhile 9. print sum For Sum test requirements: in 4, conditions must be TT, TF, FT, FF in 5, conditions must be T, F test specifications: (you complete) test suite: (you complete) Sum: Example Program

  30. White-Box Testing Loop Coverage • test requirements: skip loop; once through loop; twice through loop; max through loop • test specifications: constraints on inputs that cause loop iterations to be executed • test cases: inputs that satisfy the constraints in the test specifications For Sum (you complete)

  31. White-Box Testing Data-flow Coverage: All Uses • test requirements: sets of definition-use pairs (du-pairs); with respect to variable v, • a definition of v is a reference to v where the value of v is changed (e.g., assignment to v, input a value of v) • a usev is a reference to v where the value of v is fetched by not changed (e.g., v on right-hand side of assignment, v is output) • a definition-clear subpath for a definition d of v and a use u of v is a subpath in the CFG between d and u on which v is not redefined • test specifications • test cases

  32. 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < = 10) do 5. if (j >0) 6. sum = sum + j endif 7. i= i + 1 8. read j endwhile 9. print sum Definition-Clear Paths; All-Uses Coverage defs of i, j, sum 1,2,3 4 5 uses of i use of j 6 uses of sum, j def of sum 9 use of sum use of i defs of i, j 7,8 Def-use pairs for i: (1, (4,5)), (1, (4,9)), (1, 7), (7, (4,5)), (7, (4,9))

  33. 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < = 10) do 5. if (j >0) 6. sum = sum + j 6a. print sum endif 7. i= i + 1 8. read j endwhile 9. print sum Definition-Clear Paths; All-Uses Coverage defs sum 1,2,3 4 5 use of sum def of sum 6 9 use of sum 6a use of sum 7,8

  34. White-Box Testing Mutation Analysis/Testing Major Premise: The quality of a test set is related to the ability of that test set to differentiate the program being tested from a set of marginally different alternative programs Differentiating Programs: A test case differentiates two programs if it causes the two programs to produce different results

  35. White-Box Testing Mutation Analysis/Testing -- Terminology 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < = 10) do 5. if (j >0) 6. sum = sum + j 6a. print sum endif 7. i= i + 1 8. read j endwhile 9. print sum Mutate: make a small syntactic change

  36. White-Box Testing Mutation Analysis/Testing -- Terminology 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < = 10) do 5. if (j >0) 6. sum = sum + j 6a. print sum endif 7. i= i + 1 8. read j endwhile 9. print sum Mutate: make a small syntactic change

  37. White-Box Testing Mutation Analysis/Testing -- Terminology 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < = 10) do 5. if (j >0) 6. sum = sum + j 6a. print sum endif 7. i= i + 1 8. read j endwhile 9. print sum 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < = 10) do 5. if (j >0) 6. sum = sum - j 6a. print sum endif 7. i= i + 1 8. read j endwhile 9. print sum Mutate: make a small syntactic change

  38. White-Box Testing Mutation Analysis/Testing -- Terminology 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < = 10) do 5. if (j >0) 6. sum = sum + j 6a. print sum endif 7. i= i + 1 8. read j endwhile 9. print sum 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < = 10) do 5. if (j >0) 6. sum = sum * j 6a. print sum endif 7. i= i + 1 8. read j endwhile 9. print sum Mutate: make a small syntactic change

  39. White-Box Testing Mutation Analysis/Testing -- Terminology 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < = 10) do 5. if (j >0) 6. sum = sum + j 6a. print sum endif 7. i= i + 1 8. read j endwhile 9. print sum 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < = 10) do 5. if (j >0) 6. sum = sum + i 6a. print sum endif 7. i= i + 1 8. read j endwhile 9. print sum Mutate: make a small syntactic change

  40. White-Box Testing Mutation Analysis/Testing -- Terminology 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < = 10) do 5. if (j >0) 6. sum = sum + j 6a. print sum endif 7. i= i + 1 8. read j endwhile 9. print sum 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < 10) do 5. if (j >0) 6. sum = sum + i 6a. print sum endif 7. i= i + 1 8. read j endwhile 9. print sum Mutate: make a small syntactic change

  41. White-Box Testing Mutation Analysis/Testing -- Terminology 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < = 10) do 5. if (j >0) 6. sum = sum + j 6a. print sum endif 7. i= i + 1 8. read j endwhile 9. print sum 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < 10) do 5. if (j >0) 6. sum = sum + i 6a. print sum endif 7. i= i + 1 8. read j endwhile 9. print sum Mutate: make a small syntactic change

  42. White-Box Testing Mutation Analysis/Testing -- Terminology 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < = 10) do 5. if (j >0) 6. sum = sum + j 6a. print sum endif 7. i= i + 1 8. read j endwhile 9. print sum 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i > 10) do 5. if (j >0) 6. sum = sum + i 6a. print sum endif 7. i= i + 1 8. read j endwhile 9. print sum Mutate: make a small syntactic change Mutation: the changed statement Mutant: program with a mutated statement

  43. 1. read i 2. read j 3. sum = 0 4. while (i > 0) and (i < = 10) do 5. if (j >0) 6. sum = sum + j endif 7. i= i + 1 8. read j endwhile 9. print sum 1. SVR -- scalar variable replacement: replace each occurrence of variable v with all other variables in scope e.g., change 1 to read j read sum 2. AOR -- arithmetic operator replacement: replace each occurrence of an arithmetic operator with all possibilities e.g., change 6 to sum = sum -j, etc. Sum: Example Mutations

  44. int Sum (int I, j) int sum 1. sum = 0 2. while (i > 0) and (i < = 10) do 3. if (j >0) 4. sum = Add(sum, j) endif 7. i= Add(I, 1) 8. read j endwhile 9. return sum We now have three functions: Main, Sum, and Add How do we develop and test a system with many modules? Stubs need to simulate modules used by current module Drivers need to simulate modules that use current module Sum3: Example Program

  45. System Integration, Build Plan, Integration Testing • Development or integration strategies • Decide the order in which components of the system will be developed

  46. Order of Integration: USES Relation For any two distinct modules Mi and Mj, we say that Mi USES Mj if and only if correct execution of Mj is necessary for Mi to complete the task described in its specification. If Mi USES Mj, we also say that Mi is a client of Mj because Mi requires the services that Mj provides

  47. Deal Collect Cards Play Hand Determine Winner Announce Winner Get Hand Value Compare Hands Integration: Uses Graph Play Game

  48. Common build strategies • Big Bang • All coding precedes all integration • Bottom Up • Start at low-level utility modules • Top Down • Start at high-level control modules • Incremental/Sandwich • Integrate control modules top down and utility modules bottom up

  49. Play Game Deal Collect Cards Play Hand Determine Winner Announce Winner Get Hand Value Compare Hands Big Bang Integration

  50. Big Bang Integration • Advantages • No need to write stubs or drivers • Disadvantages • Difficult to identify units causing errors • Critical modules receive no extra testing • Major design flaws are discovered very late • No flexibility in scheduling

More Related