1 / 45

Implementation-Based Testing

CEN 5076 Class 10 – 11/07. Implementation-Based Testing. Review Class 9 Implementation-based Testing. Test Adequacy Criteria. Definitions from Zhu et al. ‘97:

carnig
Download Presentation

Implementation-Based Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CEN 5076 Class 10 – 11/07 Implementation-Based Testing Review Class 9 Implementation-based Testing

  2. Test Adequacy Criteria Definitions from Zhu et al. ‘97: • Test Data Adequacy as Stopping Rules: A test data adequacy criterion C is a function C: P X S X T → {true, false}. C(p, s, t) = true means that t is adequate for testing program p against specification s according to criterion C, otherwise t is inadequate. CEN 5076 Class 10 - 11/07

  3. Test Adequacy Criteria • Test Data Adequacy Criteria as Measurements: A test data adequacy criterion C is a function C: P X S X T → [0,1]. C(p, s, t) = r means thatthe adequate of testing the program p by the test set t w.r.t.specification s is of degree r according to criterion C. The greater the value of r, the more adequate the testing. CEN 5076 Class 10 - 11/07

  4. Test Adequacy Criteria • Test Data Adequacy Criteria as Generators: A test data adequacy criterion C is a function C: P X S → 2T. A test set t C(p, s) means that t satisfies C w.r.t. p and s, and it is said that t is adequate for (p, s) according to C. CEN 5076 Class 10 - 11/07

  5. Types of Testing Specification-based (functional) • Specifies the required testing in terms of identified features of the specification or the requirements of the software, so that a test set is adequate if all the identified features have been fully exercised. [Zhu ’97] Implementation-based (program-based) • Specifies testing requirements in terms of the program under test and decides if a test set is adequate according to whether the program has been thoroughly exercised. CEN 5076 Class 10 - 11/07

  6. Specification-based Testing cont • A specification has two facets: a syntactic structure and a semantic structure. • There are two role the specifications plays during software testing: • To provide the necessary information to check whether the output of the program is correct, the oracle problem. • Provide information to select test cases and to measure test adequacy CEN 5076 Class 10 - 11/07

  7. Specification-based Testing cont Techniques used to represent the specification: • Logic-based • State machines • Algebraic specs. Model-based Property-oriented CEN 5076 Class 10 - 11/07

  8. Specification-based Testing cont Test generation for Logic-based models: • All-Explicit variants • All-Variants, All-True, All-False, All-Primes • Each-Condition/All-Conditions • BDD-Determinants • Variable Negation • Nonbinary Variable Domain Analysis CEN 5076 Class 10 - 11/07

  9. Specification-based Testing cont Test generation for state-based models: • Piecewise coverage - all states, all events, all actions. • All transitions • All n-Transition Sequences • All Round-trip Paths • M-length Signature CEN 5076 Class 10 - 11/07

  10. Faults Revealed Strategy All possible invalid states All event and action faults Exhaustive All invalid states Up to M extra states All event and action faults M-length Signature All invalid/missing states Some extra states All event and action faults All Round-trip Paths All Transition k-tuples All event and action faults Some missing/invalid states All Transitions Some event and action faults All States All Events All Actions Fig. State-based test power hierarchy CEN 5076 Class 10 - 11/07

  11. Specification-based Testing cont • Algebraic Specifications • Algebraic spec consists of a set of operations that the operations of the software must satisfy. • An algebraic spec has two parts: • Syntactic - consists of functions name and their signatures (the input and output types). • Semantic part – consists of a list of axioms describing relation among functions CEN 5076 Class 10 - 11/07

  12. Specification-based Testing cont • Syntax part: • Functions that return values of type T (type of algebraic spec) are called constructors and transformers. • Functions that return values other than T are called observers. • Semantic part: • Allows for a list of preconditions describing the domains of functions. • Allows functions to return an error if element is outside of the domain. CEN 5076 Class 10 - 11/07

  13. Specification-based Testing cont • An equation in an algebraic spec. consists of two terms as two sides of the eqn. • A term is constructed from three types of symbols: variables, constants, and operators. • Each term has two interpretations in the context of testing. • As a sequence of calls to the operations that implement the operators specified in the spec. • As a value that is the result of the sequence of operations. CEN 5076 Class 10 - 11/07

  14. Specification-based Testing cont • For a algebraic spec to be useful it must be consistent and sufficiently complete. - Consistent must not contain any contradictory axioms i.e., no contradiction should be derivable. • Sufficiently complete if for every sequence w in W the result of applying each observer is defined. W is the set of all operation sequences consisting of constructors and transformers. • See handout for example of an algebraic spec for a priority queue ASTOOT Doong and Frankl ‘94 CEN 5076 Class 10 - 11/07

  15. Specification-based Testing cont • The regularity hypothesis [Bouge et al. 1986] assumes that for some complexity degree k, if a program satisfies an equation t(x) = t’(x) on all data of complexity not higher than k, then the program satisfies the eqn on all data. • No way to determine the complexity k s.t. only the test cases of complexity less than k need to be tested. CEN 5076 Class 10 - 11/07

  16. Specification-based Testing cont • Bouge et al. 1986 Bernot et al.1991 suggest that the selection of test cases should be based on partitioning the set of ground terms according to their complexity so that the regularity and uniformity hypotheses in the subsets in the partitioned are assumed. • The complexity of a test case is the depth of nesting of the operators in the ground term. Ground terms are terms without variables. CEN 5076 Class 10 - 11/07

  17. Specification-based Testing cont • The selection of test cases should first consider constants specified by the spec, then all the values generated by one application of operations on constants, the values generated by two applications on constants etc.. CEN 5076 Class 10 - 11/07

  18. Specification-based Testing cont Constructing Test Cases for Classes: • Test cases are usually identified from the class specification. The spec can be stated in OCL, natural language, and/or state transition diagrams. • Test case construction from pre- and postconditions • Create test cases that covers all possible combinations of situations in which a precondition can hold and a postcondition can be achieved. P. 169 – 176 in text. CEN 5076 Class 10 - 11/07

  19. Specification-based Testing cont • It is possible to express the adequacy for the spec in terms of the number of pairs of pre- and postconditions that have been covered. CEN 5076 Class 10 - 11/07

  20. Implementation-based Testing • Specifies testing requirements in terms of the program under test and decides if a test set is adequate according to whether the program has been thoroughly exercised. • There are two main groups of implementation-based structural test adequacy criteria. • Control-flow • Data-flow CEN 5076 Class 10 - 11/07

  21. Control Flow Adequacy Criteria • Flow Graph (FG) [Binder 00] • A segment is represented by a node in the FG (a sphere). A segment is one or more contiguous statements with no conditionally executed statements. • A conditional transfer of control is a branch. A branch is represented by an outgoing edge in the FG. • The entry point of a method is represented by the entry node, which is a node no incoming edges. The exit point of a method is represented by the exit node, which has no outbound edges. CEN 5076 Class 10 - 11/07

  22. Control Flow Adequacy Criteria cont Flow graph Entry A Example: Source code 1.public int Fun(int x){ 2. k = 0; 3. while (x <= 10 && k < 3){ 4. if (x%2 != 0) 5. k = k + 1; 6. x = x + 1; 7. } 8. if (x < 0){ 9. x = 10; 10. k = 0; 11. } 12. return k; } B K=0 F x<=10 && k<3 C G T F X<0 F x%2 != 0 D T T X = 10 K = 0 k=k+1 E H x=x+1 F return k I CEN 5076 Class 10 - 11/07

  23. Control Flow Adequacy Criteria cont Types of code coverage based on FG model: • Statement coverage – A set P of execution paths satisfies the statement coverage criterion iff for all nodes n in the FG, there is at least one path p in P s.t. n is on the path p. • Generate test data to execute every stmt in the program at least once. • Give a value(s) of x in the example that satisfies statement coverage. CEN 5076 Class 10 - 11/07

  24. Control Flow Adequacy Criteria cont • Branch coverage – A set P of execution paths satisfies the branch coverage criterion iff for all edges e in the FG, there is at least one path p in P s.t. p contains edge e. • Generate test data to exercise the true and false outcomes of every decision. • Give a value(s) of x in the example that satisfies branch coverage. CEN 5076 Class 10 - 11/07

  25. Control Flow Adequacy Criteria cont • Path coverage – A set P of execution paths satisfies the path coverage criterion iff P contains all execution paths from the begin node to the end node in the FG. • Generate test data to exercise all paths in the program. Note every path through a loop is considered to be different and distinct. • Give a value(s) of x in the example that satisfies path coverage. CEN 5076 Class 10 - 11/07

  26. Control Flow Adequacy Criteria cont • Cyclomatic-Number Criterion – A set P of execution paths satisfies the cyclomatic number criterion iff P contains at least one set of v independent paths, where v = e – n + p is the cyclomatic number of the FG. • n - number of vertices, e - is the number of edges, p - is the number of strongly connected components. • A graph is strongly connected if, for any two nodes a and b there exists a path from a to b and a path from b to a. CEN 5076 Class 10 - 11/07

  27. Control Flow Adequacy Criteria cont • Basis path testing – is a way of easily identifying the upper bound for the number of paths necessary in order to achieve branch coverage. v = (e + 1) - n + 1 McCabe Cycolmatic Complexity Number • Note this is for a strongly connected FG, add edge from exit node to entry node, hence e + 1. • Note some formulae include the extra edge in the “e”. • Identify the basis set for the FG example. CEN 5076 Class 10 - 11/07

  28. Control Flow Adequacy Criteria cont Types of code coverage based on program text: • Multiple Condition Coverage – A test set T is said to be adequate according to the multiple-condition-coverage criterion if, for every condition C, which consists of atomic predicates (p1, p2, …,pn), and all the possible combinations (b1, b2, …,bn) of their truth values, there is al least one test case in T s.t. the value of pi equals bi, i= 1, 2, …,n. • Generate test data to exercise all possible combinations of true and false outcomes of conditions within a decision. • Create test set for example that satisfies the MCC CEN 5076 Class 10 - 11/07

  29. Control Flow Adequacy Criteria cont Points to note: • Path coverage not always practical due to loops i.e., infinite number of paths. • Statement and branch coverage are not finitely applicable due to dead code and infeasible branches. • Can define a finitely applicable version of the criteria. CEN 5076 Class 10 - 11/07

  30. Control Flow Adequacy Criteria cont • However we may not be able to decide whether a test set satisfies a given feasible adequacy criterion e.g. whether a statement in a program is feasible is undecidable. • Research into flow graph criteria has focused on the most important subsets of paths e.g., length-n path coverage, level-i path coverage, cyclomatic-number coverage, CEN 5076 Class 10 - 11/07

  31. Data-Flow Adequacy Criteria • Data-flow analysis of test adequacy is concerned with the coverage of flow-graph paths that are significant for the data flow in the program. • Each variable occurrence is classified as either a definition occurrence or a use occurrence. • A definitional occurrence (def) of a variable is where a value is bound to the variable. • A use occurrence(use) of a variable is where the value of the variable is referred. CEN 5076 Class 10 - 11/07

  32. Data-Flow Adequacy Criteria cont • Two types of use occurrences: • Predicate use (p-use) – if the value of a variable is used to decide whether a predicate is true or false. • Computation use (c-use) – variable used in the evaluation of an expression, call-by-value argument or an output statement. • Data-flow test adequacy analysis is concerned with subpaths from nodes with a def to nodes with a use. • A def-clear path w.r.t a variable x is a path where for all nodes in the path there is no def occurrence of x. CEN 5076 Class 10 - 11/07

  33. Data-Flow Adequacy Criteria cont • A global c-use of a variable x is where no def of x precedes the c-use within the node in which it occurs. • A def of a variable x at node u reaches a c-use at node v iff there is a path from u to v s.t. p =(u, w1, w2, …,wn, v) and (w1, w2, …,wn) is a def-clear w.r.t x and the occurrence of x at v is a global c-use. • We say that the def of x at u reaches the c-use of x at v through path p. CEN 5076 Class 10 - 11/07

  34. Data-Flow Adequacy Criteria cont • A def of a variable x at node u reaches a p-use at node v iff there is a path from u to v s.t. p =(u, w1, w2, …,wn, v) and (w1, w2, …,wn) is a def-clear w.r.t x and there is a predicate occurrence of x associated with the edge from wn to v. • We say that u reaches the predicate use of x on the edge (wn, v) through path p. • If there is at least one input datum that can actually cause the execution of the path, then a def feasibly reaches a use of the def. CEN 5076 Class 10 - 11/07

  35. Data-Flow Adequacy Criteria cont • All Definitions Criterion – A set P of execution paths satisfies the all-defs criterion iff for all defs of x s.t. there is a use of x which is feasibly reached from the def, there is at least one path p in P s.t. p includes a subpath through which the def of x reaches some use of x. • All Uses Criterion – A set P of execution paths satisfies the all-uses criterion iff for all defs of x and all uses of x that the def feasibly reaches, there is at least one path p in P s.t. p includes a subpath through which that def reaches the use. CEN 5076 Class 10 - 11/07

  36. Data-Flow Adequacy Criteria cont • All Definition-Use-Paths Criterion (All DU-Paths) – A set P of execution paths satisfies the all-du-paths criterion iff for all defs of x and all paths q through which that def reaches a use of x, there is at least one path p in P s.t. q is a subpath of p, and q is cycle-free or contains only simple cycles. • Note such a path may be infeasible. CEN 5076 Class 10 - 11/07

  37. Data-Flow Adequacy Criteria cont Points to note: • Data flow testing methods make no distinction between atomic data and structured or aggregate data. • Modification and references to an element of a structured datum are regarded as modifications and references to the whole datum. Can lead to errors [Hamlet ’93]. • Dynamic data not considered in the data flow testing i.e., a pointer variable may refer to a number of data and vice versa. CEN 5076 Class 10 - 11/07

  38. Data-Flow Adequacy Criteria cont • For classes there is intra-method, inter-method, intra-class, and inter-class. • For each of the data-flow coverage criteria try to identify test cases for the example that provides adequate coverage for feasible paths. Common errors: • Using an undefined or uninitialized variable • Assigning a value to a variable more than once without an intermediate reference. • Sending a modifier message to an object more than once without an intermediate accessor method. CEN 5076 Class 10 - 11/07

  39. Data-Flow Adequacy Criteria cont • Sending a modifier message to an object more than once without an intermediate accessor method. • Deallocating or reinitializing a variable before it is used. • Using incorrect values in the signature of a polymorphic message, resulting in incorrect binding. • The container problem: Deleting a collection of object, thereby leaving its members orphaned and inaccessible. More likely in C++ than Java. CEN 5076 Class 10 - 11/07

  40. All-paths Partial Ordering of Coverage criteria Rapps and Weyuker ‘85 All-du-paths All-uses All-c-uses/ Some-p-uses All-p-uses/ Some-c-uses All-p-uses All-defs All-edges (branch coverage) A B All-nodes (statement coverage) Criteria A subsumes B CEN 5076 Class 10 - 11/07

  41. Data-Flow Adequacy Criteria cont • The subsume relation is actually a comparison of adequacy criteria according to the severity of the testing method. Does not say anything about the ability to detect faults. CEN 5076 Class 10 - 11/07

  42. Review Exam 2 • Definitions: software testing, testability, test data selection criteria, test data adequacy criteria, specification-based testing, implementation-based testing. • Know the following axioms: antiextensionality, general multiple change, antidecomposition, anticomposition. • Examples of tests that are valid but not reliable and vice versa. The formulas will be provided. CEN 5076 Class 10 - 11/07

  43. Review Exam 2 cont • Software metrics • Be able to describe how software metrics are used in testing • Know at least (3) pre-OO metrics and (3) OO metrics • Specification-based testing: • Types of coverage criteria for logic-based systems, state machines, algebraic systems. • Be able to draw a state machine for a given problem and generate test cases to provide specified coverage e.g., all transition k-tuples, all transitions, all states, all events, all actions. CEN 5076 Class 10 - 11/07

  44. Review Exam 2 cont • Implementation-based testing: • Control flow: statement coverage, branch coverage, path coverage, basis path coverage, multiple condition coverage (MCC). • Data flow: all defs, all def-use-paths, all uses. • Be able to draw a flow graph for a method; identify def-use pairs for the variables in a method. • Understand the concept of infeasible coverage criteria. CEN 5076 Class 10 - 11/07

  45. Review Exam 2 cont • Implementation-based testing: • Given a specific control flow or data flow criteria be able to generate a test set to provide adequate coverage. E.g., be able to identify the set of basis paths in a flow graph for a method and generate test cases to cover all the basis paths. CEN 5076 Class 10 - 11/07

More Related