Test Drivers and Stubs

CEN 5076 Class 12 – 11/21 Test Drivers and Stubs Test Drivers and Stubs Integration Testing System Testing

Test Drivers and Stubs • A stub is a simulator of a component. • You are unit testing component X which calls component Y. Y is either not available for testing at the moment or you do not want to be confused by Y’s actions. Solution - replace Y with a dummy component that uses the same interface but provides a “limited” set of outcomes for its processing. • A driver simulates the calling of a component and perhaps the entire environment under which the component is to be tested. CEN 5076 Class 12 - 11/21

Test Drivers and Stubs Driver cont. • A complete test driver, in addition to simulating the caller, provides the following services: • Test case initialization • Input simulation – i.e. simulated keystrokes and I/O responses. • Outcome comparison • Path instrumentation and verification – verifies that the path coverage was achieved. CEN 5076 Class 12 - 11/21

Test Drivers and Stubs Driver cont. • Reporting • Passage to next test - resetting of initial conditions and execution of the next test cases in the series. • Debugging support – Captures the environment (e.g., dumps) for crashes and/or passes control directly to debug package. CEN 5076 Class 12 - 11/21

Test Drivers • A test driver is a program that runs test cases and collects the results. [McGregor and Sykes] • Three general approaches to writing test drivers for OO programs: • Implement a function main() that can be compiled conditionally (with #define TEST) when compiling the member function definitions and then execute it. • Implement a static member function within the class that can be invoked to execute and collect the results for each test case. CEN 5076 Class 12 - 11/21

Test Drivers cont • Implement a separate class whose responsibility is to execute and collect the results for each test case. A main() function instantiates this class and sends it a message to run all the test cases. • We will use option 3. • The relationship between a class and a driver for testing it is easy to remember i.e., each class C has a tester class called CTester. • See Figure 5.12 on P. 185 for the strengths and weaknesses of the test driver designs. CEN 5076 Class 12 - 11/21

Test Drivers cont Strengths: • Easy to reuse code for driver to test a subclass. • Production code is as small as possible. • Production code is as fast as possible. Weaknesses: • New class must be created. • Care must be taken to reflect the changes in the class in the tests. In the project you should write tester classes for the classes in one subsystem using an automated tool. CEN 5076 Class 12 - 11/21

Test Drivers cont Test Driver Requirements: • Main purpose of the test driver is to run executable test cases and to report the results of running them. • Test driver should have relatively simple design – requires time and effort to design and test. • Rely primarily on code reviews to test driver code. • Test driver must be easy to maintain and adapt to changes in the spec and tests. CEN 5076 Class 12 - 11/21

Tester Object Object under test #enum Result {Fail, TBD, Pass} + Tester(logFileName : String + ~Tester() +runAllSuites() +runFunctionSuite() +runStructuralSuite() +runInteractionSuite() +totalTally() : int +passTally() : int +failTally() : int +TBDTally() : int #runBaselineSuite() : Boolean #CUTinvariantHolds() : Boolean #logTestCaseStart(testID : String) #logTestCaseResult(result : Result) #logComment(comment : String) Class class under test Fig. A class model for requirements of a test class. P. 186 CEN 5076 Class 12 - 11/21

Test Drivers cont • Test cases are organized into suites based on their origin: • functional – identified from the spec • structural – identified from the code • interaction – if they test correct operation of sequences of events on the object e.g., pairs of input/output transitions. • Tally operations in the tester can be used to check how many test cases have passed so far. CEN 5076 Class 12 - 11/21

Test Drivers cont • A driver keeps log of test case execution and the results in the file whose name is specified at the time it is instantiated. • The logTestCaseStart(), logTestCaseResult(), and logComment() operations place information in the log file. • The runBaselineSuite() operation verifies the correctness of methods in the CUT that are used by the test driver in checking the results of test cases. CEN 5076 Class 12 - 11/21

Test Drivers cont • Accessor and modifier methods are usually tested as part of the baseline test suite for a class. • The CUTinvariantHolds() operation evaluates the invariant of the CUT using the state of the current object under test (OUT). • Tester class is abstract - can provide default implementation for operations common to all (concrete) testers. Operations include: • Logging test case results, CEN 5076 Class 12 - 11/21

Test Drivers cont • Measuring heap allocation, • Providing support for timing execution of individual test cases. See Java code on P. 204 – 210. Test case methods: • In the Tester class each test case is represented by a single method. • Name of method should reflect the test case in some way. CEN 5076 Class 12 - 11/21

Test Drivers cont Test case methods cont: • The responsibility of the test case method is to construct the input state for a test case e.g., by instantiating an OUT and any objects to be passed as parameters, and then by generating the events specified by the test case. • A test case method reports the status of the result – pass, fail, or TBD to indicate some action is needed to determine the result. • Verifies that the CUT’s invariant holds for the CUT CEN 5076 Class 12 - 11/21

Test Drivers cont Void tc_testCaseID(){ report start of test case; create a new instance I of the CUT using a factory method; put I into the correct input state; generate the prescribed sequence of events on the OUT; check post-conditions and class invariants against OUT; report results; disposeOUT () } Pseudocode for a typical test case method. CEN 5076 Class 12 - 11/21

Test Drivers cont OUT Factory Methods • Classes are tested by creating instances and checking their behaviors against a set of test cases. • A test case is applied to an instance of the CUT i.e., an object under test (OUT). • The main req. w.r.t. the OUT is that attributes be specified for the inputs to the test case so that preconditions associated with a test case to be applied are met. CEN 5076 Class 12 - 11/21

Test Drivers cont OUT Factory Methods cont • A tester interface includes a set of operations to construct instances of the CUT. • A tester should implement a factory method corresponding to each constructor defined in the CUT. • Test case methods use these factory methods to create the OUT instead of constructors in the CUT (reason – test cases for the CUT still apply to subclasses of the CUT, assume substitution principle is enforced). CEN 5076 Class 12 - 11/21

Test Drivers cont Baseline Testing • Test case methods use accessor operations in the process of checking postconditions. • Therefore the constructors, modifier methods, and accessor methods for the CUT must be tested before use to check other operations. • A baseline test suite is a set of test cases that tests the operations of the CUT that are needed for the other test cases to verify their outcomes. • See approaches on P. 194. CEN 5076 Class 12 - 11/21

Integration Testing • Detects faults by focusing on small groups of components e.g. methods, classes, packages. • Two or more components are integrated and tested, if no faults are revealed then additional components are added and tested. • Most recently added component is usually the one that triggers the most recently discovered fault. • Note the order in which components are integrated can influence the total testing effort i.e. the creation of stubs and drivers. CEN 5076 Class 12 - 11/21

Integration Testing cont • Several approaches: • Big bang testing • Bottom-up testing • Top-down testing • Ordering based on dependencies of components • Big bang – assumes that all components are first tested individually and then tested together as a single system. • Debugging is a problem, i.e., where is the fault? • Reduces the # of stubs and drivers required. Traditional CEN 5076 Class 12 - 11/21

Integration Testing cont • Bottom-up – individually test each component of the bottom layer and then integrate them with components of the next layer (assumes a hierarchical structure). • Drivers required to simulate the components of higher layers. • No test stubs are required. • Top-down – unit test the components of the top layer first and then integrate the components of the next layer down. • Stubs used to simulate the components of lower layers • Drivers not required. CEN 5076 Class 12 - 11/21

Integration Testing cont • Ordering based on dependencies – a dependency graph is created (similar to a class diagram), then an order is generated. The “leaf” components are tested first then the inner components. • Problem if graph contains cycles. • Uses stubs in the event of cycles. Papers: • Revisiting Strategies for Ordering Class Integration Testing in the Presence of Dependency Cycles. Briand et al. ISSRE 2001 • A Parameterized Cost Model to Order Classes for Class-based Testing of C++ Applications. Malloy, et al. ISSRE 2003. CEN 5076 Class 12 - 11/21

Integration Testing cont • Generating a class integration test order when there are no cycles in the class diagram (object Relation Diagram - ORD) is as follows: • Perform a reverse topological sort of the classes using their dependency graph. • Topological sorting of a graph G consists of numbering the vertices of G (labeling them with numbers 1, …, n, say) s.t. for each edge, the label associated with each source vertex is strictly lower than the label associated to the target vertex. CEN 5076 Class 12 - 11/21

Topological-Sort(G) 1. call DFS(G) to compute finishing time f[v] for each vertex v 2. as each vertex is finished, insert it onto the front of a linked list 3. return the linked list of vertices socks 17/18 undershorts 11/16 watch 9/10 shoes 13/14 pants 12/15 shirt 1/8 (u, v) means that garment u must be put on before garment v belt 6/7 tie 2/5 jacket 3/4 Graph topologically sorted socks undershorts pants shoes watch shirt belt tie jacket 2/5 3/4 17/18 11/16 12/15 13/14 9/10 1/8 6/7 CEN 5076 Class 12 - 11/21

Integration Testing cont • If there are cycles in the dependency graph then we need to create stubs. • For example can we test A or B in the following graph without the creation of a stub? A B • Approach: • break the cycle, • test A (or B) with a stub of B (or A), • then test B (or A) using A (or B). CEN 5076 Class 12 - 11/21

Integration Testing cont X A B Remove edge from A to B B is the client and A is the server A stubB Test A using a stub of B A B Test B (client) using A (server) Some researchers suggest testing A using B, since A was never tested using B! A B CEN 5076 Class 12 - 11/21

Integration Testing cont Algorithm to handle cycles: Input: An Object Relation Diagram (dependency graph) Output: Major and minor test orders for the classes in the ORD. Step 1: Transform the ORD into an acyclic digraph ORD’ Step 2: Produce a topological sorting for the ORD’. The test order produced is called the major test order. Step 3: For each non-unit cluster of ORD’ do steps 4 and 5. Step 4: For each cycle of the cluster, select and remove an edge(s) to break the cycle Step 5: Produce a topological sorting for the acyclic subgraph obtained in step 4. The test order produced is called the minor test order. CEN 5076 Class 12 - 11/21

Integration Testing cont • How do you break the cycles in the ORD? Kung et al. suggest temporally removing one or more association edges. • How do you find the major clusters? The major clusters are strongly connected components – Tarjans algorithm. • Strongly-Connected-Components (G) • Call DFS (G) to compute finishing times f[u] for each vertex u • Compute GT • Call DFS (GT) but in the main loop of DFS, consider the vertices in • order of decreasing f[u] (as computed in line 1) • Output the vertices of each tree in the depth-first forest fromed in line 3 • as a separately strongly connected component. CEN 5076 Class 12 - 11/21

Integration Testing – Faults [Binder] • Missing, overlapping or conflicting functions. • An incorrect or inconsistent data structure used for a file or database. • The wrong method called due to coding error or unexpected runtime binding. • The client sending a message that violates the server’s precondition. • Wrong object bound to message (polymorphic target). CEN 5076 Class 12 - 11/21

Integration Testing - Faults • Wrong parameters, or incorrect parameter values. • Failures due to incorrect memory management allocation/deallocation. • Incorrect usage of virtual machine, ORB, or OS services. • Intercomponent conflicts: thread X will crash when process Y is running. • Resource contention: the target environment cannot allocate resources required for a nominal load. CEN 5076 Class 12 - 11/21

Testing Class Hierarchies • Incremental changes made during inheritance when class D is derived from class C: • Add one or more new operations in the interface of D and possibly a new method in D to implement each new operation. • Change the specification or implementation of an operation declared by C in one or two ways: • Change in the specification of D for an operation declared in C. • Override in D a method in C that implements an operation inherited by D. Note that either or both of these can apply. CEN 5076 Class 12 - 11/21

Testing Class Hierarchies • Add into D one or more new instance variables to implement more state and/or attributes. • Change in the class invariant in D. • Assume that inheritance is used only in accordance with the substitution principle. • Taxonomy of OO classes classifies features in derived classes as: new attribute, recursive attribute, new non-virtual routine, recursive non-virtual routine, redefined non-virtual routine, new virtual routine, recursive virtual routine, redefined virtual routine. CEN 5076 Class 12 - 11/21

Testing Class Hierarchies • The incremental changes between C and its derived class D can be used to guide the identification of what needs to be tested in D. • If D is a subtype of C then all the specification-based test cases for C also apply to D. • Many of the implementation-based and interaction-based test cases also apply. • Inherited test cases refer to the test cases for a subclass that were identified for testing the base class. CEN 5076 Class 12 - 11/21

Testing Class Hierarchies • New operations in D – introduces new functionality and new code to test. Need to add spec and impl-based test cases. • Change in D’s spec for operation declared in C – add new spec-based test cases for the op. Developed based on any weakened preconditions. Apply the coverage criteria as appropriate. • Overriden method in D – reuse all inherited spec-based test cases for the method, new code requires new impl-based and interaction test cases. CEN 5076 Class 12 - 11/21

Testing Class Hierarchies • Add one or more attributes to D – testing is usually handled when new methods or overriden methods are tested. • Change in class invariant in D – means additional postconditions for every test case. Need to rerun all inherited test cases to verify that the new invariant hold. • A spec for a class describes what the class represents and what an instance of the class can do. A class spec includes the spec for each of the operations that can be performed by an instance of the class. CEN 5076 Class 12 - 11/21

Testing Class Hierarchies Testing abstract classes: • OO language semantics usually preclude instances of abstract classes from being created. • Approaches to testing features in an abstract class: • Define a concrete derived class for the purpose of testing. • Test the abstract class as part of testing the first concrete descendent. • Used guided inspection instead of execution based testing CEN 5076 Class 12 - 11/21

Testing Class Hierarchies • Another approach uses the Taxonomy of OO classes i.e., find a near minimal set of subsets (concrete derived classes) that covers all the recursive routines in the abstract class. [Clarke’03 – Phd Thesis (Malloy adviser)] • Algorithm uses the Greedy-Set-Cover algorithm [Cormen et al. 2001] • An instance (X, F) of the set-covering problem consists of a finite set X and a family F of subsets of X, s.t. every element of X belongs to at least on subset of F. CEN 5076 Class 12 - 11/21

Testing Class Hierarchies F – is the set of all concrete derived classes X – is the set of routines in the Abstract class. C – the set of concrete derived classes Greedy algorithm works by picking at each stage the set S that covers the greatest number of remaining elements that are uncovered. CEN 5076 Class 12 - 11/21

Test Drivers and Stubs