Testing Interfaces in Multi-Program Systems

Testing Interfaces in Multi-Program Systems ECEN5043 Software Engineering of Multiprogram Systems University of Colorado, Boulder

Topics • Computational Models • Implications of nondeterminism • Model Testing • Class Testing extended • UML for state diagrams, concurrency, components • Timing and Time-outs -- UML and testing • Threads & synchronization ECEN5043 University of Colorado Testing Interfaces

Topics (cont. 2) • Dynamic Testing • Definition of component • Component Interactions • Identifying -- nonprimitive classes • Specifying • Testing Interactions • Collections • Collaborations • One-way pass along • Interaction between Testing & Design ECEN5043 University of Colorado Testing Interfaces

Topics (cont. 3) • Sampling Test Cases -- OATS revisited • Testing Off-The-Shelf Components • Protocol Testing • Test Patterns -- McGregor & Sykes; Binder • Testing Interactions at the System Level • Testing Exceptions ECEN5043 University of Colorado Testing Interfaces

Computational Models • Sequential processing -- default • Model testing: straightforward inspections • Dynamic testing: see course #1 • Concurrency • Multiple things that appear to be happening at the same time. • Central issue -- nondeterminism of interleaved statement execution • Preliminary thoughts on implications for testing? ECEN5043 University of Colorado Testing Interfaces

Computational Models (cont.) • Distributed • Characterized by: multiple processes to support a flexible architecture in which the number of participating objects can change • Can be distributed across multiple processes on the same machine -- we’re postponing this to course #3 when we focus on distributed architectures (on one or more processors). ECEN5043 University of Colorado Testing Interfaces

Implications of Nondeterminism • Nondeterminism • Difficult (impossible) to replicate a test run when the software contains multiple threads or concurrent processes • Ordering determined by the scheduler • Changes in programs NOT associated with the system under test can affect the order in which threads of the system under test are executed • If a failure is encountered and the defect is isolated and repaired, the fact that the program passes test execution is ... ? ECEN5043 University of Colorado Testing Interfaces

Nondeterminism and Static Testing • More thorough model testing at the class level • Design (and review) should investigate whether there is • a need • an appropriate provision for synchronization • During model testing, • Ensure test cases require execution of critical sections • Consider impact of different orderings with respect to • starvation • mutual exclusion • deadlock • synchronization ECEN5043 University of Colorado Testing Interfaces

But how?? • Three kinds of model testing • Is the code for one process correct by itself (are the sequential statements correct) assuming its interactions are correct • If a propos, inspect with multiple copies of same process in parallel • For shared data, different processes, have copies of each • For the critical sections • “What wouldstarvation require in this context?” Is it possible? • Similarly for mutual exclusion, deadlock,and necessarysynchronization ECEN5043 University of Colorado Testing Interfaces

Testing Multi-program Systems • Methods -- review the class testing described in previous course • state transition coverage • code coverage • pre & post condition coverage • Extend notion of state diagram to that of interacting components (programs/processes) • Extend state transition coverage similarly • Recall notion of state transitionpairscoverage ECEN5043 University of Colorado Testing Interfaces

Sample State Transition Diagram - Gomaa ECEN5043 University of Colorado Testing Interfaces

Time-Outs for Testing • It’s possible that a process/program will make a request of another process that never responds • The requester must be able to detect that and abandon the request • Design should explicitly define correct behavior for • when the request is answered • and also when it is not • the behavior may be very different in those two cases. • Model testing should verify these • Dynamic testing should also. ECEN5043 University of Colorado Testing Interfaces

Expressing Time on Interaction Diagrams • Interaction diagrams are collaboration diagrams and sequence diagrams • time expression -- resolves to a relative or absolute value of time when evaluated • timing mark -- time-related name or label on a message • timing constraint -- condition that must be satisfied with regard to time. • usually contains a time expression • may also contain a timing mark ECEN5043 University of Colorado Testing Interfaces

Sample from Collaboration Diagram Reminder: A sequence or collaboration diagram focuses on the time ordering of the messages that go back and forth between objects. v:validateLogin(userID, password) ;Login Page ;Account {v.executionTime < 3 sec} v is the timing mark. The timing mark makes it easier to refer to that message elsewhere on the sequence diagram. The expression between { } is the constraint. ECEN5043 University of Colorado Testing Interfaces

Types of Timing • balking message -- synchronous message but the sending object gives up on the message if the receiving object is not ready to accept it. • timing constraint {wait = 0} • timeout message-- synchronous message but the sender waits only for a specified period for the receiver to get ready to accept the message • timing constraint {wait = 50 ms} ECEN5043 University of Colorado Testing Interfaces

Threads & Concurrent Processes • Working definition: a thread is a unit of computation that can be scheduled • During design and design review, should explicitly consider the number of threads • Increasing it • Can simplify certain algorithms/techniques • Increase the risk of sequencing problems • Reducing it • Reduces sequencing problems • Makes software more rigid and maybe inefficient ECEN5043 University of Colorado Testing Interfaces

Concurrent Processes, Synchronization, & Notation • During design inspections, verify presence of the helpful clauses in the UML state diagram • Some languages provide a language keyword to automatically add the mechanism to prevent simultaneous access -- be sure it works as expected • In Java, the keyword synchronized is used on the signature of a method to specify the need for a synchronization mechanism (see later slide) • C++ requires explicit structures that each individual developer must construct • In C++, creation of an instance of a monitor object indicates the location at which synchronization is needed ECEN5043 University of Colorado Testing Interfaces

Java sample public synchronized int get( ) { while (available == false) { try { wait ( ); } catch (InterruptedException e) { } } available = false; notifyAll( ); return contents; } public synchronized void put (int value) { while (available == true) { try { wait ( ); } catch (InterruptedException e) { } } contents = value; available = true; notify ( ); } Example from The Java Tutorial, Addison Wesley, Lesson 15, Threads of Control, pp. 285-325. ECEN5043 University of Colorado Testing Interfaces

Dynamic TestingConcerns ECEN5043 University of Colorado Testing Interfaces

Collections & Collaborators Collections: Classes that maintain associations with instances of other classes but never actually interact with those instances Hold the objects they are handed and return them in specific orders or find them based on specific criteria Test with techniques for primitive classes Collaborators: may be addressed directly (name) or by a pointer or a reference dynamic type of object may be different from static type associated Pointers & references are polymorphic Bound to an instance of any number of classes pre & post conditions refer to states or values of collaborating objects ECEN5043 University of Colorado Testing Interfaces

Testing Collection Classes • Test driver • create instances passed as parameters in messages to collection being tested • Ensure those instances are correctly built into and removed from the collection. • Address capacity limitations • Test behavior under circumstances where it cannot allocate memory to add the new item to itself • If defensive, use negative tests also • If the collection class uses a structure (array, etc.), include test cases to fill it completely and attempt to add one more. • Test sequences of operations ECEN5043 University of Colorado Testing Interfaces

Collaborators at Program Interfaces • Collaborating class • Post condition in a class’ interface refers to the state of an instance and/or specifies that some attribute of that object is used or modified • We assume operations are specified by pre-, post condition and class invariants • Need to know if defensive design or design by contract • Changes how senders and receivers interact ECEN5043 University of Colorado Testing Interfaces

Nondeterminism & Dynamic Testing • There is no perfect way to do this but the following will help: • Begin dynamic testing with as “clean” a machine as possible • document the environment • identify apps that must run while items-under-test are running • add basic set of typical unrelated apps • For each test case (set of programs/ thread-generating processes) • Describe any modifications made to this “standard environment” • Note the order in which processes are started • Include a debugger that allows verification of order in which threads are created, executed, and deleted ECEN5043 University of Colorado Testing Interfaces

Orthogonal array testing applied to OO • Orthogonal array testing is a sampling technique to limit the explosion of test cases • Define pair-wise combinations of a set of interacting objects because most faults result from interactions due to two-way interactions • An orthogonal array is an array of values in which each column represents a factor. • A factor is a variable in an experiment; in our case, a specific class family in the software sys • Each variable takes on a certain set of values called levels; in our case, a specific class • Each cell has an instance of the class, that is, a specific state of an object ECEN5043 University of Colorado Testing Interfaces

OATS – orthogonal array testing system • The factors (columns) are combined pair-wise rather than representing all possible combinations of the levels for the factors. • Suppose we have three factors – A, B, C – and each has three levels – 1, 2, and 3 • How many possible combinations of these values are there? • How many pair-wise combinations? That is, how many combinations where a given level appears exactly twice? factor=class family; level=class; cell=instance ECEN5043 University of Colorado Testing Interfaces

a specific state a specific state a specific state ECEN5043 University of Colorado Testing Interfaces

Excerpt from Fig. 6.16, M&S, 21 x 37 ECEN5043 University of Colorado Testing Interfaces

OATS – uses a balanced design • Every level of a factor appears exactly the same number of times as every other level of that factor • Think of the rows of a table as test cases (In example, 18 of the possible 27 are not being conducted. • This is a systematic way of reducing the number of test cases. • If we decide to add more later, we know which ones have not been run. • Also logical – most errors are between pairs of objects rather than among several objects ECEN5043 University of Colorado Testing Interfaces

OATS adequacy criteria • Ability to vary how completely the software under test is covered • Exhaustive – all combinations of all factors • Minimal – only interactions between the base classes from each hierarchy are tested • Random – tester haphazardly selects cases from several of the classes; not statistically random • Representative – uniform sample ensures every class is tested to some level • Weighted representative – adds cases to the representative approach based on relative importance or risk associated with the class ECEN5043 University of Colorado Testing Interfaces

Testing off-the-shelf components – • Investigate the limits of whatever specification for it that is available • Create one if none is available • Developers will need it in order to properly use the component • Begin with extreme boundary values -- even incorrect • Do stress tests next • If worth continuing • Treat as a class interacting with rest of your product • There is usually a main class that presents itself to the user or other products. Base tests on that class. ECEN5043 University of Colorado Testing Interfaces

Protocol testing – • Protocols: • Sequences of method invocations* • Identify them by combining a method whose post condition enables the precondition of another method • Easier to see from a state diagram than from written pre- and post conditions. • Investigates whether the implementation of a class satisfies its specification. • Abstract to state diagram with interacting components ECEN5043 University of Colorado Testing Interfaces

Protocol Interaction Test Suite • Consider as the “class under test” the component’s class that presents an interface to the other components • Each protocol represents a life cycle for objects from that class in combination with instances of classes in the interacting components. • Begin with initial states of a given two objects as denoted in the state diagrams of the two corresponding classes • A sequence of states for each object • Ending with the terminal states • A test case takes the two objects through one complete sequence of methods ECEN5043 University of Colorado Testing Interfaces

Protocol between two components • The sequence of messages exchanged between them. • A component should advertise • the services it provides (through its interface) • what it requires (through its preconditions) • Should also describe the sequence in which these interactions are expected • One set of test cases defined from the protocols should exercise the integration of two components through the complete protocol. ECEN5043 University of Colorado Testing Interfaces

Example of protocol between two components • Note: can be described using sequence diagram • “user” component sends asynchronous message to the “hardware” component to start the action • Eventually hw that services multiple clients sends an asynchronous message back that the action has been started • “User” wants hw to stop -- similar exchange • No notational convention but these four occur as a group although 2 for one component and 2 for the other • Tester class plays role of the other component in a protocol • Java -- Tester object can pass itself as a parameter to the component under test and then invoke the services specified in the protocol ECEN5043 University of Colorado Testing Interfaces

Example: Timer Protocol – diagram on next slide • Trace through the state diagram to create a life cycle test. • Create the object • Send one or more attach (...) messages • Followed by enable ( ), disable ( ), and • delete ( ) • This is a simple example. • A life cycle test case provides a test of the object in the ways that it will interact with its client objects • When the object is one that interfaces with client objects in another program, this provides a realistic test sequence. ECEN5043 University of Colorado Testing Interfaces

McGregor & Sykes Timer isEnabled( ) limit( ) size( ) Timing attach(TimerObserver) notify()/observers->forAll(notify()) enable( ) Disabled Enabled disable( ) notify( ) detach(TimerObserver) delete ( ) ECEN5043 University of Colorado Testing Interfaces

Test Patterns – • Test patterns are design patterns for test software. • The pattern description explains the context within which the pattern should be considered • Provides a set of forces that guide a trade-off analysis • Explains how to construct the objects. • When developer uses a certain design pattern, tester can use a certain test pattern to structure the test code. • Binder gives several examples in some detail. ECEN5043 University of Colorado Testing Interfaces

Test Design Patterns Presented in Binder • Test Design Patterns are for designing test suites • Robert Binder presents test design patterns for over 30 architectures ECEN5043 University of Colorado Testing Interfaces

Exceptions • An alternative route for returning from a method that does not necessarily move to the next statement after the method is invoked. • The exceptional return value is an object and can be arbitrarily complex • Where the exception is thrown varies based on the depth of the aggregation hierarchy ECEN5043 University of Colorado Testing Interfaces

Testing Exceptions -- class • Class level testing: verify that each of the methods does throw the exceptions it claims to throw in the appropriate circumstance. • Coverage criteria – class throws every exception contained in the specifications of the methods • At least one test case per exception • Handled in normal class testing because each potential throw should be a postcondition clause ECEN5043 University of Colorado Testing Interfaces

Role of test driver of exceptions -- class • Test driver sets up conditions to create exception event. • Provides try block • inside that is a stimulus that invokes the method that throws the specific exception. • Exception caught by test driver • verifies correct exception • Exceptions are objects and belong to classes • catch statements can use typing system to verify that it is of the correct typeS ECEN5043 University of Colorado Testing Interfaces

Testing Exceptions – integration • Integration level -- test they are being • thrown at correct time • caught at the correct place • During inspection of system-level design model • every user-defined exception instantiated • should be traced to an object that will catch the exception • using the sequence diagram for the scenario that throws the exception • Exception handling can be model tested early ECEN5043 University of Colorado Testing Interfaces

Testing interactions at system level • Components can be so complex it’s easier to test them in the context of the application itself • Which interactions are tested at system level? • Those that can also be verified at the system level • That is, can see the direct results of the tests • There must be a direct relationship between user interface and the ability to view test results • By these criteria, many multi-program interactions must be tested as class vs. program or class vs. class because program vs. program interactions may not be visible at the system level. ECEN5043 University of Colorado Testing Interfaces

Books • McGregor & Sykes, A Practical Guide to Testing Object-Oriented Software, Addison Wesley, 2001. ISBN 0-201-32564-0 • Scott, Kendall, UML Explained, Addison Wesley, 2001. ISBN 0-201-72182-1 • Binder, Robert V., Testing Object-Oriented Systems: Models, Patterns, and Tools, Addison Wesley 2000. ISBN 0-201-80938-9 ECEN5043 University of Colorado Testing Interfaces

Orthogonal array testing applied to OO • Orthogonal array testing is a sampling technique to limit the explosion of test cases • Define pair-wise combinations of a set of interacting objects because most faults result from interactions due to two-way interactions • An orthogonal array is an array of values in which each column represents a factor. ECEN5043 University of Colorado Testing Interfaces

Orthogonal array testing applied to OO • An orthogonal array is an array of values in which each column represents a factor. • A factoris • a variable in an experiment; in our case, a specific class family in the software system • Or states of that class family • Each variable takes on a certain set of values called levels (rows); in our case, specific classes or states of those classes • Each cell is an instance of the class, that is, a specific object or a particular state ECEN5043 University of Colorado Testing Interfaces

OATS – orthogonal array testing system • The factors (columns) are combined pair-wise rather than representing all possible combinations of the levels for the factors. • Suppose we have three factors – A, B, C – and each has three levels • How many possible combinations of these values are there? 3 x 3 x 3 • How many pair-wise combinations? That is, how many combinations where a given level appears exactly twice? factor=class family; level=class; cell=instance ECEN5043 University of Colorado Testing Interfaces

How many pair-wise combinations? • A B C • 1 1 1 3 • 2 1 2 2 • 3 1 3 1 • 4 2 1 2 • 5 2 2 1 • 6 2 3 3 • 7 3 1 1 • 8 3 2 3 • 9 3 3 2 ECEN5043 University of Colorado Testing Interfaces

Mapping the terms of OATS to OO sw A state of that nstance A state of that nstance A state of that nstance ECEN5043 University of Colorado Testing Interfaces

OATS – uses a balanceddesign • Every level of a factor appears exactly the same number of times as every other level of that factor • Think of the rows of a table as test cases • This is a systematic way of reducing the number of test cases. • If we decide to add more later, we know which ones have not been run. • Also logical – most errors are between pairs of objects rather than among several objects ECEN5043 University of Colorado Testing Interfaces

Testing Interfaces in Multi-Program Systems