Software Testing

Software Testing Chapter 12: Subsystems Nagy Akos – Nick Baetens – Kevin Buyl – Matthias De Cock - Dieter De Hen

Overview • Subsystems • What are subsystems • Types of subsystems • Why test them • Subsystem Test Design Patterns • Class Association Test • Controlled Exception Test • Round-trip Scenario Test • Mode Machine Test

Subsystems • What are subsystems • Any testable collection of classes, objects, components and modules • Characteristics • Executable and testable as a whole • Whatever its size, it must be executable to be testable • Candidate for responsibility testing • Has parts that can be tested in isolation • Implements a cohesive set of responsibilities • Doesn’t provide all the functionality of the application under test

Subsystems • Types of subsystems • Small clusters • Several tightly coupled classes • Testing the classes in isolation sometimes impractical(e.g. cyclic dependencies) • Large clusters • Group of related classes or clusters • Pattern participants • Implementation of design pattern tested as a subsystem (e.g. Model-View-Controller implementation)

Subsystems • Types of subsystems (continued) • Build group • Collection of software components processed by the linkage editor to generate an executable • Task group • Collection of build groups that is linked as an executable process or that generate processes that do something interesting • Family of Ada 95 packages • Developer subclasses

Subsystems • Why test them • It is a precondition for testing at system scope • It is a practical necessity • Support for internal release during incremental, iterative development • Use cases can model requirements for any subsystem with public interface • Some classes and other components have been bundled for historical reasons and must be tested as a whole

Subsystems • Why test them (continued) • Some subsystems must work to receive additional funding • Testing budget or schedule is limited or reduced • Dependability of some subsystems is higher than of others • The target environment is unique or different or must be simulated apart from the rest of the application

Subsystem Test Design Pattern • Subsystem test plan • What features and capabilities should be tested? • How should the test plan and test suites be organized? • Structure and responsibilities determine testing approach • Key problem: • Developing a testable model of aggregate behavior > Solution: test design patterns

Class Association Test • Intent • Design a test suite to verify the implementation of required associations among classes • Context • Applicable when the SUT must correctly implement the relationships represented in a class diagram • Reveal bugs in implementation of associations

Class Association Test • Fault Model • Kinds of faults • Incorrect multiplicity • Legal combination rejected • Illegal combination accepted • Update anomaly • The link that associates several instances of one class with a single instance of another class necessarily carries some redundant information • Delete anomaly • What happens to owned objects if the owning object is deleted? (depends on design/requirements) • Missing link • Not all links are created or some links become corrupted • Wrong link • All links created but some links are incorrect or inconsistent

Class Association Test • Fault Model • Faults can be reached by exercising the code that implements the association constraint • This code lies on a path that will be executed when boundary values of the constraints are input (see Section 10.2.4, Domain Testing Model)

Class Association Test • Strategy • Test model • Multiplicity test sets • Association A:B with multiplicity (e.g. 0..1, 0..n, 1..1, 1..n, …) has 2 boundary values • Binary association A:B and B:A has 4 boundary values • Apply 1x1 test strategy (with on points, off points and in points) and cast results in domain matrix (see Section 10.2.4, Domain Testing Models) • Testing for update/delete bugs • Wrong and missing links

Class Association Test • Automation • Increase testing by interleaving anomaly tests with max‐value multiplicity tests • SUT can be exercised with an API driver interface • GUI may provide sufficient control to exercise associations • DBMS: database loader utility to setup test cases

Class Association Test • Entry criteria • Each class in the scope of the model has passed an alpha‐omega cycle • Keyed collections are typically used to implement associations. These classes should be individually tested with Quasi‐modal Class Test • Exit criteria • Each multiplicity on point test passes • Each multiplicity off point test passes • A deletion anomaly test passes • An update anomaly test passes • All tests for mutual exclusion pass

Class Association Test • Consequences • N associations  at most 8n basic multiplicity test cases (2 for each min and max boundary on A:B association and 2 for each min and max boundary on B:A association) • Typical system: several 100 associations  several 1000 test cases • Class association test requires a model of the class associations in the system under test • Difficult for maximal value testing if number of database instances is large

Controlled Exception Test • Intent • Design a test suite that will exercise exception handlers and provide a systematic way to generate all exceptions • Fault model • Safe exception handling = difficult! • Several pitfalls • Improperly passed exceptions • Exception is ignored and results in abrupt task end • Ignored exceptions resulting in mysterious failures • Looping exception structure (retry without limitation) • Wrong exception is thrown/wrong catch block activated • …

Controlled Exception Test • Test Model • To test properly exceptions must be triggered  Can be difficult!

Controlled Exception Test • Test Model • To test properly exceptions must be triggered  Can be difficult! • 3 ways for exception testing • Exception activation: Trough input and state values supplied by test cases • Exception inducement: By directly manipulating the implementation under test • Exception simulation: By using controllable wrappers for virtual machine services

Controlled Exception Test • Automation • Stubs to simulate application specific exceptions • Command-line scripts to generate anomalous conditions which (hopefully) trigger exceptions • Wrapper classes

Controlled Exception Test • Consequences • Design of exception-handling test suite requires technical understanding of which exceptions can be raised • Target environment exceptions (e.g. Java Class Libraries’ exceptions)  Well documented • Applications-specific exceptions  Typically undocumented • Testing exceptions in an operational system may be difficult!

Round-trip Scenario Test • Intent • Develops path set with minimal branch & loop coverage for UML Sequence Diagrams by extracting a control flow model • Fault Model • Lots of different round-trip scenarios • Scenario: Path trough sequence diagram • Lots of scenarios = lots of possible bugs • Incorrect or missing output • Missing functions or features • Incorrect messages passed • Deadlocks • …

Round-trip Scenario Test • Test Model • Sequence diagrams are not ideal for testing! • Transform sequence diagram to flow graph • Flow graphs are highly testable • Node: Chunk of unconditional processing • Arrows: Show which node may follow another

Test Procedure – Step 1 1. Convert Sequence Diagram into Flow Graph • Each consecutive group of messages becomes a segment (block) • Each bracketed condition becomes a decision (hexagon) • Each bracketed loop condition also becomes a decision (hexagon) – The position of loop predicates is important here (‘while’ & ’for’ loops vs. ‘until’ loops)  Often this conversion will already reveal some ambiguities and omissions!

Test Procedure – Step 2 2. Identify Paths to Test • No loops and branches? • Any scenario will reach all segments • Loops or braches? • Often astronomical number of possible paths  Running over all possible paths is in most cases infeasible!

Test Procedure – Step 2 Solution: • Take every branch at least once • Use every loop in at least 3 ways • Bypass the loop (or do minimum # iterations) • Do 1 iteration • Do the maximum number of iterations • Advanced tools are not necessary!  Use colored markers to draw the paths

Test Procedure – Step 2 Procedure: • Choose the longest or most complex entry-exit path and color it • Switch to another color and color another entry-exit path • Repeat step 2 until all decision branches are colored • Make a table with a row for each path listing the conditions that must hold for this path

Test Procedure – Step 2 Procedure - Continued • Pick any path that leads to the loop condition • Copy the paths up to the loop entry in the table and append the predicate conditions to allow one iteration, append the exit path • Pick another path to the loop entry and append the predicate conditions to allow the maximum number of iterations, append the exit path

Test Procedure – Step 2

Test Procedure – Step 3 3. Identify Special Cases Sequence diagrams do not include: • Exception handling paths • Covered by Controlled Exception Test • Bindings to polymorphic servers • The number of paths grows exponentially when a polymorphic server is several subclasses deep

Test Procedure – Step 3 • Reduce the risks for polymorphic servers! • Identify circumstances that must not occur and involve polymorphic objects without direct interface • Identify constraints that must always be observed and involve polymorphic objects without direct interface • Develop test cases for the “must” and “must not” scenarios

Test Procedure – Step 4 4. Identify inputs and states • To identify test case values just look at the path conditions for each path • For each path pick a set of input and state values that satisfies the particular path conditions • In most cases there will exist many different sets of values that will satisfy the path conditions! • Finally determine the expected results for each test case

Round-trip Scenario Test • Consequences • Accurate sequence diagram required! • Not everything is modeled  Additional tests required • Round-trip path heuristic far more effective than poking around • Expansion of paths resulting from polymorphic bindings not addressed.

Mode Machine Test • Intent • Develop a test suite for subsystem scope behavior by analyzing component behavior • Modal subsystem • Dominant control object • Implements a state pattern • Behavior is sequentially constrained • The same sequence of external events does not always produce the same response > Example: vending machines

Mode Machine Test • A correctly functioning subsystem must • Accept legal events • Reject events that are illegal • Produce a correct resultant state • Produce a correct action for each test event • Five kinds of control faults • Missing transition • Incorrect transition • Invalid resultant state • Production of a corrupt state • Sneak Path

Mode Machine Test • Creating a Test Model • Identify the dominant control object (DCO) and develop its state model • Identify the external events and actions • Map the external events onto the DCO • Develop a modal test suite for this state model

Mode Machine Test • N+ Test Suite • Conformance tests • Regular tests • Guard expansion • Boolean outcomes • Iteration expansion • Each of the items gets selected once • Implicit transition expansion • Events accepted in every state • Sneak path analysis • Illegal sequences

Mode Machine Test • Entry Criteria • Each component has met the exit criteria for the applicable test pattern • Each cluster has minimal operability requirements • Exit Criteria • There is a test for every root-to-leaf path in the expanded transition tree and each sneak path pair

Software Testing