Software Testing

Software Testing Chapter 11: Reusable Components Nagy Akos – Nick Baetens – Kevin Buyl – Matthias De Cock - Dieter De Hen

Reusable Components • Testing and Reuse • The Role Of Testing in Reuse • Reusing Test Suites • Test Design Patterns • Abstract Class Test • Generic Class Test • New Framework Test • Popular Framework Test

Testing and Reuse • Reusable packages • A single specification • Multiple implementations • Similar high-level behavior • Different low-level behavior • Example: Queue • Bounded vs Unbounded • Sequential vs Concurrent • …

Testing and Reuse • Importance of reuse • Solution to software development problems • High development time • High development cost • Frequent failures • Low maintainability • Low adaptability • Error Reduction Ratios (C++) • Design errors – 49:1 • Coding errors – 26:1 • Rework and debug time – 100:1

Testing and Reuse • Importance of testing • Leads to more reliable components • Often improves component design • Shows inelegant or difficult-to-use components • Tangible evidence of dependability

Testing and Reuse • Testing and reuse • Reuse is not limited to just the code • Producer test suites • Rerun by the consumer • Establishes reusability • Consumer test suites • Specifically tailored to an individual user • Can be incorporated by the producer

Class Instances • Abstract Class Test • abstract class instance • a subclass providing implementation • Generic Class Test • generic class instantiation • class that results if type is provided in a declaration/definition

Abstract Class Test • Intent • Develop a test suite for an abstract class • Context • Tested by developing fully implemented subclass • Test suite for the entire hierarchy • Fault model • Fault hazards • Design errors • Simple coding errors

Abstract Class Test • Strategy • Subclass must be developed • Specification-based tests for flattened subclass • Test cases for each implemented method (by using appropriate method and class scope test design patterns) • Entry criteria • May be tested when it is compiled • Exit criteria • Method scope branch coverage is the minimum

Abstract Class Test • Consequences • The test subclass and its test suite may reveal design errors • The test suite may be used as documentation for the consumers of the class

Class Instances • Abstract Class Test • abstract class instance • a subclass providing implementation • Generic Class Test • generic class instantiation • class that results if type is provided in a declaration/definition

Generic Class Test • Intent • Develop a test suite for a class that requires type parameter • Context • To produce an object we need • Generic class • Type argument • Class behaviour depends on • Implementation • Type argument • Testing • Which/how many type arguments

Generic Class Test • Fault model • We distinguish two kinds of faults • Type-dependant faults • Type-specific faults • Compiler accepts/rejects a given type or class

Generic Class Test • Type-dependant faults • Unintended interaction occurs between the implementation and some accepted class (not uniform behaviour for all types) • Combination of accepted type arguments may cause faults • Examples • Range and precision of scalar types in arguments and returned values • Use of static and class variables • ...

Generic Class Test • Type-specific faults • Unintended interaction occurs between the implementation and some accepted class • Some generic methods may be inconsistent with special-case type argument • Combination of accepted type arguments may cause faults • Support only a subset of accepted classes(may indicate poor design)

Generic Class Test • Strategy • Single type parameter • Test suite has 3 parts • Type-dependent tests • Type-specific tests • Type-overloading tests • Multiple type parameters • Exercise combination of types • As many as time permits • Automation • Test driver as a generic class • Rerun test suite for substituted type parameters

Generic Class Test • Entry criteria • Accepted classes should pass their test suite • Exit criteria • Attain 100% branch coverage on every tested instantiation of the CUT • Consequences • Passing for several type parameters → no guarantee to pass for other type instantiation • Release the test suite with the generic class

Framework Testing • New Framework Test • Applies to initial and early versions • Design a test suite and the test instantiation from the generic requirements for a new framework • Popular Framework Test • Applies to a framework that becomes widely used • Design a test suite and the test instantiation from a framework that becomes widely used. • Cover new features and perform compatibility testing

Framework Testing • Don’t need to choose • Emphasis in testing evolves • Availability of an application instantiation of the FW and the most probable kind of bugs dictate the differences New Framework Test Popular Framework Test Widespread use Many application-specific instantiations 1st Release Few, if any, instantiations

To Keep In Mind … • Extended Use Case: page 281 • Use case: describes interactions • Not sufficient for system test design • Extra info: relative frequency, sequential dependencies, domain of participating variables • Extended Use Case Test: page 722 • Develop an application system test suite by modeling essential capabilities as extended use cases.

To Keep In Mind … • Class Association Test: page 569 • Design a test suite to verify the implementation of required associations among classes • Covered in CRUD: page 732 • Verify that all basic operations are exercised for each problem domain object in the SUT • Test Oracle • Produces the results expected for a test case

New FW Test: Context • FW Development: • Requirements model without antecedent application • Generalizedfromexisting system • FW responsibilities and the Implementation UT will be new

New FW Test: Fault Model • General promises of a FW: • Complete and consistent representation of the salient aspects of the problem domain as classes and associations • Provide basic capabilities to create, read, update, delete objects of these classes and the associations among them • Enforce sequential control necessary for required behavior

New FW Test: Fault Model • Each individual component is at risk for many kinds of bugs • Design Omissions: essential behavior or representation is missing • Representational integrity bugs: allow association constraints to be broken • Control bugs: FW typically use inverted control strategy • Infrastructure bugs: typically complex code

New FW Test: Test Model • Test suite: FW analysis and design models • DemoApp: • Testable instantiation • Should provide minimal implementation of each use case identified for the entire FW • Test design: • Extended Use Case Test • Class Association Test • Control strategy: model as state machine and develop test suite with N+ Coverage

New FW Test: Test Model • What if there is predecessor system? • Use it as an oracle, figure 11.2 • DemoApp must implement features (nearly) the same as predecessor system • Salvage some of the test cases of PS • Test design from PS will provide some insights for test design of DemoApp

New FW Test: Entry Criteria • Testable system scope model • All component classes and subsystems should have passed individually adequate test suites • Consider Risks if you skip/reduce 2 • System scope test harness • Test environment installed, tested and stabilized

New FW Test: Exit Criteria • At least one test for each row in the use case’s operational relation, meeting the Extended Use Case Test exit criteria • At least one minimal test of every association in the class model, meeting the Class Association Test exit criteria • CRUD Coverage • Control model coverage: transaction coverage for a state machine or branch coverage on all sequence diagrams

New FW Test: Consequences • Cost of testing is directly proportional to number of use cases and class associations. • Producers should monitor and analyze the results of application instantiations • Release FW after components and subsystems tests pass? • Let the consumer test the FW • Interleaved debugging • Some features remain untested?

Framework Testing • New Framework Test: • Applies to initial and early versions • Design a test suite and the test instantiation from the generic requirements for a new framework • Popular Framework Test: • Applies to a framework that becomes widely used • Design a test suite and the test instantiation from a framework that becomes widely used. • Cover new features and perform compatibility testing

Popular FW Test: Context • Test suite is needed to validate the new features and to identify potential compatibility problems • FW itself is also at risk for faults of New FW Test, fewer as FW becomes mature • Failures of three new fault classes

Popular FW Test: Fault Model • Feature Interaction: behavior or side effect produced when several features are used together. => Desirable/Undesirable?? • Compatibility bug: revision to the FW breaks previously working feature • Latent bug: bug that was dormant in the FW, new instantiation reaches/triggers failures

Popular FW Test: Test Model • Feature Interaction Tests • FUT may exercise some new FI • Additional test cases from anti-requirements • Regression Tests • Compatibility Tests • Can use existing instantiations with test suites • Next increment of FW is instantiated with an application that was stabilized for previous version • Latent bugs => gain info about shift in usage • New Features: use EUCT and CAT

Popular FW Test: Entry Criteria • New FW Test Entry Criteria + • Optional: Build at least 1 actual consumer instantiation of the current release • Conduct an analysis of the new/changed features

Popular FW Test: Exit Criteria • New FW Test Exit Criteria

Popular FW Test: Consequences • New FW Test Consequences • Inform users about obsolete/unsupported features • As the FW begins to stabilize: • Rate of test decay will decrease • What happens if the existing library of test cases has revealed all bugs it can? => Pesticide Paradox

Software Testing