Software Testing 13. Integration

1. Software Testing13. Integration Daniel Riegelhaupt - Thomas De Vylder - Gertjan Loots - Tim Saquet - Philip De Smedt

2. 13. Integration • Integration in Object-oriented Development • Integration Patterns

3. 13.1.1 Definitions A Systemis composed out of Componentswho are themselvessystems of smaller components

4. 13.1.2 Integration testing is essential 3 basic kinds of testing needed to show that components are minimally interoperable: • Tests of individual components • Tests of the system resulting from the federation of components • Test of components interoperation Integrationtesting=a search for component faults that causeintercomponent failures

5. Integration testing is essential In contrast: • Component scope testing = search for intracomponent faults • System scope testing = search for faults that lead to a failure to meet a system scope responsibility

6. Integration testing is essential • Antidecomposition axiom: A test suite that achieves coverage at system scope will not necessarily result in the same coverage for its components In other words: even if the program works as a whole this doesn’t mean we test every component fully (some code might never be reached) • Anticomposition axiom: Adequate testing of components (in isolation) ≠ adequate testing of a system of components

7. Integration testing is essential Integrating most or all components in a system at the same time is usually problematic :  • Debugging is difficult (bug can be in any interface) • Last minute patches may be necessary (=> low quality, not adequately tested ) • No systematic testing can lead to undetected bugs Incremental testing !

8. Integration testing is essential • Advantages of incremental testing: • Interfaces are systematically exercised and proven to be stable before exercising unproven interfaces • Buggy interface code will be reached; if not triggered • Observe failures are most likely to come from most recently added component • Incremental testing requires discipline and commitment: choose a sequence of components by analyzing dependencies (=> architecture must be known or planned!). Next manage plan and automate

10. Integration testing takes place at all scopes and is repeated in each • development increment

11. Integration testing is essential Integration test plan must answer: • what are the major and minor development increments ? • which components interfaces / subsystems will be the focus of integration ? • In what sequence will components and their interfaces be exercised ? • Which stubs and drivers must be developed ? • Which test design patterns should be used ? • When can integration testing be considered complete ?

12. Integration testing is essential Key considerations: • Scope of an increment and dependencies of components • Testability ( # components   testability  ) • Component stability (postpone integration of volatile components, start with stable ones) • Test environment stability

13. 13.1.3 Dependency Analysis Class/cluster scope dependencies: • Composition and aggregation • Inheritance • Global variables … • pointers to object used as message parameters • … Intercomponent dependencies: • Explicit: • Message exchange • Remote procedure calls • Interprocess communication • Implicit: Not considered in Integration testing

14. Dependency Analysis Explicit dependencies: • often dictate the sequence of testing • typically correspond to interfaces that should be exercised by an integration TS

15. Dependency Analysis Make a dependency tree: • by hand (if small) • Use topological search ( Unix: tsort; commercial tools) • Can have multiple roots, cyclical dependencies Note that tools might have problems finding dependencies over: Networks, VM , OS …

18. 13.1.4 Faults Model • Configuration/version control problems • Missing, overlapping or conflicting functions • Problems with file or DB ( incorrect/ inconsistent representation; conflicting view /usage; data integrity ...) • Wrong method called due to coding or binding error • Client sends illegal message to server (violates pre condition or sequential constraints) • Wrong object bound to message • Wrong parameter or incorrect parameter value

19. Faults Model • Failures due to incorrect memory management (de)allocation • Incorrect usage of VM , ORB or OS services • Attempt by the IUT to use target environment services that are obsolete / not supported in the currant version/ release • Intercomponent conflicts (e.g. thread X will crash when process Y is running) • Resource contention: target env. can’t allocate resources needed for nominal load (e.g. Need to open 6 windows but IUT crashes when fifth is opened) IUT = implementation under test

20. 13. Integration • Integration in Object-oriented Development • Integration Patterns

21. Integration patterns • Big Bang Integration. Attempt to demonstrate system stability by testing all components at the same time. • Useful in a few limited circumstances. • Bottom-up Integration. Interleave component and integration testing by following usage dependencies. • Works well for small to medium systems.

22. Integration patterns • Top-down Integration. Interleave component and integration testing by following the application control hierarchy. • Works for almost any scope or architecture. • Collaboration Integration. Choose an order of integration according to collaborations and their dependencies. Exercise interfaces by testing one collaboration at a time. • Works for almost any scope or architecture.

23. Integration patterns • Backbone Integration.Combine Top-down Integration, Bottom-up integration, and Big Bang Integration to reach a stable system that will support iterative development. • Well suited to small to medium systems and especially useful for embedded applications. • Layer Integration. Incrementally exercise interfaces and components in a layered architecture. • Applies to layered architectures.

24. Integration patterns • Client/Server Integration. Exercise a loosely coupled network of components, all of which use a common server component. • Appropriate for client/server architectures. • Distributed Services Integration. Exercise a loosely coupled network of peer-to-peer components. • Appropriate for decentralized networks containing peer-to-peer nodes.

25. Integration patterns • High-frequency Integration. Develop and rerun an integrationtest suite test hourly, daily, or weekly. • Can be applied at almost any scope or architecture. For complete scope specificconsiderations Seebook 13.2.1

26. Big Bang Integration • Intent • Demonstrate stability with a few test runs • Context • SUT is stabilized and only a few components have been added since the last time • SUT is small and testable • Only feasible for monolithic systems • Tightly coupled system • Bad design

27. Big Bang Integration • Fault Model • Does the system run? Demonstrate that system testing can begin • Strategy • No incremental integration testing • Entire system is built and a test suite is applied

28. Big Bang Integration

29. Big Bang Integration • Entry Criteria • All components have passed component scope testing • The VM to be used is stable • No anomalies found that interfere with integration testing

30. Big Bang Integration • Exit Criteria • The test suite passes • Consequences • Many interface bugs: where to look? • Debugging is HARD: every component is suspect • Even if SUT passes: many hidden interface faults?

31. Big Bang Integration • Known Uses • Subsystem Big Bang Integration is used in Backbone Integration.

32. Bottom-Up Integration • Intent • Add components to the SUT in uses-dependency order. Components with fewest dependencies first. • Context • Stepwise verification between tightly coupled components • Test each component and then integrate it with already-tested components

33. Bottom-Up Integration • Context (2) • Useful when you can defer testing until all components have been developed • Components developed by teams • Early assessment of a component that must implement a critical requirement • Fault Model • See the Integration Faults

34. Bottom-Up Integration • Strategy • Test Model • Dependency tree • For each component: responsibility test suite • Do not only test responsibilities • Also exercise subcomponents • Test Procedure • Go from the leaves of the dep tree to the root • n levels -> n stages • Major steps (see next slide)

35. Bottom-Up Integration • Leaf-level components are coded. Test drivers are coded for the leaf-level components and the leaf level is tested by these drivers. • Components on the next higher level are coded. These components use object tested in a previous stage. Iterate for each level. • The entire system is exercised using the root-level component.

36. Bottom-Up Integration

37. Bottom-Up Integration

38. Bottom-Up Integration

39. Bottom-Up Integration • Strategy • Automation • One driver for each component • Bottom-up driver corresponds to • Independent component • Set of components • Testing several components with a single driver may be possible • But you should avoid this • Reuse of drivers! • Reduce test driver maintenance

40. Bottom-Up Integration • Entry Criteria • VM is stable • Components in each stage meet exit criteria • No anomalies interfere with integration testing • Exit Criteria • Each component meets its exit criteria • Interface (per subcomponent) has been tested once or more • All root-level components pass their test suites

41. Bottom-Up Integration • Disadvantages  • Driver development (huge code size) • Promote reuse of drivers and test components • Changes to a tested component • Revise test configuration • No direct testing of intercomponent interfaces • Limited to component that is root of the subtree under test • Getting values or exceptions from lower-level components = difficult • Critical sections are tested last • Highest level of control in root -> tested last • Unstable component indications will come when component is exercised and not when it is integrated.

42. Bottom-Up Integration • Advantages  • Start testing as soon as any leaf-level component is ready • Work in parallel • Using subtrees • Well suited to responsibility-based design • Only component interfaces are exercised. • Find mistakes in requirements

43. Bottom-Up Integration • Known Uses • Used with clusters • Integration by Contract • Test design based on the relationship between client and server classes • Object Relations • “To test a class, what other classes must be tested?” • Read “Testing Object-Oriented Systems” for more information

44. Top-down Integration • Intent • Add components to the SUT in control hierarchy order, beginning with top-level control objects • Context • Iterative and incremental approach • Control structure of the system • Modeled as a tree • Top-level components have control responsibilities • Control objects

45. Top-down Integration • Context • Control objects • Essential control strategies: high risk • Focus on control object components first • Incremental development • Add layer under control layer on each increment • Concurrent hw/sw development • Develop hardware interfaces and control components at the same time • Concurrent sw development • Develop VM and control components at the same time

46. Top-down Integration

47. Top-down Integration

48. Top-down Integration

49. Top-down Integration • Disadvantages  • Stub development • Large number of stubs coded • Unforeseen requirements in a lower-level component may change many top-level components, breaking (part of) the test suite • Stubs are implementation-specific • Interoperability of all components in the SUT is not tested until the last component replaces its stub and the test suite passes

50. Top-down Integration • Advantages  • Early testing and integration • Reduced cost of driver development • Only one driver instead of a driver per subtree • Upper-level components are tested multiple times • Each time a lower-level component is added • Regression testing! • Parallel development • Different levels of components and stubs