Distributed In Vivo Testing of Software Applications

1. Distributed In Vivo Testing of Software Applications Matt Chu, Christian Murphy, Gail Kaiser Columbia University

2. Introduction • It is infeasible to fully test a large system prior to deployment considering: • different runtime environments • different configuration options • different patterns of usage • Extending testing into the deployment environment can help reveal faults

3. In Vivo Testing • We have developed an approach called in vivo testing • Unit tests from the development environment (in vitro) are executed at run-time in the deployment environment (in vivo) • Done without affecting the state of the application • However, this approach can incur a large performance overhead

4. Distributed In Vivo Testing • In this work we present a distributed approach that takes advantage of networked “application communities” • Members of the community each run fewer tests (thus less overhead) and results are aggregated centrally

5. Benefits of Distributed In Vivo Testing • Amortizing the workload over many instances tackles the high performance cost without sacrificing the quantity of tests being conducted globally • Having a community of applications collaboratively working together increases the possibility that an instance will find these erroneous permutations of state

6. Overview • Related work • In vivo testing & the Invite framework • The distributed Invite framework • Experimental results • Conclusions and future work

7. Related Work • Perpetual testing [Clarke SAS’00] • Skoll [Memon ICSE’04] • Gamma [Orso ISSTA’02] • CBI [Liblit PLDI’03] • MORABIT [Merdes SEM’06]

8. In Vivo Testing • Unit tests are executed in the deployment environment, in the context of the running application, as opposed to a controlled development environment • Tests are run continuously as the application runs, at selected points in the program execution • Tests are run in a separate process (which has been created as an exact copy of the original), so as not to affect the state of the original process

9. In Vivo Testing: Defects Targeted • Unit tests make incorrect assumptions about the state of objects in the application • Possible field configurations were not tested in the lab • A legal user action puts the system in an unexpected state • An sequence of unanticipated user actions breaks the system • Defects that only appear intermittently

10. Invite: The in vivo testing framework • Implemented in Java • Uses AspectJ to “weave in” the instrumentation at compile-time • Configuration performed by software developer • Frequency of test execution (per method) • Reaction policy • Software deployed as usual, with no requirements on end user

11. Run a test? Execute foo no . . . yes Execute testFoo Record success/failure Invite: Running a test Method foo is about to be executed Run the test in a separate process

12. Distributed in vivo testing approach • Amortize the cost of test execution by distributing the workload over a number of instances • Cover a variety of different platforms, usage patterns, etc. • Client/Server architecture • Server can distribute tests according to targeting or sampling policies

13. Distributed Invite framework Invite Server

14. Experiment Setup • Goal: ensure that evenly distributing testing load improves performance by decreasing the number of tests each individual member in the community has to run. • Subject: Jetty WebServer • 5000 dynamic & static web pages requested from each instance

15. Experimental Results % overhead community size

16. Summary • Overhead declines as the number of members in the community grows • Analytical results: • 18 members: 25% • 45 members: 10% • 90 members: 5% • Optimal minimum overhead (from instrumentation) was 2.4%

17. Limitations & Future Work • Ensure that the test does not affect the external system state • Improve distribution of tests • Adjust frequency of test execution based on context or resource availability • Include support for capture/replay, fault localization, etc.

18. Conclusion • Distributed in vivo testing addresses one of the biggest problems of the in vivo testing approach (extremely high performance penalties) by distributing the number of tests that need to be run onto the entire community of applications • The approach maintains the same number of tests globally, and also encompasses more diverse environments

19. Distributed In Vivo Testing of Software Applications Matt Chu, Christian Murphy, Gail Kaiser Columbia University