The In Vivo Testing Approach

1. The In Vivo Testing Approach Christian Murphy, Gail Kaiser, Ian Vo, Matt Chu Columbia University

2. Problem Statement • It is infeasible to fully test a large system prior to deployment considering: • different runtime environments • different configuration options • different patterns of usage • This problem may be compounded by moving apps from single-CPU machines to multi-core processors

3. Our Solution • Continually test applications executing in the field (in vivo) as opposed to only testing in the development environment (in vitro) • Conduct the tests in the context of the running application • Do so without affecting the system’s users

4. int main ( ) { ... ... ... foo(x); int main ( ) { ... ... ... foo(x); test_foo(x); ... ... }

5. Contributions • A new testing approach called in vivo testing designed to execute tests in the deployment environment • A new type of tests called in vivo tests • An implementation framework called Invite

6. Related Work • Perpetual testing [Clarke SAS’00] • Skoll [Memon ICSE’04] • Gamma [Orso ISSTA’02] • CBI [Liblit PLDI’03] • Distributed In Vivo Testing [Chu ICST’08]

7. Number of items in the cache Their size (in bytes) Should only be incremented within “if” block Example of Defect: Cache private int numItems = 0, currSize = 0; private int maxCapacity = 1024; // in bytes public int getNumItems() { return numItems; } public boolean addItem(CacheItem i) throws ... { numItems++; add(i); currSize += i.size; return true; } Maximum capacity if (currSize + i.size < maxCapacity) { } else { return false; }

8. Insufficient Unit Test public void testAddItem() { Cache c = new Cache(); assert(c.addItem(new CacheItem())) assert(c.getNumItems() == 1); assert(c.addItem(new CacheItem())) assert(c.getNumItems() == 2); } 1. Assumes an empty/new cache 2. Doesn’t take into account various states that the cache can be in

9. Defects Targeted • Unit tests that make incomplete assumptions about the state of objects in the application • Possible field configurations that were not tested in the lab • A legal user action that puts the system in an unexpected state • A sequence of unanticipated user actions that breaks the system • Defects that only appear intermittently

10. Applications Targeted • Applications that produce calculations or results that may not be obviously wrong • “Non-testable programs” • Simulations • Applications in which exta-functional behavior may be wrong even if output is correct • Caching systems • Scheduling of tasks

11. In Vivo Testing: Process • Create test code (using existing unit tests or new In Vivo tests) • Instrument application using Invite testing framework • Configure framework • Deploy/execute application in the field

12. Run a test? Rest of program continues Stop Fork Model of Execution Function is about to be executed Execute function NO Yes Run test Create sandbox

13. Writing In Vivo Tests /* Method to be tested */ public boolean addItem(CacheItem i) { . . . } /* JUnit style test */ public void testAddItem() { Cache c = new Cache(); if (c.addItem(new CacheItem())) assert (c.getNumItems() == 1); } In Vivo boolean CacheItem i) { this; int oldNumItems = getNumItems(); i)) return oldNumItems+1; else return true;

14. Instrumentation /* Method to be tested */ public boolean __addItem(CacheItem i) { . . . } /* In Vivo style test */ public boolean testAddItem(CacheItem i) { ... } public boolean addItem(CacheItem i) { if (Invite.runTest(“Cache.addItem”)) { Invite.createSandboxAndFork(); if (Invite.isTestProcess()) { if (testAddItem(i) == false) Invite.fail(); else Invite.succeed(); Invite.destroySandboxAndExit(); } } return __addItem(i); }

15. Configuration • Each instrumented method has a set probability ρwith which its test(s) will run • To avoid bottlenecks, can also configure: • Maximum allowed performance overhead • Maximum number of simultaneous tests • Also, what action to take when a test fails

16. Case Studies • Applied testing approach to two caching systems • OSCache 2.1.1 • Apache JCS 1.3 • Both had known defects that were found by users (no corresponding unit tests for these defects) • Goal: demonstrate that “traditional” unit tests would miss these but In Vivo testing would detect them

17. Experimental Setup • An undergraduate student created unit tests for the methods that contained the defects • These tests passed in “development” • Student was then asked to convert the unit tests to In Vivo tests • Driver created to simulate real usage in a “deployment environment”

18. Discussion • In Vivo testing revealed all defects, even though unit testing did not • Some defects only appeared in certain states, e.g. when the cache was at full capacity • These are the very types of defects that In Vivo testing is targeted at • However, the approach depends heavily on the quality of the tests themselves

19. Performance Evaluation • We instrumented three C and two Java applications with the framework and varied the value ρ(probability that a test is run) • Applications were run with real-world inputs on a dual-core 3GHz server with 1GB RAM • No restraints were placed on maximum allowable overhead or simultaneous tests

20. Experimental Results Time (seconds) 0% 25% 50% 75% 100% percent of function calls resulting in tests

21. Discussion • Percent overhead is not a meaningful metric since it depends on the number of tests run • More tests = more overhead • Short-running programs with lots of tests will have significantly more “overhead” than long-running programs • For C, the overhead was 1.5ms per test • For Java, around 5.5ms per test

22. Future Work • Ensure that test does not affect the external system state (database, network, etc.) • Adjust frequency of test execution based on context or resource availability (CPU usage, number of threads, etc.) • Apply approach to certain domains, e.g. security testing

23. Conclusion • We have presented a new testing approach called in vivo testing designed to execute tests in the deployment environment • We have also presented an implementation framework called Invite • In Vivo testing is an effective technique at detecting defects not caught in the lab

24. The In Vivo Testing Approach Christian Murphy, Gail Kaiser, Ian Vo, Matt Chu Columbia University