Chris Murphy, Moses Vaughan, Waseem Ilahi, Gail Kaiser Columbia University

1. Automatic Detection of Previously-Unseen Application States for Deployment Environment Testing and Analysis Chris Murphy, Moses Vaughan, Waseem Ilahi, Gail Kaiser Columbia University

2. Overview • Deployment environment testing and analysis can find defects not found prior to release • Approaches potentially suffer from high overhead • We seek to increase the efficiency by only running tests/analysis in previously-unseen application states

3. In Vivo Testing [Murphy et al. ICST’09] • Automatically conduct tests as the software is running in the deployment environment • Unit tests, integration tests, etc. • “Runtime assertions with side effects” • Looks for defects in previously untested application states

4. In Vivo test Function f is about to be executed with input xin stateS Program continues Create a sandbox for the test Execute f(x) Execute INVtest_f(x) in state S Report violations

5. Research Question • Can the approach be made more efficient by only running tests and performing analysis in application states that the program has not seen before? • Yes, assuming we can quickly determine whether the current state has not previously been seen

6. Broader Impact • Other approaches may also use data about deployment environment execution • Monitoring: Anomaly detection • Profiling: Path coverage, line coverage, etc. • Fault localization • All of these may benefit if analysis is done only in previously-unseen states

7. Analysis Number of tests Time per test • Overhead of standard approach • Tstandard = N * ti • Overhead in previously unseen states • Tunseen = N * DSP * (td + tu + ti) • Overhead in previously seen states • Tseen = N * (1-DSP) * td • Find DSP so that Tunseen + Tseen ≤ Tstandard Distinct State Percentage Time to update list of seen states Time to determine if state has been seen before

8. Analysis Results • To be more efficient, we need: • DSP≤ (ti – td) / (ti + tu) • If td and tuare much less than ti the right side of the inequality approaches 1 • This means that even if nearly all states are distinct, this new approach will still be more efficient

9. Implementation Issues • How do we define the “state”? • How do we represent the state? • How can we quickly determine whether a state has previously been seen? • Does the previous analysis hold in the real world?

10. Defining “State” • We define “state” as “the values of all variables that are in scope at a given execution point” • For the purposes of In Vivo Testing, this can be further refined to “the values of all variables on which a function depends that are in scope at the start of the function execution”

11. Example We can statically determine that function f1 depends on: parameters p1, p2, p3; and global variables a and b

12. Representing States • Given our definition, the “state” is simply a map between variable names and values • We want to avoid element-wise comparison • We want to avoid false positives and false negatives a = 4 b = 3 … a = 2 b = 5 … a = 1 b = 8 … a = 2 b = 7 …

13. Cantor Function • Goal: Give each state a distinct value • Hashing function that assigns a distinct number to a pair of numbers [Royden 1988] • f(k1,k2) = (1/2)(k1+k2)(k1+k2+1) + k2 • Can be used recursively over a set of numerical values

14. Tracking Execution States • Even if each state has its own representation, how can we quickly determine whether a state has been seen before? • Hashtable: O(n) in worst case • Bloom filter: O(1), but allows for false positives 27 18 91 33 74 26 18 ?

15. Judy Array • Scalable, space efficient, speed efficient • Developed by Hewlett-Packard in 2001 • Highly-optimized 256-ary prefix tree data structure • Lookups are O(log256n) • Now we can quickly detect whether a state has already been seen

16. Automated Process • Statically analyze the source code to determine which parts of the state the function depends on • Create code that uses Cantor function to represent the state and Judy Array to determine whether it had already been seen • Generate instrumentation as normal, with call to function created in Step 2

17. Evaluation • In practice, is sometimes running the instrumentation (i.e., only in previously unseen states) really more efficient than always doing it? • Target: In Vivo Testing implementation in C • Sieve of Eratosthenes program run with 100 inputs, using varying percentages of distinct states ranging from: • 0%, i.e. all values are the same • 100%, i.e. all values are different

18. Results

19. Limitations & Future Work • Memory cost • Upper bound of Cantor function • Representation of complex objects • Coordinating globally-unseen states

20. Automatic Detection of Previously-Unseen Application States for Deployment Environment Testing and Analysis Chris Murphy Columbia University cmurphy@cs.columbia.edu

21. Related Work • Reducing overhead of runtime monitoring • Static analysis to remove unnecessary instrumentation [Yong & Horwitz, 2005] • Fast cases vs. slow cases [Liblit et al., 2003] • State representation for anomaly detection • [Baah et al., 2006] • [Hangal & Lam, 2002]