Christian Murphy, Kuang Shen, Gail Kaiser Columbia University

1. Using JML Runtime Assertion Checking to Automate Metamorphic Testing inApplications without Test Oracles Christian Murphy, Kuang Shen, Gail Kaiser Columbia University

2. Problem Statement • Some applications (e.g. machine learning, simulation) do not have test oracles that indicate whether the output is correct for arbitrary input • Oracles may exist for a limited subset of the input domain, and gross errors (e.g. crashes) can be detected with certain inputs or other techniques • However, it is difficult to detect subtle (computational) errors for arbitrary inputs

3. Observation • Even when there is no oracle in the general case (i.e. knowing the relationship between a particular input and its output), it may be possible to know relationships between sets of inputs and their corresponding outputs • One such approach that takes advantage of this is “Metamorphic Testing” [Chen ’98]

4. Metamorphic Testing • An approach for creating follow-up test cases based on those that have not revealed any defects • If input x produces output f(x), then the function’s “metamorphic properties” are used to guide a transformation function t, which is applied to produce a new test case, t(x) • We can then predict the expected value of f(t(x)) based on the value of f(x) obtained from the actual execution

5. Metamorphic Testing Example • Consider a function std_dev(A[ ]) • If we permute the elements in A, the result should stay the same • If we multiply each element in A by 2, the result should be doubled • These properties can be used to create a “pseudo-oracle” for the function

6. Metamorphic Testing without an Oracle • When a test oracle exists, we can know whether f(t(x)) is correct • Because we have an oracle for f(x) • So if f(t(x)) is as expected, then it is correct • When there is no test oracle, f(x) acts as a “pseudo-oracle” for f(t(x)) • If f(t(x)) is as expected, it is not necessarily correct • However, if f(t(x)) is not as expected, either f(x) or f(t(x)) (or both) is wrong

7. Our Solution • To address the problem of applications that have no test oracle, we use Metamorphic Testing at the function level • To specify the metamorphic properties, we use extensions to the Java Modeling Language (JML) and a pre-processor called Corduroy • To automate the process and address the need for initial input values, we check the properties at runtime

8. Related Work • Applying Metamorphic Testing to “non-testable programs” (applications without test oracles) • Chen IST’02, Chen ISSTA’02, Chan JWSR’07 • Checking algebraic specifications at runtime • Sankar ISSTA’91, Nunes ICFEM’06 • Checking program invariants at runtime • Flanagan FME’01, Hangal ICSE’02

9. JML Basics • Behavioral interface specification language that embraces the Design by Contract approach • Developers can specify: • preconditions (“requires”) • postconditions (“ensures”) • arbitrary assertions • Specifications can be checked dynamically using JML Runtime Assertion Checking tools

10. Corduroy Methods that check metamorphic properties Pure JML specifications that invoke test methods Process Specify functions’ metamorphic properties using extension to JML

11. Metamorphic Properties in JML /*@ @ensures \result <= 1 && \result >= -1; @meta sine(x + 2 * PI) == \result @meta -1 * sine(-x) == \result */ public double sine (double x) { ... }

12. Extension to JML: Array Functions /*@ @ensures \result >= 0; @meta std_dev(\shuffle(A)) == \result @meta std_dev(\multiply(A, 2)) == \result * 2 */ public double std_dev(double[] A) { ... }

13. Extensions to JML • Array/Collection Functions • Shuffle/permute the order of the elements • Multiply each element by a constant • Add a constant to each element • Include a new element • Exclude an element • Addressing non-determinism • Check in range of possible values • Check in a set of possible values

14. Input(s) Output Model Input(s) Caller Function Callee Function Check Metamorphic Properties Output

15. Implementation /*@ @meta average(\multiply(A, 2)) == \result * 2 */ public double average(double[] A) { ... } protected boolean __metaTest_average (double[] A, double result) { return JMLDouble.approximatelyEqualTo (average(Corduroy.multiply(A, 2)), result * 2); } @ensures __metaTest_average(A, \result);

16. Case Studies • We investigated the WEKA and RapidMiner toolkits for Machine Learning in Java • For WEKA, we tested four apps: • Naïve Bayes, Support Vector Machines (SVM), C4.5 Decision Tree, and k-Nearest Neighbors • For RapidMiner, we tested one app: • Naïve Bayes

17. Experimental Setup • For each of the five apps, we specified 4-6 metamorphic properties of selected methods (based on our knowledge of the expected behavior of the overall application) • Testing was conducted using data sets from UCI Machine Learning Repository • Goal was to determine whether the properties held as expected

18. Findings • Discovered defects in WEKA k-NN and WEKA Naïve Bayes related to modifying the machine learning “model” • This was the result of a variable not being updated appropriately • Discovered a defect in RapidMiner Naïve Bayes related to determining confidence • There was an error in the calculation

19. Limitations • Specifying and checking the properties was simplified by use of the tool, but still requires some knowledge of the app to determine the properties in the first place • The approach works well for functions that take input and produce output, but in some case requires more complex properties than can be expressed in JML

20. Future Work • Reducing testing time by checking properties in parallel • Implementations for other languages • More empirical studies to determine effectiveness and applicability to other domains

21. Summary • We have presented a testing approach that uses metamorphic testing of individual functions in applications that do not have test oracles • These are specified via an extension to the JML specificationlanguage • We have also presented an implementation framework called Corduroy, which converts the specification of metamorphic properties into test methods

22. Using JML Runtime Assertion Checking to Automate Metamorphic Testing inApplications without Test Oracles Chris Murphy cmurphy@cs.columbia.edu

23. Categories of Metamorphic Properties • Additive: Increase (or decrease) numerical values by a constant • Multiplicative: Multiply numerical values by a constant • Permutative: Randomly permute the order of elements in a set • Invertive: Reverse the order of elements in a set • Inclusive: Add a new element to a set • Exclusive: Remove an element from a set • ML apps such as ranking, classification, and anomaly detection exhibit these properties [Murphy SEKE’08]

24. Specifying More Complex Properties /*@ @meta test_myFunc(x, \result) */ public double myFunc (int x) { ... } private boolean test_myFunc (int x, double result) { ... }

25. Addressing Side Effects // holds result of last call to “average” private double value; /*@ @assignable value; @meta average(\multiply(A, 2)) == value * 2 */ public void average(double[] A) { ... value = ... // no return value }

26. Addressing Side Effects protected boolean __metaTest_average (double[] A) { double __value = value; // backup try { return JMLDouble.approximatelyEqualTo (average(Corduroy.multiply(A, 2)), value * 2); } finally { value = __value; // restore } } __value