Parameterized Unit Tests

1. Parameterized Unit Tests By Nikolai Tillmann and Wolfram Schulte Proc. of ESEC/FSE 2005 Presented by Yunho Kim Provable Software Lab, KAIST TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAA

2. Introduction Parameterized unit tests Framework Conclusion Contents Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

3. Unit testing is a method of testing that verifies the individual units of source code are working properly From http://en.wikipedia.org/wiki/Unit_testing • A unit test consists of a sequence of method calls with assertions • Unit tests are often served as simple specifications IntroductionWhat Is Unit Testing? [TestMethod] void TestAdd(){ ArrayList a = new ArrayList(0); // Make an array list object o = new object(); a.Add(o); // Push a value into it Assert.IsTrue(a[0] == o); // Check that value was added } Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

4. Unit tests tools do not provide any guidance for • which tests should be written for high code coverage • how to come up with a minimal number of test cases • what guarantees the test cases provide • What about other array lists and other objects? IntroductionWhat Is the Problem? [TestMethod] void TestAdd(){ ArrayList a = new ArrayList(0); // Make an array list object o = new object(); a.Add(o); // Push a value into it Assert.IsTrue(a[0] == o); // Check that value was added } Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

5. Parameterized unit tests are generalized unit tests by allowing parameters • Parameterized test methods are specifications of the behavior of the methods-under-tests • PUTs describe a set of traditional unit tests which can be obtained by instantiating the parameterized test methods IntroductionParameterized Unit Tests(PUTs) [PexMethod] void TestAdd(ArrayList a, object o){ // Given arbitrary array // lists and objects Assume.IsTrue(a!=null); // Assume a  null int i = a.Count; a.Add(o); // Push a value into it Assert.IsTrue(a[i] == o); // Check that value was added } Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

6. Introduction Parameterized unit tests Framework Conclusion Contents Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

7. Pex generates Unit Testsfrom hand-written Parameterized Unit Teststhrough Automated Exploratory Testingbased on Dynamic Symbolic Execution. Parameterized Unit TestsPex: a PUTs framework From http://research.microsoft.com/pex/default.aspx Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

8. Symbolic execution is used to automatically and systematically produce the minimal set of actual parameters(test inputs) Parameterized Unit TestsTest Case Generation • Algorithms • For each formal parameter a symbolic variable is introduced • For each code path, a path condition is built over symbolic variables • Loops and recursions are approximated up to a specified number of unfoldings • After all constraints are collected, find solutions for the constraints • Finally, deduce what inputs cause a code path to be taken Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

9. The number of possible paths to consider can grow exponentially with the number of control flow decisions Parameterized Unit TestsReusing Parameterized Unit Tests 1 class Bag{ 2 Hashtable h = new Hashtable(); 3 4 void Add(object o){ 5 if (h.ContainsKey(o)) h[o] = (int)h[o] + 1; 6 else h[o] = 1; 7 } 8 int multiplicity(object o){ 9 if (h.ContainsKey(o)) return (int)h[o]; 10 else return 0; 11 } 12 } During symbolic execution, ContainsKey method is symbolically run twice 1 [PexMethod] 2 Void AddMultiplicityTest(Bag b, objectx){ 3 Assume.IsTrue(b!=null & x!=nul); 4 int m = b.multiplicity(x); 5 b.Add(x); 6 Assert.IsTrue(m+1 == b.Multiplicity(x)); Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

10. If we have axioms summarizing the hash table’s operations, we can avoid execution of the hash table Parameterized Unit TestsReusing Parameterized Unit Tests [TestClass, UsingAxioms(typeof(HashtableTests))] Class BagTests{ [PexMethod] void AddMultiplicityTest() {  }  } Provided axioms are re-written as constraints [TestClass, ProvidingAxioms(typeof(Hashtable))] Class HashtableTests{ [TestAxiom] void NothingContainedInitially(object key){  } [TestAxiom] void SetImpliesContainsAndGet(){  }  } Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

11. Introduction • Parameterized unit tests • Framework • Symbolic state • Constraints • Symbolic Evaluation • Axioms • Conclusion Contents Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

12. Symbolic states are like concrete states except that symbolic states can contain symbolic expressions Symbolic expressions E is described by the following grammar ObjectId is an infinite set of potential object identifiers VarId, TypeId and FuncId are a set of variable, type and function identifiers respectively FrameworkSymbolic State E = o object ids 2ObjectId | v variables 2VarId | t types 2TypeId | f( ) function application 2FuncId | 8 .E universal quantification Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

13. There are two types of function symbols • Predefined function symbols have a fixed interpretation in the theorem provers used • add(x, y), equals(x, y), subtype(x, y) • null, void, 0, 1, 2, … (constants are considered as nullary functions) • Storage function symbols operating on maps • update(m, x, y) denotes the update of map m at index x to the new value y • select (m, x) selects the value of map m at index x • Uninterpreted function symbols represent properties of objects and method calls appearing in axioms FrameworkSymbolic State Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

14. Uninterpreted function symbols represent properties of objects and method calls appearing in axioms • For example, type(x) denotes the type of object x, fieldf(x) denotes the address of field f of object x • For each method m of the program with n parameters, up to three uninterpreted function symbols are introduced, ms, mx, mr • They are used to summarize different dynamic aspects of m • Let h be an expression denoting the state of the heap in which m( ) is called • ms(h, ) denotes the resulting state of the heap • mr(h, ) denotes the return value of the call • mx(h, ) represents the type of an exception FrameworkSymbolic State Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

15. There are two kinds of heaps, ‘extensional’ and ‘intensional’ heap • The extensional heap is represented by nested applications of the map update function • For example, the execution of the code fragment turns a given extensional heap He into H’eassuming the locations p, q hold the expressions t, u • Extensional heap operations have fixed interpretations in the theorem provers FrameworkSymbolic State p.f=1; q.g=2 H’e = update(update(He, fieldf(t), 1), fieldg(u), 2) Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

16. The intensional heap is described by a history of method invocations ms(Hi, ) represents the sequence of method calls followed by a call to m( ) Consider for example the execution of the following code fragment assuming that t is the expression held in location o Intensional heap operations are uninterpreted functions FrameworkSymbolic State ArrayList a = new ArrayList(); a.Add(o); H’i = Adds (ArrayLists(Hi,a),a,t) Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

17. A symbolic state is a 5-tuple S = (O, A, He, Hi, X) • O½ObjectId • A is a stack of activation records associated with a method, a program counter, as well as a store for the formal parameters and local variables • He and Hi are expression denoting the extensional heap and the intensional heap respectively • X is an object expression denoting the current exception • S+1 is like S except that the program counter has been incremented • The set of all symbolic states is called State FrameworkSymbolic State Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

18. A constraint on a symbolic state is a pair C = (BG, PC) BG is the static background only depending on the program declarations PC is the dynamic path condition built up during symbolic execution The set of all constraints is called Constraints A constrained state is a pair (S, C) Framework Constraints Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

19. The one-step relation describes the effect of the current instruction from a given constrained state (S, C) • Most instructions are handled in the standard way • A method call instruction pushes a new activation record onto the program stack • The interesting cases are object creation, conditional branch, member access, assume and assert Framework Symbolic Evaluation !µ (State£Constraints) £ (State£Constraints) Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

20. New object creation of type T • Let o2ObjectId – O(S) be a fresh object identifier • (S, C) ! (S’, CÆtype(o) = T) where S’ is like S+1 except that He(S) is replaced with • Where f1,  , fn are the fields of T and v1,  , vn default values • Conditional branch with a condition c and label l • If (S, CÆc) is feasible, then (S, C) ! (S’, CÆc) where the program counter in S’ is set to l • If (S, CÆ: c) is feasible, then (S, C) ! (S+1, CÆ:c) Framework Symbolic Evaluation update(update (h,fieldf1 (o), v1) , fieldfn (o), vn) Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

21. Member access with target expression t and result location r • If (S, CÆtnull) is feasible, then (S, C) ! (S’, CÆtnull) where S’ is like S+1 except that the location r holds (He (S), fieldf(t)) • If (S, CÆt = null) is feasible, then (S, C) ! (S’’, CÆtype(e) = NullReferenceExceptionÆt = null) where S’’ is like S except that the current exception X = e • Assume.IsTrue with condition c • If (S, CÆc ) is feasible, then (S, C) ! (S+1, CÆc) • Assert.IsTrue with condition c is semantically equivalent to Framework Symbolic Evaluation If (!c) throw new AssertFailedException(); Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

22. PUT can be seen as a summary, an axiom of external behavior of method-under-test • To summarize the set of methods M, we refine the behavior of ! relation for method calls to M • Call to a method m2M • Let H’i = ms (Hi (S), ). • If (S, CÆmx (Hi(S), ) = void) is feasible, (S, C) ! (S’, CÆmx(hi(S), ) = void) where S’ is S+1 except that Hi is replaced with H’i • IF (S, CÆmx (Hi(S), ) void) is feasible, (S, C) ! (S’’, CÆmx(Hi (S), ) void) where S’’ is S+1 except that Hi is replaced with H’i and X = mx(hi(S), ) Framework Axioms Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

23. PUTs generate axiom formulas for by exploring a test axiom method like we do for test case generation PUTs explore the method with a modified one-step relation !‘ !‘ is like except that Assert.IsTrue(c) in a state (S, C) is treated like Assume.IsTrue(c) and a new axiom formula 8h, .PC(C) !c is generated and conjoined BG(C) Parameterized Unit TestsAxioms [PexMethod] void TestAdd(ArrayList a, object o){ // 0 Assume.IsTrue(a!=null); // 1 int i = a.Count; // 2 a.Add(o); // 3 Assert.IsTrue(a[i] == o); // 4 } Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

24. For example Parameterized Unit TestsAxioms [PexMethod] void TestAdd(ArrayList a, object o){ // 0 Assume.IsTrue(a!=null); // 1 int i = a.Count; // 2 a.Add(o); // 3 Assert.IsTrue(a[i] == o); // 4 } Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

25. For example Exploring TestAdd generates the following universally quantified formula as an axiom Parameterized Unit TestsAxioms [PexMethod] void TestAdd(ArrayList a, object o){ // 0 Assume.IsTrue(a!=null); // 1 int i = a.Count; // 2 a.Add(o); // 3 Assert.IsTrue(a[i] == o); // 4 } 8h,a,o. (a = nullÇsubtype(type(a), ArrayList)) Æ anull! getItemr ( Adds (getCounts (h,a),a,o), a, getCountr (h,a)) = o Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

26. The authors presented the concept of parameterized unit tests, a generalization of unit tests • PUTs can be turned into axioms, which summarize three different aspects of the methods’ behavior • The axioms can be reused during symbolic execution Conclusion Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST

27. Parameterized Unit Tests by Parameterized Unit Tests in Proc. of ESEC/FSE 2005 Reference Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST