For Developer Testing

1. Separation of Concerns For Developer Testing Tao Xie Peking University, China North Carolina State University, USA In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU ASE

2. Background – Separation of Concerns • Separation of Concerns (SoC) • Aspect-Oriented Software Development (AOSD) • “Killer Apps” for SoC/AOSD? • Logging, Security Checking, … • Any Other Application Scenarios in Practice? • Tool Support for Software Testing?

3. ? = Software Testing Setup + Expected Outputs Test inputs Program Outputs Test Oracles

4. ? = Software Testing Problems + Expected Outputs Test inputs Program Outputs Test Oracles • Test Generation • Generating high-quality test inputs (e.g., achieving high code coverage)

5. ? = Software Testing Problems + Expected Outputs Test inputs Program Outputs Test Oracles • Test Generation • Generating high-quality test inputs (e.g., achieving high code coverage) • Test Oracles • Specifying high-quality test oracles (e.g., guarding against various faults)

6. The Recipe of Unit/Dev Testing • Three essential ingredients: • Data • Method Sequence • Assertions void TestAdd() { int item = 3; var list = new ArrayList(); list.Add(item); Assert.AreEqual(1, list.Count); }

7. Outline • Parameterized Unit Testing • Separate Test Generation and Behavior Specification • Test Generalization • Localize Crosscutting Concerns of Behavior Under Test • Object Creation with Factory Method • Localize Crosscutting Concerns of Object Creation • Environment Isolation with Moles • Localize Crosscutting Concerns of Environment Dependency

8. Parameterized Unit Testing [Tillmann&Schulte ESEC/FSE 05] • Parameterized Unit Test = Unit Test with Parameters • Separation of concerns • Data is generated by a tool • Developer can focus on functional specification void TestAdd(ArrayListlist, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); }

9. Dynamic Symbolic Execution Choose next path • Code to generate inputs for: Solve Execute&Monitor void TestMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } Negated condition a==null F T a.Length>0 T F Done: There is no path left. a[0]==123… F T Data null {} {0} {123…} Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null && a.Length>0 && a[0]==1234567890 Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==1234567890

10. Pex:Visual Studio Power Tool http://research.microsoft.com/projects/pex/ • Download counts (20 months)(Feb. 2008 - Oct. 2009 ) • Academic: 17,366 • Devlabs: 13,022 • Total: 30,388 Dynamic Symbolic Execution [Tillmann&deHalleux TAP 08]

11. Pex for FunWeb-based Learning Tool Try it at http://www.pexforfun.com/ 669,584 clicked 'Ask Pex!‘ since 2010 summer

12. Parameterized Unit TestingGetting Popular Parameterized Unit Tests (PUTs) commonly supported by various test frameworks • .NET: Supported by .NET test frameworks • http://www.mbunit.com/ • http://www.nunit.org/ • … • Java: Supported by JUnit 4.X • http://www.junit.org/ Generating test inputs for PUTs supported by tools • .NET: Supported by Microsoft Research Pex • http://research.microsoft.com/Pex/ • Java: Supported by Agitar AgitarOne • http://www.agitar.com/

13. Parameterized vs. Conventional Unit Tests PUTs vs. CUTs • Benefits over CUTs • Help describe behaviours for all test arguments • Address two main issues with CUTs • Missing test data (that would exercise important behavior) • Low fault-detection capability • Redundant test data and scenarios(that exercises the same behaviour) • Redundant unit tests

14. Outline • Parameterized Unit Testing • Separate Test Generation and Behavior Specification • Test Generalization • Localize Crosscutting Concerns of Behavior Under Test • Object Creation with Factory Method • Localize Crosscutting Concerns of Object Creation • Environment Isolation with Moles • Localize Crosscutting Concerns of Environment Dependency

15. An Example using IntStack Three CUTs • CUT1 and CUT2 exercise push with different test data • CUT3 exercises push when stack is not empty • Two main issues with CUTs: • Fault-detection capability: undetected defect where things go wrong when passing a negative value to push • Redundant tests: CUT2 is redundant with respect to CUT1 public void CUT1() { intelem = 1; IntStackstk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count()); } public void CUT2() { intelem = 30; IntStackstk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count()); } public void CUT3() { int elem1 = 1, elem2 = 30; IntStackstk = new IntStack(); stk.Push(elem1); stk.Push(elem2); Assert.AreEqual(2, stk.Count()); }

16. Test Generalization: IntStack An equivalent PUT Three CUTs public void CUT1() { intelem = 1; IntStackstk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count()); } public void PUT(int[] elem) { Assume.IsTrue(elem != null); IntStackstk = newIntStack(); for(inti in elem) stk.push(elem); Assert.AreEqual(elem.Length, stk.count()); } public void CUT2() { intelem = 30; IntStackstk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count()); } • No need to describe test data • Generated automatically • Single PUT replaces multiple CUTs • With reduced size of test code public void CUT3() { int elem1 = 1, elem2 = 30; IntStackstk = new IntStack(); stk.Push(elem1); stk.Push(elem2); Assert.AreEqual(2, stk.Count()); } [Thummalapenta et al. FASE 11]

17. Ring a Bell? Refactoring Clones? • Refactoring clone instances (crosscutting concerns) to a single copy (aspects) • Need pointcuts to specify join points, i.e., which original code locations to weave the aspects • Test generalization is different • Original code locations (old CUTs) are thrown away • PUTs + Pex generate new CUTs  No need of join points or pointcuts

18. Outline • Parameterized Unit Testing • Separate Test Generation and Behavior Specification • Test Generalization • Localize Crosscutting Concerns of Behavior Under Test • Object Creation with Factory Method • Localize Crosscutting Concerns of Object Creation • Environment Isolation with Moles • Localize Crosscutting Concerns of Environment Dependency

19. Object Creation void TestAdd(ArrayListlist, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); }

20. Object Creation • Pex uses public methods to configure non-public object fields • Heuristics built-in to deal with common types • User can help if needed [PexFactoryMethod(typeof(ArrayList))] public static ArrayList Create(intcapacity, object[] items) { varlist = new ArrayList(capacity); foreach (varitem in items) list.Add(item); return list; }

21. “Join Point” for Factory Method void TestAdd(ArrayListlist, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); } [PexFactoryMethod(typeof(ArrayList))] public static ArrayList Create(intcapacity, object[] items) { varlist = new ArrayList(capacity); foreach (varitem in items) list.Add(item); return list; }

22. Outline • Parameterized Unit Testing • Separate Test Generation and Behavior Specification • Test Generalization • Localize Crosscutting Concerns of Behavior Under Test • Object Creation with Factory Method • Localize Crosscutting Concerns of Object Creation • Environment Isolation with Moles • Localize Crosscutting Concerns of Environment Dependency

23. Unit Testing is Not That Easy • Components depend on other components • Hidden Integration Tests boolIsFileEmpty(string file) { var content = File.ReadAllText(file); return content.Length == 0; } File.ReadAllText(file); void FileExistsTest() { File.Write(“foo.txt”, “”); var result = IsFileEmpty(“foo.txt”) Assert.IsTrue(result); } File.Write(“foo.txt”, “”);

24. Isolation is Critical • Slow, complicated setup, non-deterministic tests • Solution: Replace by Simpler Environment (“mocking”) • Testable Design: Abstraction layer + Dependency Injection + Mocks for testing • Simply use virtual methods (i.e., overridenable) • Hard-coded Design: No abstraction layer, static methods, sealed types (i.e., not inheritable).

25. Stubs and Moles Framework • Replace any .NET method with a Delegate • similar to a C++ function pointer • Method can be overridden? Use Stubs • Interfaces • Abstract classes • Virtual methods in non-sealed types • Method cannot be overridden? Use Moles

26. Hard-coded Design • Existing external components cannot be re-factored • SharePoint, ASP .NET, … • Need mechanism to stub non-virtual methods • Static methods, methods in sealed types, constructors

27. Moles– Delegate Based Detours • Method redirected to user delegate, i.e. moled • Requires code instrumentation MFile.ReadAllTextString = file => “”; expression lambda : (input parameters) => expression bool result = IsFileEmpty(“foo.txt”); Assert.IsTrue(result); [de Halleux&Tillmann TOOLS 10]

28. Moles under the Hood mscorlib.dll File.ReadAllText(string name) { } .NET Runtime Just In Time Compiler File.ReadAllText(string name) { var d = GetDetour(); if (d != null) return d(); } push ecx push edx push eax ExtendedReflection

29. Isolated Parameterized Unit Testing • Automated White box Analysis does not ‘understand’ the environment • Isolate Code using Stubs and Moles ??? if (DateTime.Now == new DateTime(2000,1,1)) throw new Y2KException(); DateTime.Now public void TestY2k(DateTimedt) {MDateTime.NowGet = () => dt ... } MDateTime.NowGet = () => dt DateTime.Now == dt

30. Separation of Concerns for Developer Testing • Parameterized Unit Testing • Separate Test Generation and Behavior Specification • Test Generalization • Localize Crosscutting Concerns of Behavior Under Test • Object Creation with Factory Method • Localize Crosscutting Concerns of Object Creation • Environment Isolation with Moles • Localize Crosscutting Concerns of Environment Dependency Application Scenarios of Separation of Concerns in Practice

31. Thank you! http://research.microsoft.com/projects/pex/ Questions ? http://www.pexforfun.com/ http://research.microsoft.com/en-us/projects/pex/community.aspx#publications http://pexase.codeplex.com/ https://sites.google.com/site/asergrp/