Automated Developer Testing: Achievements and Challenges

1. Automated Developer Testing:Achievements and Challenges • Tao Xie • North Carolina State University • In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU ASE

2. Why Automate Testing? • Software testing is important • Software errors cost the U.S. economy about $59.5 billion each year (0.6% of the GDP) [NIST 02] • Improving testing infrastructure could save 1/3 cost [NIST 02] • Software testing is costly • Account for even half the total cost of software development [Beizer 90] • Automated testing reduces manual testing effort • Test execution: JUnit, NUnit, xUnit, etc. • Test generation: Pex, AgitarOne, ParasoftJtest, etc. • Test-behavior checking: Pex, AgitarOne, ParasoftJtest, etc.

3. Automation in Developer Testing • Developer testing • http://www.developertesting.com/ • Kent Beck’s 2004 talk on “Future of Developer Testing”http://www.itconversations.com/shows/detail301.html • This talk focuses on tool automation indeveloper testing (e.g., unit testing) • Not system testing etc. conducted by testers

4. ? = Software Testing Setup + Expected Outputs Test inputs Program Outputs Test Oracles

5. ? = Software Testing Problems + Expected Outputs Test inputs Program Outputs Test Oracles • Test Generation • Generating high-quality test inputs (e.g., achieving high code coverage)

6. ? = Software Testing Problems + Expected Outputs Test inputs Program Outputs Test Oracles • Test Generation • Generating high-quality test inputs (e.g., achieving high code coverage) • Test Oracles • Specifying high-quality test oracles (e.g., guarding against various faults)

7. The Recipe of Unit Testing • Three essential ingredients: • Data • Method Sequence • Assertions void TestAdd() { int item = 3; var list = new List(); list.Add(item); Assert.AreEqual(1, list.Count); }

8. Parameterized Unit Testing [Tillmann&Schulte ESEC/FSE 05] • Parameterized Unit Test = Unit Test with Parameters • Separation of concerns • Data is generated by a tool • Developer can focus on functional specification void TestAdd(List list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); }

9. Parameterized Unit Tests areAlgebraic Specifications • A Parameterized Unit Test can be read as a universally quantified, conditional axiom. void TestReadWrite(Res r, string name, string data) {Assume.IsTrue(r!=null & name!=null && data!=null); r.WriteResource(name, data);Assert.AreEqual(r.ReadResource(name), data); }  string name, string data, Res r: r ≠ null ⋀ name ≠ null ⋀ data ≠ null ⇒ equals( ReadResource(WriteResource(r, name, data).state, name), data)

10. Parameterized Unit TestingGetting Popular Parameterized Unit Tests (PUTs) commonly supported by various test frameworks • .NET: Supported by .NET test frameworks • http://www.mbunit.com/ • http://www.nunit.org/ • … • Java: Supported by JUnit 4.X • http://www.junit.org/ Generating test inputs for PUTs supported by tools • .NET: Supported by Microsoft Research Pex • http://research.microsoft.com/Pex/ • Java: Supported by AgitarAgitarOne • http://www.agitar.com/

11. Test Generation • Human • Expensive, incomplete, … • Brute Force • Pairwise, predefined data, etc… • Random: • Cheap, Fast • “It passed a thousand tests” feeling • Dynamic Symbolic Execution: Pex, CUTE,EXE • Automated white-box • Not random – Constraint Solving

12. Dynamic Symbolic Execution Choose next path • Code to generate inputs for: Solve Execute&Monitor void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } Negated condition a==null F T a.Length>0 T F Done: There is no path left. a[0]==123… F T Data null {} {0} {123…} Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null && a.Length>0 && a[0]==1234567890 Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==1234567890

13. Challenges of DSE • Loops • Fitnex [Xie et al. DSN 09] • Generic API functions e.g., RegEx matching IsMatch(s1,regex1) • Reggae [Li et al. ASE 09-sp] • Method sequences • MSeqGen [Thummalapenta et al. ESEC/FSE 09] • Environments e.g., file systems, network, db, … • Parameterized Mock Objects [Marri et al. AST 09] Opportunities • Regression testing [Taneja et al. ICSE 09-nier] • Developer guidance (cooperative developer testing)

14. NCSU Industry Tech Transfer • Loops • Fitnex [Xie et al. DSN 09] • Generic API functions e.g., RegEx matching IsMatch(s1,regex1) • Reggae [Li et al. ASE 09-sp] • Method sequences • MSeqGen [Thummalapenta et al. ESEC/FSE 09] • Environments e.g., file systems, network, db, … • Parameterized Mock Objects [Marri et al. AST 09] Applications • Test network app at Army division@Fort Hood, Texas • Test DB app of hand-held medical assistant device at FDA

15. Pex on MSDN DevLabsIncubation Project for Visual Studio • Download counts (20 months)(Feb. 2008 - Oct. 2009 ) • Academic: 17,366 • Devlabs: 13,022 • Total: 30,388

16. NCSU Industry Tech Transfer • Loops • Fitnex [Xie et al. DSN 09] • Generic API functions e.g., RegEx matching IsMatch(s1,regex1) • Reggae [Li et al. ASE 09-sp] • Method sequences • MSeqGen [Thummalapenta et al. ESEC/FSE 09] • Environments e.g., file systems, network, db, … • Parameterized Mock Objects [Marri AST 09] Applications • Test network app at Army division@Fort Hood, Texas • Test DB app of hand-held medical assistant device at FDA

17. Explosion of Search Space There are decision procedures for individual path conditions, but… • Number of potential paths grows exponentially with number of branches • Without guidance, same loop might be unfolded forever Fitnex search strategy [Xie et al. DSN 09]

18. DSE Example Test input: TestLoop(0, {0}) public boolTestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } Path condition: !(x == 90) ↓ New path condition: (x == 90) ↓ New test input: TestLoop(90, {0})

19. DSE Example Test input: TestLoop(90, {0}) public boolTestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } Path condition: (x == 90) && !(y[0] == 15) ↓ New path condition: (x == 90) && (y[0] == 15) ↓ New test input: TestLoop(90, {15})

20. Challenge in DSE Test input: TestLoop(90, {15}) public boolTestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } Path condition: (x == 90) && (y[0] == 15) && !(x+1 == 110) ↓ New path condition: (x == 90) && (y[0] == 15) && (x+1 == 110) ↓ New test input: No solution!?

21. A Closer Look Test input: TestLoop(90, {15}) public boolTestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } Path condition: (x == 90) && (y[0] == 15) && (0 < y.Length) && !(1 < y.Length) && !(x+1 == 110) ↓ New path condition: (x == 90) && (y[0] == 15) && (0 < y.Length) && (1 < y.Length)  Expand array size

22. A Closer Look Test input: TestLoop(90, {15}) public boolTestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } We can have infinite paths! (both length and number) Manual analysis  need at least 20 loop iterations to cover the target branch Exploring all paths up to 20 loop iterations is practically infeasible: 220paths

23. Fitnex: Fitness-Guided Exploration public boolTestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } Test input: TestLoop(90, {15, 15}) Key observations: with respect to the coverage target, • not all paths are equally promising for flipping nodes • not all nodes are equallypromising to flip • Our solution: • Prefer to flip nodes on the most promisingpath • Prefer to flip the most promisingnodes on path • Use fitness function as a proxy for promising

24. Fitness Function • FF computes fitness value (distance between the current state and the goal state) • Search tries to minimize fitness value [Tracey et al. 98, Liu at al. 05, …]

25. Fitness Function for (x == 110) public boolTestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } Fitness function: |110 – x |

26. Compute Fitness Values for Paths FitnessValue public boolTestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } (x, y) (90, {0}) 20 (90, {15}) 19 (90, {15, 0}) 19 (90, {15, 15}) 18 (90, {15, 15, 0}) 18 (90, {15, 15, 15}) 17 (90, {15, 15, 15, 0}) 17 (90, {15, 15, 15, 15}) 16 (90, {15, 15, 15, 15, 0}) 16 (90, {15, 15, 15, 15, 15}) 15 … Fitness function: |110 – x | Give preference to flip a node in paths with better fitness values. We still need to address which node to flip on paths …

27. Compute Fitness Gains for Branches FitnessValue public boolTestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } (x, y) (90, {0}) 20 (90, {15})  flip b4 19 (90, {15, 0})  flip b2 19 (90, {15, 15})  flip b4 18 (90, {15, 15, 0})  flip b2 18 (90, {15, 15, 15})  flip b4 17 (90, {15, 15, 15, 0})  flip b2 17 (90, {15, 15, 15, 15})  flip b4 16 (90, {15, 15, 15, 15, 0})  flip b2 16 (90, {15, 15, 15, 15, 15})  flip b4 15 … Fitness function: |110 – x | Branch b1: i < y.Length Branch b2: i >= y.Length Branch b3: y[i] == 15 Branch b4: y[i] != 15 • Flipping branch node of b4 (b3) gives us average 1 (-1) fitness gain (loss) • Flipping branch node of b2 (b1) gives us average 0 (0) fitness gain (loss)

28. Compute Fitness Gains for Branches • Fitness gains: • FGain(b) := F(p) – F(p’) • FGain(b’) := F(p’) – F(p) • Compute the average fitness gain for each program branch over time p p’ n n b’ b …. …. F(p) is the fitness value of p F(p’) is the fitness value of p’

29. Implementation in Pex • Pex maintains global search frontier • All discovered branch nodes are added to frontier • Frontier may choose next branch node to flip • Fully explored branch nodes are removed from frontier • Pex has a default search frontier • It tries to create diversity across different coverage criteria • Frontiers can be combined in a fair round-robin scheme

30. Implementation in Pex We implemented a new search frontier “Fitnex”: • Nodes to flip are prioritized by their composite fitness value: F(pn) – FGain(bn), where • pn is path of node n • bn is explored outgoing branch of n • Fitnex always picks node with lowest composite fitness value to flip. • To avoid local optimal or biases, the fitness-guided strategy is combined with Pex’s search strategies

31. Evaluation Subjects A collection of micro-benchmark programs routinely used by the Pex developers to evaluate Pex’s performance, extracted from real, complex C# programs • Ranging from string matching like • if (value.StartsWith("Hello") && • value.EndsWith("World!") && • value.Contains(" ")) { … } • to a small parser for a Pascal-like language where the target is to create a legal program.

32. Search Strategies Under Comparison • Pex with the Fitnex strategy • Pex without the Fitnex strategy • Pex’s previous default strategy • Random • a strategy where branch nodes to flip are chosen randomly in the already explored execution tree • Iterative Deepening • a strategy where breadth-first search is performed over the execution tree

33. Evaluation Results #runs/iterations required to cover the target Pex w/o Fitnex: avg. improvement of factor 1.9 over Random Pex w/ Fitnex: avg. improvement of factor 5.2 over Random

34. Object Creation • Pex normally uses public methods to configure non-public object fields • Heuristics built-in to deal with common types • User can help if needed void (Foofoo) { if (foo.Value == 123) throw … [PexFactoryMethod] Foo Create(Bar bar) { return new Foo(bar);}

35. QuickGraph Example • A graph example from QuickGraph library • interface IGraph • { • /* Adds given vertex to the graph */ • void AddVertex(IVertex v); • /* Creates a new vertex and adds it to the graph */ • IVertexAddVertex(); • /* Adds an edge to the graph. Both vertices should • already exist in the graph */ • IEdgeAddEdge(IVertex v1, Ivertex v2); • } 35 35

36. Method Under Test • Desired object state for reaching targets 1 and 2: graph object should contain vertices and edges method sequence • Class SortAlgorithm • { • IGraph graph; • public SortAlgorithm(IGraphgraph) { • this.graph= graph; • } • public void Compute (IVertex s) { • foreach(IVertex u in graph.Vertices) • { • //Target 1 • } • foreach(IEdge e in graph.Edges) • { • //Target 2 • } • } • }

37. Method Under Test • Applying Randoop, a random testing approach that constructs test inputs by randomly selecting method calls Example sequence generated by Randoop VertexAndEdgeProvider v0 = new VertexAndEdgeProvider(); Boolean v1 = false; BidirectionalGraph v2 = new BidirectionalGraph((IVertexAndEdgeProvider)v0, v1); IVertex v3 = v2.AddVertex(); IVertex v4 = v0.ProvideVertex(); IEdge v15 = v2.AddEdge(v3, v4); v4 not in the graph, so edge cannot be added to graph. • Achieved 31.82% (7 of 22) branch coverage • Reason for low coverage: Not able to generate graph with vertices and edges

38. New MSeqGen Approach • Mine sequences from existing code bases • Reuse mined sequences for achieving desired object states A Mined sequence from an existing codebase VertexAndEdgeProvider v0; boolbVal; IGraphag = new AdjacencyGraph(v0, bVal); IVertex source = ag.AddVertex(); IVertex target = ag.AddVertex(); IVertex vertex3 = ag.AdVertex(); IEdge edg1 = ag.AddEdge(source, target); IEdge edg2 = ag.AddEdge(target, vertex3); IEdge edg3 = ag.AddEdge(source, vertex3); Graph objectincludes both vertices and edges • Use mined sequences to assist Randoop and Pex • Both Randoop and Pex achieved 86.40% (19 of 22) branch coverage with assistance from MSeqGen

39. Challenges Addressed by MSeqGen • Existing codebases are often large and complete analysis is expensive •  Search and analyze only relevant portions • Concrete values in mined sequences may be different from desired values •  Replace concrete values with symbolic values and use dynamic symbolic execution • Extracted sequences individually may not be sufficient to achieve desired object states •  Combine extracted sequences to generate new sequences

40. MSeqGen: Code Searching • Problem: Existing code bases are often large and complete analysis is expensive • Solution: • Use keyword search for identifying relevant method bodies using target classes • Analyze only those relevant method bodies Target classes: System.Collections.Hashtable • QuickGraph.Algorithms.TSAlgorithm Keywords: Hashtable, TSAlgorithm Shortnames of target classes are used as keywords

41. MSeqGen: Sequence Generalization • Problem: Concrete values in mined sequences are different from desired values to achieve target states • Solution: Generalize sequences by replacing concrete values with symbolic values Method Under Test Class A { int f1 { set; get; } int f2 { set; get; } void CoverMe() { if (f1 != 10) return; if (f2 > 25) throw new Exception(“bug”); } } Mined Sequence for A A obj = new A(); obj.setF1(14); obj.setF2(-10); obj.CoverMe(); Sequence cannot help in exposing bug since desired values are f1=10 and f2>25

42. MSeqGen: Sequence Generalization • Replace concrete values 14 and -10 with symbolic values X1 and X2 Generalized Sequence for A Mined Sequence for A A obj = new A(); obj.setF1(14); obj.setF2(-10); obj.CoverMe(); int x1 = *, x2 = *; A obj = new A(); obj.setF1(x1); obj.setF2(x2); obj.CoverMe(); • Use DSE for generating desired values for X1 and X2 • DSE explores CoverMemethod and generates desired values (X1 = 10 and X2 = 35)

43. Improvement of State-of-the-Art • Randoop • Without assistance from MSeqGen: achieved 32% branch coverage  achieved 86%branch coverage • In evaluation, help Randoop achieve 8.7% (maximum 20%) higher branch coverage • Pex • Without assistance from MSeqGen: achieved 45% branch coverage  achieved 86%branch coverage • In evaluation, help Pex achieve 17.4% (maximum 22.5%) higher branch coverage 43 43

44. Test Oracles • Write assertions and Pex will try to break them • Without assertions, Pex can only find violations of runtime contracts causing NullReferenceException, IndexOutOfRangeException, etc. • Assertions leveraged in product and test code • Pex can leverage Code Contracts

45. ? = Summary:Automated Developer Testing + Expected Outputs Test inputs Program Outputs Test Oracles Division of Labors • Test Generation • Test inputs for PUT generated by tools (e.g., Pex) • Fitnex: guided exploration of paths [DSN 09] • MSeqGen: exploiting real-usage sequences [ESEC/FSE 09] • Test Oracles • Assertions in PUT specified by developers

46. Thank you http://research.microsoft.com/pex http://pexase.codeplex.com/ https://sites.google.com/site/asergrp/

47. Code Contracts • http://research.microsoft.com/en-us/projects/contracts/ • Library to state preconditions, postconditions, invariants • Supported by two tools: • Static Checker • Rewriter: turns Code Contracts into runtime checks • Pex analyses the runtime checks • Contracts act as Test Oracle • Pex may find counter examples for contracts • Missing Contracts may be suggested

48. Example: ArrayList Class invariant specification: public class ArrayList { private Object[] _items; private int _size; ... [ContractInvariantMethod] // attribute comes with Contracts protected void Invariant() { Contract.Invariant(this._items != null); Contract.Invariant(this._size >= 0); Contract.Invariant(this._items.Length >= this._size); }

49. ParameterizedModels

50. Unit Testing vs. Integration Testing • Unit test: while it is debatable what a ‘unit’ is, a ‘unit’ should be small. • Integration test: exercises large portions of a system. • Observation: Integration tests are often “sold” as unit tests • White-box test generation does not scale well to integration test scenarios. • Possible solution: Introduce abstraction layers, and mock components not under test