Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

1. PexxxxWhite Box Test Generationfor .NETNikolai Tillmann, Peli de Halleux Microsoft Research

2. Unit Testing Today A unit test is a small program with assertions. void AddTest() { HashSet set = new HashSet(); set.Add(7); set.Add(3); Assert.IsTrue(set.Count == 2); } Many developers write such unit tests by hand.

3. Vision: Parameterized Unit Testing void AddSpec(int x, int y) { HashSet set = new HashSet(); set.Add(x); set.Add(y); Assert.AreEqual(x == y, set.Count == 1); Assert.AreEqual(x != y, set.Count == 2); } Parameterized Unit Tests separate two concerns: • The specification of externally visible behavior. (assertions) • The selection of internally relevant test inputs. (coverage)

4. Demo: Parameterized Unit Test Write [PexMethod] with parameters, invoke Analysis

5. Demo: Pex Exploration Results Table: dataand results

6. Demo: Generated Unit Tests Generated Test Inputs are stored as C# Unit Tests

7. Pex: Overview • Purpose: Test input generator • Starting from parameterized unit tests • Generated tests emitted as traditional unit tests • Goal: Test suite that covers all reachable statements • Technology: Dynamic symbolic execution • Whole-program, white-box code analysis • At the level of the .NET instructions (bytecode) • Symbolic execution based on monitoring and re-execution • Path conditions obtained by monitoring dataflow, branches • Iterative selection of potential execution paths • Constraint solver determines test inputs for paths

8. Dynamic Symbolic Execution By Example Choose next path Solve Execute&Monitor Code to generate inputs for: Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==123456890 Observed constraints a==null a!=null && !(a.Length>0) a==null && a.Length>0 && a[0]!=1234567890 a==null && a.Length>0 && a[0]==1234567890 Input null {} {0} {123…} void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } Negated condition a==null T F Done: There is no path left. a.Length>0 T F a[0]==123… F T

9. Going from simple example Into the real world How to test this code? (Actual code from .NET base class libraries.)

10. Real World (II)

11. Real World (III) • There is no detailed specification. • Most basic test oracle: • Code should not crash, i.e. throwing unexpected exceptions Possible test case, written by Hand

12. Pex – Demo Test input, generated by Pex 12

13. Test Input Generation byDynamic Symbolic Execution Initially, choose Arbitrary Run Test and Monitor Solve TestInputs Constraint System Execution Path KnownPaths Choose an Uncovered Path Record Path Condition

14. Test Input Generation byDynamic Symbolic Execution a[0] = 0; a[1] = 0; a[2] = 0; a[3] = 0; … Initially, choose Arbitrary Run Test and Monitor Solve TestInputs Constraint System Execution Path KnownPaths Choose an Uncovered Path Record Path Condition

15. Test Input Generation byDynamic Symbolic Execution Initially, choose Arbitrary • Path Condition: • … ⋀ magicNum != 0x95673948 Run Test and Monitor Solve TestInputs Constraint System Execution Path KnownPaths Choose an Uncovered Path Record Path Condition

16. Test Input Generation byDynamic Symbolic Execution Initially, choose Arbitrary Run Test and Monitor • … ⋀ magicNum != 0x95673948 • … ⋀ magicNum == 0x95673948 Solve TestInputs Constraint System Execution Path KnownPaths Choose an Uncovered Path Record Path Condition

17. Test Input Generation byDynamic Symbolic Execution a[0] = 206; a[1] = 202; a[2] = 239; a[3] = 190; Initially, choose Arbitrary Run Test and Monitor Solve TestInputs Constraint System Execution Path KnownPaths Choose an Uncovered Path Record Path Condition

18. Test Input Generation byDynamic Symbolic Execution Initially, choose Arbitrary Run Test and Monitor Solve TestInputs Constraint System Execution Path KnownPaths Choose an Uncovered Path Record Path Condition

19. Test Input Generation byDynamic Symbolic Execution Initially, choose Arbitrary Run Test and Monitor Solve TestInputs Constraint System Execution Path KnownPaths Choose an Uncovered Path Record Path Condition Finds only real bugs No false warnings Result: small test suite, high code coverage

20. Advanced Features • Command-line tool • Explores your code on your build server • Generates HTML reports • Parameterized Mock Objects • Stubs and mock objects can use parameters too • Parameterized Object Factories • Factories that take parameters help Pex

21. Assumptions and Assertions void PexAssume.IsTrue(bool c) { if (!c) throw new AssumptionViolationException(); } void PexAssert.IsTrue(bool c) { if (!c) throw new AssertionViolationException(); } • Assumptions and assertions are explored just like all other branches; Pex will try to break them! • Executions which cause assumption violations are ignored, not reported as errors or test cases

22. Limitations • Assumption: Environment is deterministic • "Environment" includes all code that is not monitored, e.g. native code, uninstrumented code, code that executes on another machine on the network… • Pex simply ignores non-deterministic behavior • Assumption: Program is single-threaded • Limitations of constraint solver • Little support for floating point arithmetic • approximation with rationals (linear arithmetic only) • No support for System.Decimal arithmetic • Bounds on time and memory consumption • http://research.microsoft.com/projects/Z3/

23. Summary • Pex automates test input generation for .NET programs • Pex enables Parameterized Unit Testing • Used in Microsoft to test core .NET components • Pexis publicly available for academic use. • Watch our website for future announcements.http://research.microsoft.com/Pex