1 / 38

Pex White Box Test Generation for .NET Nikolai Tillmann, Microsoft Research

Pex White Box Test Generation for .NET Nikolai Tillmann, Microsoft Research Evolutionary Testing, Dagstuhl , August 28 th , 2008. Background: Unit Testing. A unit test is a small program with assertions. void AddTest () { HashSet set = new HashSet (); set.Add (7); set.Add (3);

brooke
Download Presentation

Pex White Box Test Generation for .NET Nikolai Tillmann, Microsoft Research

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. PexWhite Box Test Generation for .NETNikolai Tillmann, Microsoft Research Evolutionary Testing, Dagstuhl, August 28th, 2008

  2. Background: Unit Testing A unit test is a small program with assertions. void AddTest() { HashSet set = new HashSet(); set.Add(7); set.Add(3); Assert.IsTrue(set.Count == 2); } Many developers write such unit tests by hand. This involves • determining a meaningful sequence of method calls, • selecting exemplary argument values (the test inputs), • stating assertions.

  3. Vision: Parameterized Unit Testing void AddSpec(int x, int y) { HashSet set = new HashSet(); set.Add(x); set.Add(y); Assert.AreEqual(x == y, set.Count == 1); Assert.AreEqual(x != y, set.Count == 2); } Parameterized Unit Tests separate two concerns: • The specification of externally visible behavior (assertions) • The selection of internally relevant test inputs (coverage) Parameterized Unit Testingare algebraic specifications!

  4. Pex: Overview • Purpose: Test input generator • Starting from parameterized unit tests • Generated tests emitted as traditional unit tests • Goal: Test suite that covers all reachable statements • Technology: Dynamic symbolic execution • Whole-program, white-box code analysis • At the level of the .NET instructions (bytecode) • Symbolic execution based on monitoring and re-execution • Path conditions obtained by monitoring dataflow, branches • Iterative selection of potential execution paths • Solving path conditions with SMT solver

  5. Dynamic Symbolic Execution By Example Choose next path Solve Execute&Monitor Code to generate inputs for: Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==123456890 Observed constraints a==null a!=null && !(a.Length>0) a==null && a.Length>0 && a[0]!=1234567890 a==null && a.Length>0 && a[0]==1234567890 Input null {} {0} {123…} void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } a==null T F Done: There is no path left. a.Length>0 T F a[0]==123… F T

  6. Demo: Parameterized Unit Test

  7. Demo: Pex Exploration Results

  8. Demo: Generated Unit Tests Generated Test Inputs are persisted as C# Unit Tests

  9. Going from an academic example Into the real world How to test this code? (Actual code from .NET base class libraries.)

  10. Real World (II)

  11. Real World (III) • There is no specification. • Most basic test oracle: • Code should not crash, i.e. throwing unexpected exceptions Possible test case, written by Hand

  12. Pex – Demo Test input, generated by Pex 12

  13. Test Input Generation byDynamic Symbolic Execution Initially, choose Arbitrary Run Test and Monitor Solve TestInputs Constraint System Execution Path KnownPaths Choose an Uncovered Path Record Path Condition

  14. Test Input Generation byDynamic Symbolic Execution a[0] = 0; a[1] = 0; a[2] = 0; a[3] = 0; … Initially, choose Arbitrary Run Test and Monitor Solve TestInputs Constraint System Execution Path KnownPaths Choose an Uncovered Path Record Path Condition

  15. Test Input Generation byDynamic Symbolic Execution Initially, choose Arbitrary • Path Condition: • … ⋀ magicNum != 0x95673948 Run Test and Monitor Solve TestInputs Constraint System Execution Path KnownPaths Choose an Uncovered Path Record Path Condition

  16. Test Input Generation byDynamic Symbolic Execution Initially, choose Arbitrary Run Test and Monitor • … ⋀ magicNum != 0x95673948 • … ⋀ magicNum == 0x95673948 Solve TestInputs Constraint System Execution Path KnownPaths Choose an Uncovered Path Record Path Condition

  17. Test Input Generation byDynamic Symbolic Execution a[0] = 206; a[1] = 202; a[2] = 239; a[3] = 190; Initially, choose Arbitrary Run Test and Monitor Solve TestInputs Constraint System Execution Path KnownPaths Choose an Uncovered Path Record Path Condition

  18. Test Input Generation byDynamic Symbolic Execution Initially, choose Arbitrary Run Test and Monitor Solve TestInputs Constraint System Execution Path KnownPaths Choose an Uncovered Path Record Path Condition

  19. Test Input Generation byDynamic Symbolic Execution Initially, choose Arbitrary Run Test and Monitor Solve TestInputs Constraint System Execution Path KnownPaths Choose an Uncovered Path Record Path Condition Finds only real bugs No false warnings Result: small test suite, high code coverage

  20. Constraint Solving: Z3 • SMT-Solver (“Satisfiability Modulo Theories”) • Decides logical first order formulas with respect to theories • SAT solver for Boolean structure • Decision procedures for relevant theories: uninterpreted functions with equalities,linear integer arithmetic, bitvector arithmetic, arrays, tuples • Model generation for satisfiable formulas • Models used as test inputs • Incremental solving • Enables efficient model minimization

  21. Enumerating Paths Problem: • Reachable code not known initially • Without guidance, same loop might be unfolded forever

  22. Enumerating Paths:Search Strategies Solution: • A strategy selects an already known feasible path prefix, after which to take the next branch not yet seen • Meta-strategy: Fair choice between different strategies • Individual strategies reflect common program patterns: • Try to go along each branch once • If all have been tried once, then twice, etc. • Try to construct paths that go along all pairs of branches • If all pairs have been tried, try all triples, etc. • If all branches have been considered, use more fine grained partitioning that also considers call stacks • …

  23. A case study (I) • We applied Pex on a core .NET component • Already extensively tested for several years • Assertions written by developers • >10,000 public methods • >100,000 basic blocks • Sandbox • Restriction of access to external resources (files, registry, unsafe code, …) • 10 machines (P4, 2Ghz, 2GB RAM) running for 3 days • Exploration started from simple, generated parameterized unit tests (one per public method); assertions embedded in code

  24. A case study (II) Coverage achieved: • 43% block coverage • 36% arc coverage Errors found: • A significant number of benign errors, e.g. NullReferenceException, IndexOutOfRangeException, … • 17 unique errors involving • violation of developer-written assertions, • exhaustion of memory, • other serious issues.

  25. A case study (III) • Automatically achieved coverage on selected classes for core .NET component

  26. Limitations • Assumption: Environment is deterministic • "Environment" includes all code that is not monitored, e.g. native code, uninstrumented code • Pex prunes non-deterministic behavior • Assumption: Program is single-threaded • Potential solution: control and explore thread scheduling like all other test inputs • Limitations of constraint solver • Z3 has no built-in theories for floating point arithmetic • approximation with rationals (linear arithmetic only) • Bounds on Z3's time and memory consumption

  27. Open and not fully solved problems / Current and future work • Search strategies to enumerate paths (ET?) • Determining method sequences to configure private fields (ET? Yes!) • Summarizing portions of execution treewhile still allowing flexible search strategies • Automatic inference of symbolic environment models • Beyond coverage metrics: Measuring quality of tests • Dealing with multithreaded programs

  28. Some Recent Related Work • Program model checkers • JPF, Kiasan/KUnit (Java), XRT (.NET) • Combining random testing and constraint solving • DART (C), CUTE (C), EXE (C), • jCUTE(Java), • SAGE (X86) • …

  29. Summary • Parameterized Unit Tests separate two concerns • Specification of externally visible behavior • Selection of test inputs to cover internal behavior • Pex automates test input generation • Dynamic Symbolic Execution platform for .NET • Custom strategies can be plugged in • Used in Microsoft to test core .NET components • Opportunities for Evolutionary Algorithms • Pex is publicly available for academic use. http://research.microsoft.com/Pex

  30. Thank you http://research.microsoft.com/Pex

  31. Test Input Generation byDynamic Symbolic Execution Constraint solving is just hard. Initially, choose Arbitrary TestInputs Constraint System Execution Path KnownPaths Reachability is undecidable! (Loops)

  32. Assumptions and Assertions void PexAssume.IsTrue(bool c) { if (!c) throw new AssumptionViolationException(); } void PexAssert.IsTrue(bool c) { if (!c) throw new AssertionViolationException(); } • Assumptions and assertions are explored just like all other branches • Executions which cause assumption violations are ignored, not reported as errors or test cases

  33. Symbolic State Representation Similar to representation of verification conditions in ESC/Java, Spec#, … Terms for • Primitive types (integers, floats, …) • Constants • Unary and binary expressions • ‘struct’ types • Tuples • Instance fields of classes • Mapping of references to values • Elements of arrays, memory accesses through pointers • Mapping of integers to values • …

  34. Term normalization Goal: Efficient representation of evolving program states • Reduction of ground terms to constants • Sharing of syntactically equal sub-terms • BDDs over if-then-else terms to represent logical operations • Tries/Patricia Trees to representassociative-commutative-with-unit operators • Normal form of polynomials • Update trees • Other simplification rules, e.g. • \forallx. ceq(vtable(x, m1), m2) => ceq(objecttype(x), t)where m2 overrides m1, and t is the sealed declaring type of m2

  35. Constraint Solving: Preprocessing Independent constraint optimization + Constraint caching (similar to EXE) • Idea: Related execution paths give rise to "similar" constraint systems • Example: Consider x>y⋀ z>0 vs. x>y⋀ z<=0 • If we already have a cached solution for a "similar" constraint system, we can reuse it • x=1, y=0, z=1 is solution for x>y⋀z>0 • we can obtain a solution for x>y⋀z<=0by • reusing old solution of x>y: x=1, y=0 • combining with solution of z<=0: z=0

  36. Why Dynamic Symbolic Execution? • Most interesting programs are beyond the scope of static symbolic execution. • Dynamic symbolic execution will systematically explore the conditions in the code which the constraint solver understands. • And happily ignore everything else, e.g. • Calls to native code • Difficult constraints (e.g. precise semantics of floating point arithmetic) • Result: Under-approximation, which is appropriate for testing Calls to external world Unmanaged x86 code Unsafe managed .NET code (with pointers) Safe managed .NET code

  37. Historical note: XRT XRT: Exploring Runtime • Interpreter for .NET programs • Static symbolic execution • Used Simplify to determine unsatisfiability of path constraints • Successful for self-contained programs • Used today on a large scale within Microsoft for quality assurance purposes as the core of the model-based testing tool “Spec Explorer 2007”. • Does not work well for real-world programs • All environment behavior must be modeled • Modeling of entire environment is often not feasible

  38. Creating complex objects Problem: • Pex can collect constraints over private fields, constraint solver determines assignment for private fields • How to bring object into desired state? • Private fields cannot be initialized freely, but only through constructor and other methods Approach taken by Pex: • Automatic selection of constructor and state-modifying methods based on static code analysis • Exploration of constructor and methods to find non-exceptional paths

More Related