Parameterized Unit Testing

ICTSS’11 Tutorial Parameterized Unit Testing • Nikolai Tillmann • Microsoft Research

Outline • First part: 14:00 - 15:00pm • Second part: 15:30 - 17:00pm

Outline • Unit Testing • Isolation • Parameterized Unit Testing • Data Generation byDynamic Symbolic Execution • Patterns • Limitations and other Details

About the exercises… • Interactive: http://pexforfun.com • Demos: Pex for Visual Studiohttp://research.microsoft.com/pex • Requires Windows, .NET 2.0 or 4.0, ideally Visual Studio 2008 / 2010

Unit Testing

Quiz: Unit testing • What is a unit test?

Unit Testing • A unit test is a small program with assertions • Test a single (small) unit of code • Let’s play a game! void AddAndCount() { var list = new List(); list.Add(3); Assert.AreEqual(1, list.Count); }

The Code Under Test string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach(var line in lines){ intindex = line.IndexOf('='); string name = line.Substring(index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } } return null; } t:\myapp.ini A=B Foo=C C=D

Quiz: Coverage • How much block coverage do we need? • 50% • 80% • 100% • Block coverage alone is not enough

Quiz: Coverage [TestMethod] void ExistingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“Foo=b”}); Reader.ReadFooValue();} • Do we need more tests to get 100% cov.? [TestMethod] void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[0]); Reader.ReadFooValue();}

Quiz: Assertions • Why write Assertions (or not)? • Documentation • Double check your program • Please your manager • Prevent future bugs • Validate user inputs • Catch errors early

Quiz: Assertions • Which Assertions should you write? • Assert.IsTrue(value == “b”); • Assert.IsTrue(value == null); • Assert.IsTrue(String.IsNullOrEmpty(value)) • Assert.IsTrue(false); • No assertions [TestMethod] void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“a=b”}); string value = Reader.ReadFooValue(); Assert.IsTrue(????);}

Quiz: Coverage +Assertions • What gives you confidence in the code? • High coverage, no assertions • Low coverage, many assertions • High coverage, many assertions • Low coverage, no assertions • I wrote it

Quiz: Coverage +Assertions • What gives you confidence in the code? • High coverage, few assertions • Low coverage, many assertions • High coverage, many assertions • Low coverage, no assertions • I wrote it

The Code Under Test string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach(var line in lines){ intindex = line.IndexOf('='); string name = line.Substring(index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } } return null; }

Quiz: Isolation string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); ... } • In the example, what are the external dependencies? • Network Share • Local Disk • No file system, all in memory

Quiz: Isolation • What is the problem with a Local Disk? • Mapping already exists • Cannot run tests concurrently • Disk full • Access rights

Quiz: Isolation • How do you mitigate the Local Disk issues? • Always run on the same machine, same hardware, same credentials, same time, same temperature, same solar system configuration • Refactoring: use Streams • Refactoring: introduce IFileSystem • Refactoring: pass the lines as parameter • Change implementation of File.ReadAllLines

Quiz: Definition of Unit Test • What is a Unit Test? • A Unit Test is a program that runs fast the code under test, without environment dependencies, with assertions. • What is a Unit Test Suite? • A set of Unit Tests which achieves high code coverage

Isolation

Dependency hell var lines = File.ReadAllLines(@"t:\v.ini"); • How do you changeFile.ReadAllLines? • Override static method • Changing the .NET Framework source code(and recompiling it) • Rewrite application in JavaScript • Code instrumentation

Replace any .NET methodwith a delegate var lines = File.ReadAllLines(@"t:\myapp.ini"); What if we could replace File.ReadAllLines? File.ReadAllLines = delegate(string fn) MFile.ReadAllLinesString= delegate(string fn) { return new string[0]; }; Moles

Motivation for Moles • Why another isolation framework? • Specifically designed to enable Pex • Simple, well-defined semantics • “Replace any .NET method”

Mole Types Code Generation // System.IO public static class File{ public static string[] ReadAllLines(string fn);} // System.IO.Moles public class MFile{ public static Func<string, string[]> ReadAllLinesString; } // delegate R Func<T, R>(T t);

Injecting Detours at Runtime // System.IO public static class File { public static string[] ReadAllLines(string fn) { if (MFile.ReadAllLinesString != null) return MFile.ReadAllLines(fn); … original code } } Automatically injected at runtime

MolesDemo • Start fromReadFooValueproblem • Create test project • Add moles for mscorlib • Write test using moles • Run test • Debug test • You can pick up project fromReadFooValueIsolation\ReadFooValueIsolation.sln

Quiz: Moles Usage • When should you use Moles (and not)? • Always use Moles to solve isolation issues • With Moles, one does not need to use interfaces anymore • Moles only should be used for 3rd party API, use interfaces for isolation in your APIs • Moles can be used in production code • Moles lets you get away with untestable APIs • Moles make test cases more robust • With Moles, you do not need integration tests anymore • Moles make tests easier to understand • Moles is for poor programmers, real programmers rely on interfaces

Exercise (Optional / homework) • Step-by-step tutorial: • research.microsoft.com/documentation.aspx • “Unit Testing with Microsoft Moles” • “Unit Testing SharePoint Foundation with Microsoft Pex and Moles”

What you learnedso far • The Definition of Unit Testing • Unit Test Isolation • Identify External Dependencies • Replace each External API with your delegate

Parameterized Unit Testing

The Recipe of Unit Testing • Three essential ingredients: • Data • Method Sequence • Assertions void AddAndCount() { int item = 3; var list = new List(); list.Add(item); var count = list.Count; Assert.AreEqual(1, count); }

Quiz: list.Add(???) list.Add(???); • Which value matters? • 0 • 1 • int.MaxValue, int.MinValue • -1 • 1000 • it does not matter • I don’t know until I read the code

Parameterized Unit Testing void AddAndCount(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); } • Parameterized Unit Test = Unit Test with Parameters • Separation of concerns • Specification of behavior • Data to achieve coverage for any list, for any item, … adding 1 item increments Count by 1

Parameterized Unit Tests areAlgebraic Specifications • A Parameterized Unit Test can be read as a universally quantified, conditional axiom. void ReadWrite(string name, string data) {Assume.IsTrue(name != null && data != null); Write(name, data);varreadData = Read(name); Assert.AreEqual(data, readData); } forall. string name, string data: name not_equal null and data not_equal null implies equals( ReadResource(name,WriteResource(name,data)), data)

Parameterized Unit Testingis going mainstream Parameterized Unit Tests (PUTs) commonly supported by various test frameworks • .NET: Supported by .NET test frameworks • http://www.mbunit.com/ • http://www.nunit.org/ • … • Java: Supported by JUnit 4.X • http://www.junit.org/ Generating test inputs for PUTs supported by tools • .NET: Supported by Microsoft Research Pex • http://research.microsoft.com/Pex/ • Java: Supported by AgitarAgitarOne • http://www.agitar.com/

Problem We cannot execute parameterized unit tests without data. Where does the data come from? • Random data generator • Real customer data • Ranges • Some values hand-picked by dev/tester

Data Generationby Dynamic Symbolic Execution

Data Generation Challenge Goal: Given a program with a set of input parameters, automatically generate a set of input values that, upon execution, will exercise as many statements as possible How would you do it?

Dynamic Symbolic ExecutionCombines many ideas Combines ideas from • symbolic execution • to capture behavior symbolically • model checking • to perform systematic bounded state space exploration • testing • to monitor and check concrete behavior • constraint solving • to find inputs

Dynamic Symbolic ExecutionThe high-level algorithm • This choice decides search order • Search order decides how quickwe can achieve high code coverage! • Incomplete constraint-solver leads tounder-approximation Combines concrete and symbolic execution Algorithm: Set J := ∅ (J is set of already analyzed program inputs) Loop Choose program input i ∉ J (stop if no such i can be found) Output i Execute P(i); record path condition C (in particular, C(i) holds) Set J := J ∪ C (viewing C as the set { i | C(i ) } ) End loop Loop does not terminate if number of execution paths is infinite (in the presence of loops/recursion)

Dynamic Symbolic ExecutionExample Choose next path • Code to generate inputs for: Solve Execute&Monitor void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } a==null F T a.Length>0 T F Done: There is no path left. a[0]==123… F T Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null && a.Length>0 && a[0]==1234567890 Data null {} {0} {123…} Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==1234567890

http://pex4fun.com/CoverMe

Parameterized Unit Testing