Parameterized Unit Testing

1. Pex Class Parameterized Unit Testing • Nikolai Tillmann • Microsoft Research

2. Outline • Unit Testing • Coverage • Assertions • Isolation • Parameterized Unit Testing • Data Generation by Dynamic Symbolic Execution • Using Pex • Patterns • Wrapping up

3. About Pex… • Pex generates tests for .NET applications • Pex includes Moles, a framework to isolate code • Pex is a Visual Studio Power Tool(separate download, academic/commercial license available) • Minimum: Windows XP, .NET 2.0 • Ideally: Visual Studio 2010 Professional • http://research.microsoft.com/pex/downloads.asp

4. Unit Testing

5. Quiz: Unit testing • What is a unit test?

6. Unit Testing • A unit test is a small program with assertions • Test a single (small) unit of code • Let’s play a game! void AddAndCount() { var list = new List(); list.Add(3); Assert.AreEqual(1, list.Count); }

7. The Code Under Test string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach(var line in lines){ intindex = line.IndexOf('='); string name = line.Substring(index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } } return null; }

8. Quiz: Coverage [TestMethod] void ExistingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“Foo=b”}); Reader.ReadFooValue();} • Do we need more tests to get 100% coverage? string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach(var line in lines){ intindex = line.IndexOf('='); string name = line.Substring(index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } } return null; }

9. Quiz: Coverage [TestMethod] void ExistingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“Foo=b”}); Reader.ReadFooValue();} • Do we need more tests to get 100% coverage? [TestMethod] void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[0]); Reader.ReadFooValue();}

10. Quiz: Coverage • How much statement coverage do we need? • 50% • 80% • 100% • Statement coverage alone is not enough

11. Quiz: Coverage • How much statement coverage do we need? • 50% • 80% • 100% • Statement coverage alone is not enough

12. Quiz: Assertions if (name == "Foo") { var value = line.Substring(index + 1); Debug.Assert(???); return value; } • Which are good Assertions in the Product? • Debug.Assert(true); • Debug.Assert(value != null); • Debug.Assert(value.Length == line.Length – (index + 1));

13. Quiz: Assertions if (name == "Foo") { var value = line.Substring(index + 1); Debug.Assert(???); return value; } • Which are good assertions in the product code? • Debug.Assert(true); • Debug.Assert(value != null); • Debug.Assert(value.Length==line.Length–(index+1));

14. Quiz: Assertions • Good assertions in test code? • Assert.IsTrue(value == “b”); • Assert.IsTrue(value == null); • Assert.IsTrue(String.IsNullOrEmpty(value)) • Assert.IsTrue(false); • No assertions [TestMethod] void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“a=b”}); string value = Reader.ReadFooValue(); Assert.IsTrue(????);}

15. Quiz: Assertions • Good assertions in test code? • Assert.IsTrue(value == “b”); • Assert.IsTrue(value == null); • Assert.IsTrue(String.IsNullOrEmpty(value)) • Assert.IsTrue(false); • No assertions [TestMethod] void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“a=b”}); string value = Reader.ReadFooValue(); Assert.IsTrue(????);}

16. Quiz: Assertions • Which Assertions should you write? • Assert.IsTrue(value == “b”); • Assert.IsTrue(value == null); • Assert.IsTrue(value == “Foo”); • No assertions. [TestMethod] void ExistingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“Foo=b”}); string value = Reader.ReadFooValue(); Assert.IsTrue(????);}

17. Quiz: Assertions • Which Assertions should you write? • Assert.IsTrue(value == “b”); • Assert.IsTrue(value == null); • Assert.IsTrue(value == “Foo”); • No assertions. [TestMethod] void ExistingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“Foo=b”}); string value = Reader.ReadFooValue(); Assert.IsTrue(????);}

18. Quiz: Assertions • Why write Assertions (or not)? • Documentation • Double check your program • Please your manager • Prevent future bugs • Validate user inputs • Catch errors early

19. Quiz: Assertions • Why write Assertions (or not)? • Documentation • Double check your program • Please your manager • Prevent future bugs • Validate user inputs • Catch errors early

20. Quiz: Coverage +Assertions • What gives you confidence in the code? • High coverage, few assertions • Low coverage, many assertions • High coverage, many assertions • Low coverage, no assertions • I wrote it

21. Quiz: Coverage +Assertions • What gives you confidence in the code? • High coverage, few assertions • Low coverage, many assertions • High coverage, many assertions • Low coverage, no assertions • I wrote it

22. The Code Under Test string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach(var line in lines){ intindex = line.IndexOf('='); string name = line.Substring(index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } } return null; }

23. Quiz: Isolation string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); ... } • In the example, what are the external dependencies? • Network Share • Local Disk • No file system, all in memory

24. Quiz: Isolation string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); ... } • In the example, what are the external dependencies? • Network Share • Local Disk • No file system, all in memory

25. Quiz: Isolation • What is the problem with a Local Disk? • Mapping already exists • Cannot run tests concurrently • Disk full • Access rights

26. Quiz: Isolation • What is the problem with a Local Disk? • Mapping already exists • Cannot run tests concurrently • Disk full • Access rights

27. Quiz: Isolation • How do you mitigate the Local Disk issues? • Always run on the same machine, same hardware, same credentials, same time, same temperature, same solar system configuration • Refactoring: use Streams • Refactoring: introduce IFileSystem • Refactoring: pass the lines as parameter • Change implementation of File.ReadAllLines

28. Quiz: Isolation • How do you mitigate the Local Disk issues? • Always run on the same machine, same hardware, same credentials, same time, same temperature, same solar system configuration • Refactoring: use Streams • Refactoring: introduce IFileSystem • Refactoring: pass the lines as parameter • Change implementation of File.ReadAllLines

29. A tester’s wish list • Code-under test should not depend on hard-coded environment dependencies: • Wouldn’t it be nice if we could change meaningof File.ReadAllLines for testing purposes? var lines = File.ReadAllLines(@"t:\myapp.ini"); File.ReadAllLines = delegate(string fileName) { return new string[]{“a=b”}; }

30. Molingmethods • Code-under test should not depend on hard-coded environment dependencies: • With Moles (part of Pex), we can do it:We detour all future calls to ReadAllLines to a delegate that returns {“a=b”}: var lines = File.ReadAllLines(@"t:\myapp.ini"); MFile.ReadAllLinesString = delegate(string fileName) { return new string[]{“a=b”}; }

31. Definition of Unit Test • What is a Unit Test? • A Unit Test is a program that exercises the code under test fast, without environment dependencies, with assertions. • What is a Unit Test Suite? • A set of Unit Tests which achieves high code coverage

32. Parameterized Unit Testing

33. The Recipe of Unit Testing • Three essential ingredients: • Data • Method Sequence • Assertions void AddAndCount() { int item = 3; var list = new List(); list.Add(item); var count = list.Count; Assert.AreEqual(1, count); }

34. Quiz: list.Add(???) list.Add(???); • Which value matters? • 0 • 1 • int.MaxValue, int.MinValue • -1 • 1000 • it does not matter • I don’t know until I read the code

35. Parameterized Unit Testing void AddAndCount(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); } • Parameterized Unit Test = Unit Test with Parameters • Separation of concerns • Specification of behavior • Data to achieve coverage

36. Parameterized Unit Tests areAlgebraic Specifications • A Parameterized Unit Test can be read as a universally quantified, conditional axiom. void ReadWrite(string name, string data) {Assume.IsTrue(name != null && data != null); Write(name, data);varreadData = Read(name); Assert.AreEqual(data, readData); } forall. string name, string data: name not_equal null and data not_equal null implies equals( ReadResource(name,WriteResource(name,data)), data)

37. Parameterized Unit Testingis going mainstream Parameterized Unit Tests (PUTs) commonly supported by various test frameworks • .NET: Supported by .NET test frameworks • http://www.mbunit.com/ • http://www.nunit.org/ • … • Java: Supported by JUnit 4.X • http://www.junit.org/ Generating test inputs for PUTs supported by tools • .NET: Supported by Microsoft Research Pex • http://research.microsoft.com/Pex/ • Java: Supported by AgitarAgitarOne • http://www.agitar.com/

38. Problem We cannot execute parameterized unit tests without data. Where does the data come from? • Random data generator • Real customer data • Ranges • Some values hand-picked by dev/tester

39. Data Generationby Dynamic Symbolic Execution

40. Data Generation Challenge Goal: Given a program with a set of input parameters, automatically generate a set of input values that, upon execution, will exercise all reachable statements How would you do it?

41. Dynamic Symbolic ExecutionAn iterative process • Code to generate inputs for: void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } Initially, choose Arbitrary TestInputs Constraint System Execution Path KnownPaths

42. Dynamic Symbolic ExecutionStep-by-step Choose next path • Code to generate inputs for: Solve Execute&Monitor void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } a==null F T a.Length>0 T F Done: There is no path left. a[0]==123… F T Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null && a.Length>0 && a[0]==1234567890 Data null {} {0} {123…} Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==1234567890

43. Dynamic Symbolic ExecutionExercise • What tests will Pex generate? Choose next path Solve Execute&Monitor void CoverMe2( int[] a, int i) { if (a[i] == 1 + a[i + 1]) throw new Exception("bug"); } F T T F F T Constraints to solve Observed constraints Data

44. www.pexforfun.com/coverme www.pexforfun.com/parameterizedunittesting Online Exercises

45. Dynamic Symbolic Executionwith Pex • Create new project in Visual Studio. • Insert CoverMe method below. • Right-click on CoverMe method, • and select “Run Pex”. • Inspect results in table. • Save tests, …, have fun! public static void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); }

46. Dynamic Symbolic ExecutionDemo

47. Dynamic Symbolic ExecutionDemo Generated Test Inputs are persisted as C# Unit Tests

48. Motivation: Unit Testing HellResourceReader • How to test this code?(Actual code from .NET base class libraries)

49. Motivation: Unit Testing HellResourceReader

50. DemoResourceReader [PexClass, TestClass] [PexAllowedException(typeof(ArgumentNullException))] [PexAllowedException(typeof(ArgumentException))] [PexAllowedException(typeof(FormatException))] [PexAllowedException(typeof(BadImageFormatException))] [PexAllowedException(typeof(IOException))] [PexAllowedException(typeof(NotSupportedException))] public partial class ResourceReaderTest { [PexMethod] public unsafe void ReadEntries(byte[] data) { PexAssume.IsTrue(data != null); fixed (byte* p = data) using (var stream = new UnmanagedMemoryStream(p, data.Length)) { var reader = new ResourceReader(stream); foreach (var entry in reader) { /* reading entries */ } } } }