370 likes | 604 Views
TL51. Contract Checking and Automated Test Generation With Pex. Mike Barnett RSDE Microsoft Research. Nikolai Tillmann RSDE Microsoft Research. Two Tools. One Talk. Code Contracts F or .NET. Mike Barnett RSDE Microsoft Research. Automated Test Generation W ith Pex.
E N D
TL51 Contract Checking and Automated Test Generation With Pex Mike BarnettRSDE Microsoft Research Nikolai TillmannRSDE Microsoft Research
Two Tools. One Talk. Code ContractsFor .NET Mike Barnett RSDE Microsoft Research Automated TestGeneration WithPex Nikolai Tillmann RSDE Microsoft Research
Contract Library in .NET 4.0 announcing Code Contracts for .NET Coming soon: Tools for contracts at
How Do You Know What It Does? string s = o.GetDescription(i); … s.Length … stringGetDescription(int x); /// <param name=“x”> /// should be greater than zero /// </param> /// <returns> /// non-null string /// </returns> stringGetDescription(int x);
Let’s Fix It For Once And All! Design-by-Contract meets .NET! stringGetDescription(int x) { Contract.Requires( 0 < x ); Contract.Ensures(Contract.Result<string>() != null ); … }
Contracts • Contracts document design decisions • Preconditions, postconditions, invariants • Facilitate code evolution • Contracts are executable • Checked documentation • Test oracles • Contracts help static verification • Early error detection • Clarify responsibility at API boundaries • Assertions help, but everyone has their own flavor • Difficult to tool against • Not seen as a contract between 2 parties
demo Introduction Mike Barnett RSDE Research in Software Engineering
What Can I Do With Contracts? classRational {public Rational(int n, int d) {Contract.Requires( 0 < d );this.N = n;this.D = d; }} Static Checking Runtime Checking Test Generation (Pex) Documentation
What Contracts Can I Write? • Requires • What must be true at method entry • Ensures • What must be true at method exit • Invariants • What must be true at all method exits • Assertions • What must be true at a particular point • Assumptions • What should be true at a particular point
What Can I Put In A Contract? • Any boolean expression • In your favorite programming language! • Including method calls (but must be marked Pure) • Quantifiers • Contract.ForAll(0,A.Length,i => A[i] > 0); • Contract.Exists(0,A.Length,i => A[i] > 0); • Contract.Result • refer to the return value of the method • Contract.OldValue • refer to values at method entry
How Do I Write A Contract? publicvirtualint Add(object value){ Contract.Requires( value != null );Contract.Ensures( Count == Contract.OldValue(Count) + 1 ); Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) );if(count == items.Length) EnsureCapacity(count + 1); items[count] = value; returncount++;} • Features • Declarative • Language expression syntax • Type checking / IDE • Special Encodings • Result and OldValue voidObjectInvariant() {Contract.Invariant( items != null );}
demo Runtime Checking Mike Barnett RSDE Research in Software Engineering
How Does It Work? publicvirtualint Add(object value){ Contract.Requires( value != null ); Contract.Ensures( Count == Contract.OldValue(Count) + 1 ); Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) ); if (_size == _items.Length) EnsureCapacity(_size+1); _items[_size] = value; return _size++; } Executable Runtime Contract Checking .method public hidebysignewslot virtual instance int32 Add(object 'value') cil managed { .locals init (int32 'Contract.Old(Count)', int32 'Contract.Result<int>()') ldarg.0 call instance int32 TabDemo.BaseList::get_Count() stloc.3 ldarg.1 ldnull ceq ldc.i4.0 ceq ldstr "value != null" call void __RewriterMethods::RewriterRequires$PST06000009(bool, string) ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.0 ldfld object[] TabDemo.BaseList::items ldlen conv.i4 ceq ldc.i4.0 ceq stloc.1 ldloc.1 brtrue IL_004d nop ldarg.0 ldarg.0 ldfld int32 TabDemo.BaseList::count ldc.i4.1 add call instance void TabDemo.BaseList::EnsureCapacity(int32) nop nop ldarg.0 ldfld object[] TabDemo.BaseList::items ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.1 stelem.ref ldarg.0 dup ldfld int32 TabDemo.BaseList::count dup stloc.2 ldc.i4.1 add stfld int32 TabDemo.BaseList::count ldloc.2 stloc.0 br IL_0072 ldloc.0 stloc.s 'Contract.Result<int>()' br IL_007a ldarg.0 call instance int32 TabDemo.BaseList::get_Count() ldloc.3 ldc.i4.1 add ceq ldstr "Count == Contract.Old(Count) + 1" call void __RewriterMethods::RewriterEnsures$PST0600000B(bool, string) ldloc.s 'Contract.Result<int>()' ldloc.s V_4 ceq ldstr "Contract.Result<int>() == Contract.Old(Count)" call void __RewriterMethods::RewriterEnsures$PST0600000B(bool, string) ldloc.s 'Contract.Result<int>()' ret } csc/vbc/… .method public hidebysignewslot virtual instance int32 Add(object 'value') cil managed { ldarg.1 ldnull ceq ldc.i4.0 ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Requires(bool) ldarg.0 call instance int32 TabDemo.BaseList::get_Count() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ldc.i4.1 add ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Result<int32>() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.0 ldfld object[] TabDemo.BaseList::items ldlen conv.i4 ceq ldc.i4.0 ceq stloc.1 ldloc.1 brtrue.s IL_0069 ldarg.0 ldarg.0 ldfld int32 TabDemo.BaseList::count ldc.i4.1 add call instance void TabDemo.BaseList::EnsureCapacity(int32) ldarg.0 ldfld object[] TabDemo.BaseList::items ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.1 stelem.ref ldarg.0 dup ldfld int32 TabDemo.BaseList::count dup stloc.2 ldc.i4.1 add stfld int32 TabDemo.BaseList::count ldloc.2 stloc.0 br.s IL_008b ldloc.0 ret } // end of method BaseList::Add ReleaseCompile /d:CONTRACTS_FULL csc/vbc/… ccrewrite .method public hidebysignewslot virtual instance int32 Add(object 'value') cil managed { ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.0 ldfld object[] TabDemo.BaseList::items ldlen conv.i4 ceq ldc.i4.0 ceq stloc.1 ldloc.1 brtrue.s IL_0029 ldarg.0 ldarg.0 ldfld int32 TabDemo.BaseList::count ldc.i4.1 add call instance void TabDemo.BaseList::EnsureCapacity(int32) ldarg.0 ldfld object[] TabDemo.BaseList::items ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.1 stelem.ref ldarg.0 dup ldfld int32 TabDemo.BaseList::count dup stloc.2 ldc.i4.1 add stfld int32 TabDemo.BaseList::count ldloc.2 stloc.0 br.s IL_004b ldloc.0 ret } • Inheritance • Style checks
Static Contract Checking • No silver bullet • But helps catch errors earliest • Best used in a focused manner • Guides development • Discovers implicit assumptions • Propagates assumptions • Not only explicit contracts • Dereferencing null • Indexing arrays • Arithmetic exceptions
What Do You Ship? src src src src ReleaseAssemblies Contract ReferenceAssemblies PowerLib.dll PowerLib.Contracts.dll + (minimal runtime checks) All contracts, no code
Contract Reference Assemblies publicvirtualint Add(object value){ Contract.Requires( value != null ); Contract.Ensures( Count == Contract.OldValue(Count) + 1 ); Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) ); if (_size == _items.Length) EnsureCapacity(_size+1); _items[_size] = value; return _size++; } Contract ReferenceAssembly .method public hidebysignewslot virtual instance int32 Add(object 'value') cil managed { ldarg.1 ldnull ceq ldc.i4.0 ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Requires(bool) ldarg.0 call instance int32 TabDemo.BaseList::get_Count() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ldc.i4.1 add ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Result<int32>() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.0 ldfld object[] TabDemo.BaseList::items ldlen conv.i4 ceq ldc.i4.0 ceq stloc.1 ldloc.1 brtrue.s IL_0069 ldarg.0 ldarg.0 ldfld int32 TabDemo.BaseList::count ldc.i4.1 add call instance void TabDemo.BaseList::EnsureCapacity(int32) ldarg.0 ldfld object[] TabDemo.BaseList::items ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.1 stelem.ref ldarg.0 dup ldfld int32 TabDemo.BaseList::count dup stloc.2 ldc.i4.1 add stfld int32 TabDemo.BaseList::count ldloc.2 stloc.0 br.s IL_008b ldloc.0 ret } // end of method BaseList::Add .method public hidebysignewslot virtual instance int32 Add(object 'value') cil managed { ldarg.1 ldnull ceq ldc.i4.0 ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Requires(bool) ldarg.0 call instance int32 TabDemo.BaseList::get_Count() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ldc.i4.1 add ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Result<int32>() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) } // end of method BaseList::Add ccRefGen Compile /d:CONTRACTS_FULL
Interface Contracts [ContractClass(typeof(CloneableContract))]publicinterfaceICloneable { object Clone(); } [ContractClassFor(typeof(ICloneable))]publicclassCloneableContract : ICloneable { publicobject Clone() {Contract.Ensures( Contract.Result<object>() != null ); … } } All classes implementing the interface inherit the contract
Code Contracts Summary • Contract library class enables contracts in all .NET languages • No restrictions on what can be expressed • Contracts are being used in the BCL today • Contract library is a core component of .NET 4.0 • Same contracts used for • Runtime checking • Static checking • Documentation generation
Get Pex for Visual Studio 2010 at announcing Do you care about quality?Shake your code with Pex. Pex
Testing Before Pex • Testing is tedious • Too easy to miss cases • Old tests get stale • Too much legacy code − what does it do? Don’t give up, Pex has arrived.
demo Tests for Freewith Pex Pex Nikolai Tillmann RSDE Microsoft Research
What The Demo Showed • It’s painless: Just right-click your code • It’s effective: Test cases for free • It’s fun: • Inspect table, • Save tests, • file work items, or • debug right away
demo Code Coverage for Free with Pex Pex Nikolai Tillmann RSDE Microsoft Research
What The Demo Showed • Pex automatically generatedcomprehensive test suitewith high code coverage • Pex finds contract violations • The generated test suite • integrates withVisual Studio Team Test • can be used asBuild Verification Test (BVT)
There Is More… • A traditional Unit Test is amethod without parameters. Handwritten [TestMethod] voidAddZeroTest() { var r = new Rational(1,2); var q = r + 0; Assert(r == p); }
There Is More…Parameterized unit testing with pex • A Parameterized Unit Test is simply atest method that takes parameters • Pex generates traditional Unit Tests (in C#) which fix particular values Handwritten Automatically generated Pex [PexMethod] voidAddZeroTest( Rational r) { var q = r + 0; Assert(r == p); } [TestMethod] void AddZeroTest01() { var r = new Rational(1,2); this.AddZeroTest(r); } calls
A Tester's Nightmare: ResourceReader • How to test this code?(Actual code from .NET base class libraries)
Parameterized Unit Tests • [PexAllowedException(typeof(ArgumentException))] • [TestClass] • public partial classResourceReaderTest{ • [PexMethod] • public void ReadEntries(byte[] data) { • if (data == null) return; • fixed (byte* p = data) • using (varstream = • new UnmanagedMemoryStream(p, data.Length)) { • varreader = new ResourceReader(stream); • foreach(varentry in reader) • { /* reading entries */ } • } • } • }
demo Parameterized Unit Testing Pex Nikolai Tillmann RSDE Microsoft Research
PexUnderstands The Code • Pex does not guess. • Pex does not generate random inputs, • enumerateall possible values, or • make you write test input generators • Instead, Pex analyzes your .NET code. • Pexpartitions inputs into equivalence classes • One equivalence class per branching behavior • Test inputs computed by Z3, the world classconstraint solver for program analysisfrom Microsoft Research • Precise inter-procedural, path-sensitiveanalysis • As a result, you geta small test suite with high code coverage
Pex Summary • Pex generates small test suites with high code coverageand bug reports for free • Use previously generated test suites asBuild Verification Test (BVT) • Reduce test maintenance costsby parameterized unit testing • Pex has been used in Microsoftto test core .NET components • Download Pex for Visual Studio 2010 CTP onhttp://msdn.microsoft.com/DevLabs
Summary • Code Contracts for .NET:preconditions, postconditions, invariants http://research.microsoft.com/Contracts/ • Pex: test generation for .NET http://research.microsoft.com/Pex/ Research in Software Engineering
Evals & Recordings Please fill out your evaluation for this session at: This session will be available as a recording at: www.microsoftpdc.com
Q&A Please use the microphones provided
© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.