Doing more with contracts Bertrand Meyer, Ilinca Ciupa, Andreas Leitner, Lisa Liu

1. Doing more with contracts Bertrand Meyer, Ilinca Ciupa, Andreas Leitner, Lisa Liu SOFSEM, Harrachov, January 2007 ETH Zurich, Chair of Software Engineering

2. The three questions • What does it expect? • What does it promise? • What does it maintain?

3. Progress in software quality • Tools, techniques, methods, languages, • software quality in general… … have been improving steadily… … but not enough! Market forces work not for very high quality but for good enough software Collective decision to turn IT into a service business

4. Where best to apply our efforts? End-user applications Specialized components Relies on General-purpose components Compilers & operating systems

5. Components should be contracted • Definition of what each element of the functionality: • Expects (precondition) • Promises (postcondition) • Maintains (invariant) • Does not have to be complete (but wait)

6. Design by Contract • Every software element is intended to satisfy a certain goal, for the benefit of other software elements (and ultimately of human users). • This goal is the element’s contract. • The contract of any software element should be • Explicit • Part of the software element itself

7. A class with contracts • class • BANK_ACCOUNT • create • make • feature make (n: STRING) -- Set up with namen require n /= Void do name := n balance := 0 ensure name = n end • name : STRING • balance : INTEGER • deposit ( v : INTEGER) • -- Add amountv • do • balance := balance + v • ensure • balance = old balance + v • end • invariant • name /= Void • balance >= 0 • end require n /= Void ensure balance = oldbalance + v invariant name /= Void balance >= 0 ensure name = n

8. The correctness of a class • For every creation procedurecp :{precp} docp{INV andpostcp} • For every exported routiner :{INVandprer} dor{INV andpostr} createa.make (…) S1 a.f (…) S2 a.g (…) S3 a.h (…) S4

9. What we do with contracts today • Write better software • Analysis • Design • Reuse • Implementation • Bug avoidance • Document software automatically • Help project managers do their job (with run-time monitoring) • Perform systematic testing • Guide the debugging process

10. Contracts for analysis, specification • deferred classVATinherit • TANK • feature • in_valve, out_valve: VALVE • fill is-- Fill the vat. • requirein_valve.open • out_valve.closed • deferredensurein_valve.closed • out_valve.closed • is_full • end • empty, is_full, is_empty, gauge, maximum, ... [Other features] ... • invariant • is_full = (gauge >= 0.97 *maximum)  and  (gauge <= 1.03 *maximum) • end

11. Seamless development with Eiffel & contracts Example classes: • Single notation, tools, concepts, principles • Continuous, incremental development • Keep model, implementation, documentation consistent • Reversibility PLANE, ACCOUNT, TRANSACTION… Analysis STATE, COMMAND… Design Implemen-tation HASH_TABLE… TEST_DRIVER… V&V Generali-zation TABLE…

12. Documentation: the contract view of a class • class • BANK_ACCOUNT • create • make • feature make (n: STRING) -- Set up with namen require n /= Void do name := n balance := 0 ensure name = n end • name : STRING • balance : INTEGER • deposit ( v : INTEGER) • -- Add amountv • do • balance := balance + v • ensure • balance = old balance + v • end • invariant • name /= Void • balance >= 0 • end require n /= Void ensure balance = oldbalance + v invariant name /= Void balance >= 0 ensure name = n

13. Lists with cursors before after "Prague" 1 count Cursor forth index

14. Command “forth″

15. Where the cursor may go before after item 0 1 count count+1 Valid cursor positions

16. From the invariant of class LIST Valid cursor positions

17. B D A C Inheritance Client Contracts and inheritance ris require  a1: A … ensure a1.r (…)  Correct call in C: ifa1.then a1.r (...) -- Herea1.holds end r ++ ris require  ensure  ++Redefinition

18. Assertion redeclaration rule • When redeclaring a routine, we may only: • Keep or weaken the precondition • Keep or strengthen the postcondition

19. Assertion redeclaration rule in Eiffel • A simple language rule does the trick! • Redefined version may have nothing (assertions kept by default), or • require elsenew_pre • ensure thennew_post • Resulting assertions are: • original_preconditionornew_pre • original_postconditionandnew_post

20. Contracts as a management tool • High-level view of modules for the manager: • Follow what’s going on without reading the code • Enforce strict rules of cooperation between units of the system • Control outsourcing

21. Contracts for testing and debugging • Contracts provide the right basis: • Testing is there to find bugs • A bug is a discrepancy between intent and reality • Contracts describe intent • A contract violation always signals a bug: • Precondition violation: bug inclient • Postcondition violation: bug inroutine • In EiffelStudio: select compilation option for run-time contract monitoring at level of class, cluster or system.

22. Anecdotal & non-anecdotal evidence • HP 1: • invariant • r = 2 ^ i • HP 2: Eiffel messes up our great system! • Axa Rosenberg: postcondition fails in deep_clone of TWO_WAY_LIST ! • Patrice Chalin study (Concordia): Eiffel programmers do use contracts day in, day out.

23. Should we test software? • Some evidence of why we should: • Limitations of proof technology • Properties not formally specified • Tests as complement of proofs?

24. Testing is tedious From a survey of 240 companies in N. America and Europe: • 8% release software to beta sites without testing • 83% of their developers don't like to test code. • 53% don't like to test their own code because they find it tedious. • 30% don't like to test because they find testing tools inadequate.

25. “Automated testing” • What can be automated: • Test execution • Robust execution • Regression testing • Test case generation (test suites) • Test result verification (test oracles) • Test scheduling • Test case minimization

26. Automated vs. manual unit tests Automatically generated test cases complement manual ones: • Contracts may be “wrong” • Implementation is unlikely to establish postcondition on input satisfying A -> ¬B . • Test cases will likely only be written for B (and not for A -> ¬B ).

27. AutoTest: an automatic test framework • Input is a set of classes • AutoTest generates instances and calls features with automatically selected arguments • Automatic tests rely on contracts: • Precondition violations: skip • Postcondition/invariant violation:bingo! • Manual tests can be added explicitly: • Relevant ones are determined automatically • Any test (manual or automated) that fails becomes part of the test suite

28. AutoTest demo

29. Automated testing and slicing • auto_test system.ace BANK_ACCOUNT STRING create {STRING} v1 v1.wipe_out v1.append_character (’c’) v1.append_double (2.45) create {STRING} v2 v1.append_string (v2) v2.fill (’g’, 254343) ... create {BANK_ACCOUNT} v3.make (v2) v3.deposit (15) v3.deposit (100) v3.deposit (-8901) ... • class • BANK_ACCOUNT • create • make • feature • make (n: STRING) • require • n /= Void • do • name := n • balance := 0 • ensure • name = n • end • name : STRING • balance : INTEGER • deposit (v : INTEGER) • do • balance := balance + v • ensure • balance = • oldbalance + v • end • invariant • name /= Void • balance >= 0 • end

30. Robust execution

31. Some AutoTest results (random strategy) ROUTINES TESTS

32. Result analysis • (Raluca Borga, U. of Cluj-Napoca, diploma work at ETH) • Fault classification (manual): • Specification error • Implementation error • Inheritance • Don’t know • About 50% implementation, 50% specification • Results polluted by “void” issue (Will no longer exist thanks to attached types, see ECOOP 2005 paper)

33. AutoTest strategies • Random strategy • Use random input • Planning strategy • Employ information from postcondition to satisfy preconditions • ...

34. Basic strategy • Object pool • For each type, initialize through creation procedures (constructors) • This requires proper ordering of classes • Diversify through procedures • Routine arguments • Basic values: heuristics for each type • Objects: get from pool • Invoke selected routines • AutoTest tests all routines, including inherited ones (“Fragile base class” issue)

35. n n m m Adaptive Random Testing (ART, Chen et al.) • Basic idea: random testing may find bugs faster if inputs evenly spread over range of possible values • Example: • r (m, n: INTEGER) do … end ART Random inputs

36. Extending ART to objects • So far ART only applied to numeric inputs, on which a total order relation exists • Multi-field objects are not members of a totally ordered set • Need to define notion ofdistancebetween objects

37. Δ = Object distance (Ilinca Ciupa) • p ↔ q • combination ( • type_distance(p.type, q.type), • field_distance(p, q), • recursive_distance( {[p.r ↔ q.r] | r  Reference_attributes } )

38. The role of argument-less boolean queries • Externally, boolean queries often serve as preconditions and appear in invariants • Internally, boolean queries often serve as branch conditions • Lists:before, after, off, is_first, is_last, is_empty… • Account:is_overdraft… before after is_last is_first "Prague" 1 count Cursor index

39. Boolean query conjecture The argument-less boolean queries of a well-written class yield a partition of the object state space that helps the testing process.

40. Boolean query coverage (Lisa Liu) A set of tests for a class satisfies boolean query coverage if and only if the execution of these tests can cover all the reachable abstract object states for that class. • Strategy: • Through constraint solving, generate states that satisfy boolean queries in invariant (e.g. afterimpliesoff) • Through theorem proving, remove states that do not satisfy rest of invariant (e.g. count<capacity)

41. before after is_last is_first "Prague" 1 count Cursor index

42. Experimental results

43. AutoTest developments • Large-scale extensive tests (cluster computing) • Comparison with manual efforts • Scaling up (memory, time) (mostly done) • Rigorous assessment of testing criteria & strategies • Complete integration with EiffelStudio environment • Background, unobtrusive, continuous testing • Automatic regression testing • Distributed cooperative testing (“Testi@home”)

44. Towards a real framework • Pluggable strategies • Flexible execution mechanism • Robust execution mechanism • Unified interface • Integrate tightly with manual testing • Concise output

45. first right right right Proving classes • class • LINKED_LIST [G] • feature • … • remove_front • -- Remove first item • require • not empty • do • first := first.right • ensure • count = oldcount – 1 • first = olditem (2) • end • … • end

46. Issues in proving classes • Providing full contracts • Devising models of the underlying world • Devising proof rules (i.e. semantics of the programming language) • Taking advantage of inheritance on the proof side • Doing the proofs, mechanically

47. The contract language • In Eiffel (and other formalisms based on Design by Contract): mostly boolean expressions of the language, plus some extensions such as old. • Seems not expressive enough — but is!

48. Beefing up expressive power Components to prove(e.g. EiffelBase) Eiffel Model Library

49. Eiffel Model Library (also known as MML) • (With Bernd Schoeller, Tobias Widmer) • Classes correspond to mathematical concepts: • SET [G], FUNCTION [G, H ], TOTAL_FUNCTION [G, H ], RELATION [G, H ], SEQUENCE [G ], … • Completely applicative: no attributes (fields), no implemented routines (all completely deferred) • Specified with contracts (unproven) reflecting mathematical properties • Expressed entirely in Eiffel

50. Example MML class • classSEQUENCE [G] feature • count : NATURAL • -- Number of items • last : G • -- Last item • extended (x): SEQUENCE [G] • -- Identical sequence except x added at end • ensure • Result.count = count + 1 • Result.last = x • Result.sub (1, count) ~ Current • mirrored : SEQUENCE [G] • -- Same items in reverse order • ensure • Result.count = count • … • … • end