Notes on: Is Proof More Cost-Effective Than Testing?

1. Notes on: Is Proof More Cost-Effective Than Testing? Prepared by Stephen M. Thebaut, Ph.D. University of Florida by Steve King, Jonathan Hammond, Rob Chapman, Andy Pryor

2. UK Defense Standard 00-55 • For procurement of safety critical software in defense equipment • Perceived emphasis on formal methods (formal specification and design, proofs of correctness) • Purpose of this paper: report on work showing that using formal methods on a large scale is practical and effective at both the specification and code level

3. SHOLIS • Ship Helicopter Operating Limits Information System – safety critical system aiding safe operation of helicopters on naval vessels • Z-based formal specification and SPARK programming language (subset of Ada) • Omitted features: gotos, aliasing, default parameters for subprograms, side-effects in functions, recursion, tasks, user-defined exceptions, exception handlers, & generics • simplification of other features

4. Proofs • Specification-Based • Approx. 150 proofs in 500 pages • Consistency of global variables and constants, existence of initial states and checking of preconditions • Code-Based • Approx. 9,000 verification conditions verified • Predicate transformsused for non-looping programs • Invariants used to prove partial (weak) correctness of looping programs + separate arguments for termination

5. Some Observations • Skills needed: knowledge of Z, proof by cases and contradiction, understanding of imperative programming constructs, familiarity with proof concepts such as loop invariants (not that much!) • Proof validation: formal peer + IV&V reviews; proofs were not inspected by the customer • Z proof appeared to be substantially more efficient at finding faults than the most efficient testing phase.

6. Some Observations (cont’d) • Code proofs appeared to be more efficient than unit testing, despite the fact that substantial amounts of unit testing were completed before the bulk of code proof started.

7. Specification using “Z” • Z is a mature and expressive notation for model-based specification. It combines formal and informal descriptions and incorporates graphical highlighting. • The basic building blocks of Z-based specifications are schemas. • Schemas identify state variables and define constraints and operations in terms of those variables.

8. Schemas • Schemas are comprised of three parts: a name, a signature, and a predicate. • The signature declares (state) entities introduced in a schema and their type. • Signature declarations are given in the form identifier: type. • The predicate defines logical relationships between entities in the declaration. • Z is based on the predicate calculus + typed sets.

9. Graphical Layout of a Schema Signature Predicate

10. Schemas (Cont.) • Schemas describe state entities or operations on state entities. • To describe state entities,declared variables comprise the entities while predicates specify their invariant properties. • To describe operations,declarations consist of initial and final state variables, inputs, and outputs, while predicates specify the relations among them.

11. Example 1 Specification of a generic container entity: modeled as natural numbers Container contents: N capacity: N contents  capacity invariant property

12. Example 1 (Cont.) Specification of a generic indicator entity: Indicator light: {off, on} reading: N danger_level:N light = on  reading  danger_level

13. Example 1 (Cont.) Specification of a storage tank entity: Storage_tank Container Indicator reading = contents capacity = 500 danger_level = 50 Predicates on separate lines are separated by an implicit “AND”.

14. Example 1 (Cont.) Expanded specification of a storage tank entity: Storage_tank contents: N capacity: N light: {off, on} reading: N danger_level:N contents  capacity light = on  reading  danger_level reading = contents capacity = 500 danger_level = 50

15. Example 1 (Cont.) Specification of a nominal storage tankfill operation: “Delta” indicates that values of one or more variables will be changed. “?” indicates that amount is an INPUT. Fill-OK  Storage-tank amount?: N contents + amount?  capacity contents’ = contents + amount? “dash” indicates the value of contents AFTER the operation

16. Example 1 (Cont.) Specification of an exceptional storage tankfill operation: “Xi” indicates that NO variables will be changed by the operation. “!” indicates that r is an OUTPUT. OverFill  Storage-tank amount?: N r!: seq CHAR capacity < contents + amount? r! = “Insufficient tank capacity – Fill cancelled”

17. Example 1 (Cont.) Complete specification of the storage tank fill operation: Fill Fill-OK V OverFill

18. Key Points • Model-based specification relies on a state model of the system using mathematical entities such as the predicate calculus and typed sets. • Functions / operations may be specified by defining their effect on system state. • It is normal to specify functions / operations incrementally and then combine the fragments to produce a complete specification.

19. Key Points (Cont.) • Z specifications are comprised of a number of schemas that introduce typed names and define predicates over those names. They are distinguished from surrounding text by graphical highlighting. • Schemas are building blocks which may be combined and used in other schemas. The effect of including schema A in schema B is that schema B inherits the names and predicates of schema A.

20. Notes on: Is Proof More Cost-Effective Than Testing? Prepared by Stephen M. Thebaut, Ph.D. University of Florida by Steve King, Jonathan Hammond, Rob Chapman, Andy Pryor