Formalism & Informality in Software Engineering

1. Formalism & Informalityin Software Engineering Michael Jackson jacksonma@acm.org Soft-Ware Conference Belfast April 10 2002

2. The Nature of Formality • Herman Weyl (quoted approvingly by Abelson & Sussman): “We now come to the decisive step of mathematical abstraction: we forget about what the symbols stand for. [The mathematician] need not be idle; there are many operations he may carry out with these symbols, without ever having to look at the things they stand for”. • The procedure is: • 1. make a symbolic description of the domain • 2. reason mathematically by manipulating the symbols according to formal rules • 3. infer domain properties from the conclusions • Note: The set of potentially relevant considerations is small; all are considered in making the symbolic description

3. The Nature of Informality • Hamlet (quoted approvingly by William Shakespeare): Horatio: “O day and night, but this is wondrous strange!” Hamlet: “And therefore as a stranger give it welcome. There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy.” • The procedure is: • 1. make a definition, or a statement, or a rule • 2. find a wondrous strange case you can not handle • 3. recognise the consideration you had ignored • 4. repeat 1, 2, 3 quant suff • Note: Informality is not vagueness: it’s unbounded relevance (there is always something else to be considered)

4. Formal And Informal Domains • Formal domains (ie domains for which formalisation is adequate) • Computer programs • Symbol manipulation procedures for determining the effects of sequencesof operations are guaranteed to give correct results • Chess • A game can be completely described in symbols; the description can be analysed to determine the outcome with guaranteed correctness • Informal domains (ie domains for which formalisation is not adequate) • Football • A game can not be completely described in symbols • Chess • 12.1 High standards of etiquette are required of all players (FIDE) • The Laws of Chess cannot cover all possible situations that may arise during a game, nor can they regulate all administrative questions (FIDE)

5. // Formalise Line 1 of song // Formalise Line 2 of song // Eliminate Universal Quantifier // Eliminate Universal Quantifier Applying Formalism To an Informal Domain* • “Everybody loves my baby ... but my baby loves only me” • Formalisation •  x · Loves (x, MyBaby) •  y · Loves (MyBaby, y)  y = Me • Reasoning •  x · Loves (x, MyBaby) Loves (MyBaby, MyBaby) •  y · ( Loves (MyBaby, y)  y = Me ) Loves (MyBaby, MyBaby)  MyBaby = Me • Conclusion: I am my baby • Something is wrong  but how could we have known? * an example due to David Gries

6. Characteristics of Informal Domains • The denotations of terms are not exact • What, exactly, counts as ‘over the line’ in football? • What, exactly, counts as a ‘distract or annoy the opponent’ in chess? • What, exactly, counts as ‘loves’ in a popular song? • No universally quantified statement is perfectly true • (This is a universally quantified statement) • Every human being has a unique human mother • But what about ‘the first homo sapiens’? What about Adam and Eve? • All books of the same title and author are equivalent • But what about different editions? • The exception proves the rule • That is: the rule must be tested by considering the boundary cases • ‘Every human being has a unique human mother’ is a good enough rule for most purposes (eg in last three millennia)

7. Does Problem Informality Invalidate Formal Techniques? • No! • We must predict (imperfectly) the (informal) effects of the (formal) machines we build • We must formalise (some of) the informal parts of the problem • The hardest part of our work is formalisation • “... one of the greatest difficulties in software development is formalization — capturing in symbolic representation a worldly computational problem so that the statements obtained by following rules of symbolic manipulation are useful statements once translated back into the language of the world. The formalization problem is the essence of requirements engineering ...”W L Scherlis, responding to E W Dijkstra “On the Cruelty of Really Teaching Computer Science, CACM 32,12, December1989

8. A B ProblemDomain Customer’sRequirement H/W-S/WMachine Two Projections Of the Problem Domain • Requirement phenomena at B • Borrow book • Reserve book • Book is available • Lose book • Interface phenomena at A • Book barcode read • Member card read • Enter borrow request • Print availability letter Library AdministrationSystem • Interface phenomena at A • Button line high • Motor relay on • Motor direction up • Floor sensor on • Requirement phenomena at B • User presses button • Lift arrives • Lift doors open • User enters lift Lift ControlSystem

9. A B ProblemDomain Customer’sRequirement H/W-S/WMachine The Formal Computing Machine And the Informal World • Machine Behaviour Domain Properties  Customer Requirement • formalinformal  informal Machine/WorldInterface(Formal : Informal) CustomerRequirement(Mostly Informal) Formal Machine(Symbol Manipulator) Problem Domain(Mostly Informal)

10. Formalisation and Informality in Software Engineering • Software engineering must deal with a combination of the formal and the informal • The computer is effectively formal; the world is informal • We must understand how they interact ... • … so we must formalise informal domains • Some techniques and tools for the formalisation task: • Explicit description subjects • Designations • Unknown states • Graceful degradation of descriptive truth

11. World ofBooksAuthors &c Model  RealWorld Modelling Machine DB Model DBModel InformationMachine Display  Model Display Explicit Description Subjects  1 • “Each book has exactly one author” · “Author may be NULL” • Do these descriptions describe ... • The model? • The modelled domain? • The modelling process?

12. /R1;R2 after(50s)/R2;G2 3: Stop1  Go2 1:(no light) 2: Stop1  Stop2 after(120s)/G2;R2 after(120s)/G1;R1 after(50s)/R1;G1 4: Go1  Stop2 5: Stop1  Stop2 Explicit Description Subjects  2 • The requirement, the specification, the domain properties or the machine? • R1, R2, G1, G2 are shared events controlled by the machine • Stop1, Stop2, Go1, Go2 are private domain -controlled states • This description confuses all three subjects

13. Designations  1 • Formalising denotation of terms in informal domains • “Every human being has a unique human mother” • What does ‘mother’ mean? • Designations fix denotations (but not perfectly) • “mother(x,y)  x is the genetic mother of y” • formal term recognition rule • The formal term denotes a class of ‘primitive observation’ phenomenon • The recognition rule has to be ‘good enough’ for the problem context • Avoiding terms that can’t be adequately designated • “An airline flight (eg ‘BA178’) is … ???” • Use definition to build more elaborate terminology • Choose the base terms (designated phenomena) very carefully • The most appropriate base terms are often events

14. -0.2 -0.1 0.0 0.1 0.2 2.8 2.9 3.0 3.1 3.2 Designations  2 • How should we choose base terms in an informal domain? • Digital electronic engineers show how OK OK On Off • ‘On’ and ‘Off’ are designated to be 3.00.2 and 0.00.2 • Clocking ensures that the voltage is not examined except when it is known to be either in [-0.15,0.15] or in [2.85,3.15] • Usually, we can’t control the problem domain like this … • ... but we can choose phenomena in the particular problem context that ‘stay away from the definition boundary’ ... • … and sometimes we can allow human intervention

15. Designations  3a • Human intervention in system operation can be used to resolve difficulties arising from inadequate recognition rules • “Where cases are not precisely regulated by an Article of the Laws... The Laws assume that arbiters have the necessary competence, sound judgement and absolute objectivity. Too detailed a rule might deprive the arbiter of his freedom of judgement and thus prevent him from finding the solution to a problem dictated by fairness, logic and special factors.” Preface to the FIDE Laws of chess • “Dogs accompanying fare-paying passengers may travel at one-half of the third-class fare. Travel by other animals is subject to the discretion of the Company’s station master at the station of departure.” Regulations and Bye-Laws, London & South Western Railway Company, 1857

16. Designations  3b • Human intervention during system operation can be used to resolve difficulties arising from inadequate recognition rules

17. R R 1:(no light) 1:(no light) 2: Stop 2: Stop R R G G G G G R ? 4: 3: Go 3: Go Unknown States • Explicit ignorance • Restricted event sequences • What if a G event occurs in state 2? • Light Unit inhibits it? • Environment guarantees it won’t? • We forgot to say? • Unit enters the unknown state (?) • In ? “all bets are off” • No exit from ? • We can be more specific later

18. Graceful Degradation of Descriptive Truth • A formal description of an informal domain is not fully correct • The source program has these properties … • … but not if the programmer has made a syntax error … • … or the wrong file has been presented to the compiler • A (sufficiently) correct description by ordered approximations • 1. Syntactically correct source program • 2. Incorrect source program with syntax errors of certain classes • 3. Nonsense text: not a source program • Benefit of separating the approximations • Clarifies the domain properties needed for each requirement • Leaves composition complexity until components are understood • In a physical domain • Fault-tree analysis can guide levels of approximation

19. What We Rely On In Formalisation • The purpose of formalisation: • To allow us to design our machines, and to reason about their effects in the problem domain • The underlying assumption of formalisation: • Our formal descriptions are accurate enough because … • … the problem domain is informal, but not malicious • Viewed as non-determinism, the inaccuracy is neither angelic (always what you want) nor demonic (never what you want) • The underlying assumption breaks down in security applications • For security problems, the problem domain is malicious

20. The Limits of Formal Security Models(And Of All Other Formalisations Too) • The lesson I learned was that security models and formal methods do not establish security. They only establish security with respect to a model, which by its very nature is extremely simplistic compared to the system that is to be deployed, and is based on assumptions that represent current thinking. Even if a system is secure according to a model, the most common (and successful) attacks are the ones that violate the model's assumptions. Over the years, I have seen system after system defeated by people who thought of something new. Dorothy Denning: National Computer Systems Security Award Acceptance Speech, October 1999 [communicated by Jim Yuill, NCSU]

21. The Role of Formalisation In an Informal World • Informality is unbounded relevance ... • There are always more considerations to take into account • The moon may rise • The passenger may arrive carrying a tortoise • The attacker may measure power consumption • … so formalisation can’t guarantee correctness in an informal domain • At best, it can help us to construct a system that is correct enough often enough • If formal reasoning shows an error, there are probably errors • If formal reasoning shows no error, there are probably errors • Formal reasoning based on formalisation shows only the presence of bugs, not their absence • But, like testing, it is indispensable