A Few Remarks About Formal Development of Secure Systems

1. Eric Jaeger, UPMC/LIP6 & SGDN/DCSSI Thérèse Hardin, UPMC/LIP6 A Few Remarks About Formal Development of Secure Systems

2. Foreword • This talk is about development of secure systems • To some extent it is also applicable to safety • Examples related to software development, but the concepts are generic • We consider the application of deductive formal methods (e.g. Coq, PVS, B, Focal, etc.) to such developments • Yet some of the underlying concepts are also valid for other formal approaches, such as Model Checking, Static Analysis, Typing, etc. • We are convinced of the interest of formal methods and support their use, yet we would like to give some warnings • Our main questions are: • What is the level of confidence one can have in a formally developed secure system? • How is it possible to improve this level of confidence?

3. Specification Driven Development Book of Specifications Local consistency check Formal Specification … What … Correctnessthrough traceability Refinement check Refinement Refinement Refinement check Refinement Implementation … How …

4. What is a Formal Specification? • A specification expressed in a language with an explicit and clear semantic (e.g. a mathematical/logical language) • Describing (ideally) the What rather than the How • High level, abstract, declarative, non-deterministic • Derived from the Book of Specifications • Through a manual translation activity (some tools, e.g. animation of models, translation to UML, can help to ensure the validity of this step) • Sometimes only partial, either due to restriction to relevant parts of the system… or because of the limitations of the chosen formal language • Example: library of functions on sets of abstract values • A set is a mathematical structure, not a concrete representation • Some operators (such as the choice operator) are non-deterministic

5. What is Refinement? • A formal concept addressing the question of the correctness • Going from a high-level, abstract, declarative, non-deterministic representation to a detailed, concrete, imperative, deterministic one • Support refinement, event refinement, B refinement, etc. • Used here in a very generic meaning, covering the activity of deriving a correct (proven) implementation from a specification • Including choice of concrete representations, of algorithms, etc. • Example: from the specification of a library on sets, deriving an implementation on sorted lists • Lists of concrete values sorted using a total order and a sorting algorithm (both chosen by the developer) • The choice operator is implemented by head (first element of the list)

6. Decomposition The program shall provide a square root function on natural values Translation √:ℕ→ℕ s.t. ∀x, |√(x)²-x|≤ε(x) or alternatively √(x)²≤x<(√(x)+1)² Refinement Validity proof A total deterministicspecification, but notyet a program! Refinement Validity proof Refinement Validity proof An Example of Formal Development The train shall automatically decelerate to stop at the reference point int sqrt(int x) { i≔x ; while i≠(i+x/i)/2 do i≔(i+x/i)/2 ; return i }

7. Threat Model • From the Book of Specifications, • The Good Guy: • Checks the formal specification and the code • Ensures that the correctness proofs are provided and are valid • The Bad Guy: • Chooses the formal method and the formal tool • Writes the formal specification • Produces the refinements, the correctness proofs and the code • If needed, interferes with or uses the implementation Book of Specifications Formal Specification … What … Refinement Refinement Refinement Implementation … How …

8. Poor practices, oversights or deviations that can endanger the security of formally developed systems When Formal Methods Encounter Misbehaviours

9. Inconsistent Specifications • It is possible to write inconsistent specifications • Enforcing as an axiom something false, asking a value to be equal to both 0 and 1… or more generally specifying a function that cannot exist • Provided one inconsistency, it is possible to prove anything • Reductio ad absurdum: to prove P, assume ¬P and prove False… That is: using an inconsistency you can prove any (flawed) system to be correct • There is no automated or even standard way to detect inconsistencies; solutions or mitigations include: • Avoid when possible axioms, and check specifications of types, values and functions, adding where possible consistency proof obligations and providing implementation (even simplified ones) • Analyse the concrete meaning of the proofs • Tools such as automated testing, model animation can help…

10. Unsatisfiable Specifications Invalid axiom Empty (but consistent) type Inconsistent Specifications MACHINE absurd VARIABLES v INVARIANT v∈ℕ ∧ v=0 ∧ v=1 ASSERTION 0=1 MACHINE absurd CONSTANTS f PROPERTIES f∈ℕ→ℕ ∧ ∀x,y·x<y⇒f(x)>f(y) ASSERTION 0=1 Inductive ℤ:Set := +:ℕ→ℤ | -:ℕ→ℤ. Axiom zero_unsigned:+(0)=-(0). Theorem absurd:False. Inductive Lst:Set := cons:ℕ→Lst→Lst. Theorem absurd:∀ (l:Lst), False

11. Insufficient Specifications • Specifications are insufficient when they fail to capture all the relevant aspects of a system (lazy developer syndrome) • Even provided a serious and honest developer, formal specifications are written in formal languages having explicit semantics… yet • Some work hypothesis (simplifications) can be built-in, that is some aspects of the reality are just left behind the formal horizon • The formal semantics are not always mastered by developers, and at some point simplified interpretations can be harmful • Solutions or mitigations include: • Be aware of the semantics and of their “limitations” • Enriching the formal model e.g. by adding dummy observers, adopting a different granularity, or embedding additional aspects (complex)

12. This transient state is NOT considered during proof This code can be provencorrect… yet do not expecta result from ^(10,10) ! Insufficient Specifications MACHINE airlock VARIABLES gate1,gate2 INVARIANT gate1,gate2∈{open,closed} ∧ ¬(gate1=open ∧ gate2=open) OPERATIONS open1≙ IF gate2=closed THEN gate1:=open close1 ≙ gate1:=closed open2 ≙ IF gate1=closed THEN gate2:=open ; IF attack THEN gate1:=open ; wait ; gate1:=closed close2 ≙ gate2:=closed Inductive ℕ:Set := 0:ℕ | S:ℕ→ℕ +(n m:ℕ):ℕ := n+0⇒n | n+S(m’)⇒S(n+m’) *(n m:ℕ):ℕ := n*0⇒0 | n*S(m’)⇒n+(n*m’) ^(n m:ℕ):ℕ := n^0⇒1 | n^S(m’)⇒n*(n^m’)

13. Partial and Non-Deterministic Specifications • Good principle: Specify only what you need • Formal specifications can be partial (either as a pre-condition or as a “don’t care”) • Both versions are pretty similar and ultimately correspond to degrees of freedom left to the developer • Similarly formal specifications can be non-deterministic • A non-deterministic specification does not specify randomness, but only capture relevant constraints • The degree of freedom left to the developer by partial or non-deterministic specifications is easily underestimated • Solutions or mitigations include: • Use defensive style instead of the offensive one • Tools such as flow analysis can help…

14. Common implementation HeadI(l:[ℕ]) := [] ⇒ Secret | h::_ ⇒ h This pre-condition specifies readonly when file fexists; a malicious developer can correctly refine readby implementing ANY behaviour when the file f does not exist… including creating an invisible new file Partial and Non-Deterministic Specifications Guard Head1(l:[ℕ])(p:l≠[]):{x:ℕ | ∃ l’:[ℕ], l=x::l’} Pre-condition Head2(l:[ℕ]):{x:ℕ | l≠[]→∃ l’:[ℕ], l=x::l’} MACHINE filesystem … INVARIANT … cpt=card(Files) … OPERATIONS out←read(f,u) ≙ PRE f∈Files THEN … out:=content(f) END

15. A Remark on the Refinement Concept • Refinement is a defined concept in several formal methods… and an underlying concept in many others • Going from a specification to a (correct) implementation • Refinement does not aim at automating the development process, but rather at offering tools to validate the development • Refinement is generally transitive and monotone • Transitive to allow an arbitrary number of steps in the development process, monotone to be compatible with decomposition • Yet the most important fact about refinement is that it preserves properties of the system • If the specification has a given property, then the refinement has the same property… This captures the meaning of correctness

16. Time ? Memory ? Power ? Heat ? Order ? PID ? ... An Effective Refinement • Consider again the sentence “If the specification has a given property, then the refinement has the same property” • This looks like Leibniz’ equality : ∀ P, P(Spec)⇒P(Ref) • That is, this definition is too strong • Refinement needs just to check that the specification and the refinement are sufficiently similar • By considering only the “relevant” properties of a system • In particular refinement is (usually) only extensional • That is a black box approach, “quicksort=heapsort” • But this myopia may have security consequences • Basically, some security properties may not be preserved • Yet hardly a paradox, provided this presentation ;-)

17. The malicious developer takes advantage of the non-determinism of thespecification… but the real “paradox” here is that this operation wasnever expected to depend upon any internal or external value The Refinement Paradox MACHINE boolean OPERATIONS out←get ≙ out:=true out:=false How many correct refinements of this specification? Too many! Including this one… MACHINE covert_channel REFINES boolean VARIABLE dmp INVARIANT dmp∈ℕ INITIALISATION dmp:=private_key OPERATIONS out←get ≙ IF dmp%2=0 THEN out:=true ELSE out:=false ; dmp:=dmp/2

18. Refinement • Elimination of non-determinism in the specification does not help • This is not convenient, nor does it address the fundamental cause • Furthermore, a total and determinist specification can be refined into a partial non-determinist specification by choosing an adequate concrete representation of the abstract values (e.g. lists to represent sets) • Again, this can be seen as a problem of mastering the semantics of the formal method used • Do not read more in a specification than what it really says • More generally, any property that is not extensional is likely to be not expressible and/or not preserved during refinement • Some security or safety properties are not extensional (confidentiality, space and time constraints for the execution, one-way functions, etc.) • Ad hoc approaches have to be considered (critical review, static analysis, flow analysis, etc.)

19. Looking at the whole process • The confidence in the “adequacy” of an implementation with regard to a specification results from a lot of steps ; beyond the validity of the specification and of its refinements, we can also question: • The validity of the mathematical theory supporting a formal method • The correction of the tools implementing this theory • ... And the correction of the compiler, of the design of the microprocessor, of the assumed laws of physics, etc. • Looking hard at it, you indeed found interesting glitches • On the theoretical side: Inconsistent theories, theorems that are not theorems, terms that are not allowed by the syntax, and not completely valid translations of the implementation, etc. (not in the same method) • On the tool side: various bugs in the provers for example

20. What about the use? • Even in an ideal world, a system whose security has been proven is unlikely to be unconditionally secure • Are ALL the hypothesis, either explicit or implicit, enforced? • Classical sources of disruptions include: • Breaking implicit hypotheses associated to the used formal method • Types violations (cf. PKCS#11 vulnerability) • Errors or fault injections during transmission and storage • Completion of the model versus isolation of the actual system • The security proof helps the Good Guy to identify the sufficient hypotheses under which security is guaranteed • And the Bad Guy, rather than looking for a non-existing bug, can focus on breaking the weak hypotheses

21. Are formal methods useless? Of course not! Conclusion

22. A Summary • Formal methods are not (yet) the silver bullet for development • They can be tricky to master; writing a good specification can be difficult as it is easy to make too optimistic intuitive interpretations • They generally do not aim at tackling with any possible concern • There may be subtle theoretical problems or bugs in the tools (rare) • One of the important aims of this talk is to help for a rationale evaluation of the genuine level of assurance • The worst possible situation in security is not to use a vulnerable system, but to use it while assuming it is secure • Identifying possible source of concerns is also a good way to plan a roadmap for improvement

23. A Positive Conclusion • Using formal methods still greatly improve the confidence • They can today build a world without buffer overflows, for example • Our concerns are mostly related to security with worst case hypotheses (often stupidly asking the Bad Guy to be the developer)… • The current concerns have solutions • Formal methods can be improved to be usable by trained engineers rather than only by experts… and engineers can be trained • Works are on-going to improve the confidence in the theories and tools • Addressing security concerns can be done very efficiently by using (deductive) formal methods in combination with other approaches such as model animation, flow analysis, testing, etc.

24. Some Arbitrary References A note on the confinement problem, B.W. Lampson, Commun. ACM 1973 Seven myths of formal methods, A. Hall, IEEE Software 1990 Seven more myths of formal methods J.P. Bowen & M.G. Hinchey, IEEE Software 1995 Ten commandments of formal methods J.P. Bowen & M.G. Hinchey, IEEE Software 1995 On the security of PKCS#11, J. Clulow, CHES 2003 Breaking the model: finalisation and a taxonomy of security attacks J.A. Clark & S. Stepney & H. Chivers, 2004 Ten commandments of formal methods... Ten years later J.P. Bowen & M.G. Hinchey, IEEE Computer 2006 On the notion of vacuous truth, M. Samer & H. Veith, LPAR 2007 Why would you trust B? E. Jaeger & C. Dubois, LPAR 2007 A few remarks about formal development of secure systems E. Jaeger & T. Hardin, HASE 2008

25. An Advertising: FoCaLize • Environment for the development of certified programs • Functional and “object-like” features as well as means to write formal specifications and proofs (verified by the Coq proof checker) • Inheritance and refinement mechanisms to derive efficient executable code (translated to OCaml) from a specification • Library of mathematical structures with complex algorithms • Beyond those technical characteristics, the Focalize project aims at • Improving confidence in Focalize (theory, compiler, prover…) • Providing a simpler (and more automated) formal method • Favouring reuse of developments • Addressing safety and security specificities (SSURF project) http://focalize.inria.fr