Software Security Monitors: Theory & Practice

1. Software Security Monitors:Theory & Practice David Walker Princeton University (joint work with Lujo Bauer and Jay Ligatti)

2. General-purpose Security Monitors • A security monitor (program monitor) is a process that runs in parallel with an untrusted application • monitors examine application actions • decide to allow/disallow application actions • may terminate an application, log application actions, etc. • monitors detect, prevent, and recover from erroneous or malicious behavior at run time • monitors generalize specific enforcement mechanisms such as access control lists, etc. Software Security Monitors

3. What is a security monitor? Monitors analyze & transform untrusted application actions: Input Stream Monitor Output Stream a3 a2 a2 a1 a4 a2 a2 a1 … … Application generates actions to be input into monitor Machine executes actions output by monitor Software Security Monitors

4. Possible Monitor Actions • Acceptthe action • Halt the application • Suppress (skip) the operation • Insert some computation • Also: replace results; raise exceptions Software Security Monitors

5. Formalizing security monitors • Security monitors => formal automata that transform a stream of program actions • Given: a set of possible program actions A • Monitors are deterministic state machines: (Q, q0, T) where Q = state set q0 = start state T = transition function Software Security Monitors

6. Operational Semantics • Single step (determined by T): • (Sin, q)  (Sin’, q’) • Multi-step (reflexive, transitive closure of T): • (Sin, q)  (Sin’, q) • Output sequence is observable • Input sequences are not observable So So Software Security Monitors

7. A Hierarchy of Security Monitors We classify monitors based on their transformational abilities (ie: based on T). Insert Suppress OK Halt Truncation Suppression Insertion Edit             Software Security Monitors

8. An Example: E-Banana.com • Set of application actions: A = { take(n), // take n bananas pay(n), // pay for n bananas browse, // browse for bananas receipt // commit } • Edit Automaton: pay(n) take(n) pn browse browse take(n) pay(n) start init tpn tn receipt pay(n);take(n);receipt Software Security Monitors

9. Edit Automata • Definition: (Q,q0,T) • where T = (t,e,i) • State transition function t • t : action x state  state • Emission function e • e : action x state  {+,-} • Insertion function i • i : action x state  action sequence x state Software Security Monitors

10. Edit Automata • Operational Semantics • (S, q)  (S’, q’)if S=a;S’ and t(a,q)=q’ and e(a,q)= + • (S, q)  (S’, q’)if S=a;S’ and t(a,q)=q’ and e(a,q)= - • (S, q)  (S, q’)if S=a;S’ and i(a,q)=(Sins, q’) • (S, q)  (empty, q)otherwise a (E-Accept) (E-Suppress) Sins (E-Insert) (E-Halt) Software Security Monitors

11. Security Policies • A program execution is a sequence of actions • A Security Property is a predicate over executions. • Example Properties: • P(S) iff bananas taken equal bananas paid for in S • Access control, resource bounds policies are policies • Non-properties: • Relations between different executions of a program • Information-flow policies Software Security Monitors

12. What does it mean to enforce a policy? • Principle of Soundness All observable outputs obey the policy sequences Sin .  state q’ .  sequence So 1. (Sin, q0)  (empty, q’) 2. P(So) • Principle of Transparency Semantics of executions that already obey policy must be preserved 3. P(Sin) (Sin So) So Software Security Monitors

13. Some Useful Equivalences • Remove/Insert unnecessary actions • fclose(f);fclose(f) fclose(f) • Replace a sequence with equivalent actions • socket(S);send(S,m) socketSend(S,m) • Permute independent actions • fopen(f);fopen(g) fopen(g);fopen(f) • Necessary properties: • reflexive, symmetic & transitive • S S’  P(S) P(S’) Software Security Monitors

14. E-Banana.com • Equivalence Rules: 1) (browse; S)  S 2) (S1; take(n); pay(n); S2)  (S1; pay(n); take(n); S2) Software Security Monitors

15. Conservative Enforcement • Enforcer satisfies Soundness but not necessarily Transparency •  properties P . ( sequence S . P(S))  P can be conservatively enforced Conservative Software Security Monitors

16. Effective Enforcement • Enforcer satisfies Soundness and Transparency • provides some flexibility for the enforcer to edit the execution sequence • guarantees the final results of running the application with the monitor are semantically equivalent to running the application without the monitor Effective Conservative Software Security Monitors

17. Precise Enforcement • Definition • Enforcer satisfies Soundness and Transparency • Enforcer must output actions in lock-step with application • Motivation • In some scenarios, operations cannot be delayed without disrupting application semantics Precise Effective Conservative Software Security Monitors

18. What properties can be enforced? • The enforceable properties depend upon • the definition of enforcement (conservative, effective, precise) • the class of automaton (truncation, suppression, insertion, edit) • the space of possible input programs • if the monitor can assume certain “bad” executions do not occur, it can enforce more properties • static program analysis (type systems; proof-carrying code) can constrain program execution in ways useful to run-time monitors Software Security Monitors

19. Effective Enforcement • An E-Banana.com policy: • Our edit automaton is an effective enforcer: • It satisfies Soundness • It satisfies Transparency • Proofs are by induction over the possible inputs • Less powerful automata (truncation, suppression and insertion) cannot enforce the E-Banana property • Proof by contradiction shows either Soundness or Transparency will be violated browse*; ((take(n);pay(n) | pay(n);take(n)) ; receipt)* Software Security Monitors

20. A Simple Theorem • Theorem: Any decideable predicate P on executions is a property that can be effectively enforced by some edit automaton • Proof: construct a transactional edit automaton that suppresses and logs program actions when ¬P(S) and commits (outputs) when P(S), for every initial sequence of actions S in a program execution Software Security Monitors

21. Effectively Enforceable Properties Editing Properties Insertion Properties Trunc. Prop. Suppression Properties Software Security Monitors

22. Summary of theoretical results • We have developed the following rigorous methodology for reasoning about run-time security: • Define the computational framework using formal operational semantics • Define what it means to enforce a policy • Prove results about enforceable policies & mechanisms from definitions 1 & 2 Software Security Monitors

23. Future Work/Research Ideas • Proper definitions of enforcement for infinite execution sequences • Understanding edit automata on infinite sequences • Understand transactional policies & develop “transaction automata” • what can they enforce? • Incorporate more practical elements into the model • security environment; cryptographic secrets • replacement of results, exceptions and program state Software Security Monitors

24. Polymer, the Language • Polymer • A domain-specific language for programming security monitors (ie: edit automata) • Java + a couple of simple extensions: • atomic policy definitions encapsulating • a set of security-relevant actions • security state • decision procedure that produces security “suggestions” (halt, suppress action, insert action, etc) • compositional policy definitions involving • higher-order policy combinators Software Security Monitors

25. Securing Untrusted Applications untrusted code describes security-relevant program points Java application policy interface instrumented application separately compiled from policy contains hooks to call monitor Software Security Monitors

26. Securing Untrusted Applications Java application implements dynamic security policy policy interface policy implementation instrumented application combines application and policy secure application Software Security Monitors

27. Atomic Polymer Policy new policy definition extends policy class class limitFiles extends Policy { private int openFiles = 0; private int maxOpen = 0; limitFiles(int max) { maxOpen = max; } .... } private policy state policy constructor Software Security Monitors

28. Atomic Polymer Policy Continued class limitFiles extends Policy { private int openFiles = ... private int maxOpen = ... public ActionPattern[] actions = new ActionPattern[] { <File fileOpen(String)>, <void fileClose(File)> }; .... } set of policy- relevant methods Software Security Monitors

29. Atomic Polymer Policy Continued class limitFiles extends Policy { private int openFiles = ... private int maxOpen = ... public ActionPattern[] actions = ... Suggestion before(Action a) { aswitch (a) { case fileOpen(String s) : if (++openFiles <= maxOpen) return Suggestion.OK(); else return Suggestion.Halt(); case fileClose(File f) : ... policy behavior Software Security Monitors

30. Atomic Polymer Policy Continued class limitFiles extends Policy { private int openFiles = ... private int maxOpen = ... public ActionPattern[] actions = ... Suggestion before(Action a) { aswitch (a) { case fileOpen(String s) : if (++openFiles <= maxOpen) return Suggestion.OK(); else return Suggestion.Halt(); case fileClose(File f) : ... Software Security Monitors

31. Atomic Polymer Policy Continued class limitFiles extends Policy { public ActionPattern[] actions = ... private int openFiles = ... private int maxOpen = ... Suggestion before(Action a) { aswitch (a) { case fileOpen(String s) : if (++openFiles <= maxOpen) return Suggestion.OK(); else return Suggestion.Halt(); case fileClose(File f) : ... Software Security Monitors

32. Complex Monitors • Combine atomic policies defined over a variety of different resources • eg: sample applet policy • file system access control • number of files opened • restricted network access • no network access after local file is read • communication with applet source only Software Security Monitors

33. Policy Combinators • Programmers may write parameterized policy combinators: • And, Or, Forall, Exists, Chinese wall,... P1 P2 AndPolicy:  s2 s1 s Software Security Monitors

34. Policy Combinators class AndPolicy extends Policy { private Policy p1; private Policy p2; AndPolicy(Policy pol1, Policy pol2) { p1 = pol1; p2 = pol2; ... } } first-class policies Software Security Monitors

35. Policy Combinators class AndPolicy extends Policy { ... Suggestion before(Action a) { Suggestion s1 = p1.before(a); Suggestion s2 = p2.before(a); if (s1.isOK() && s2.isOK()) return Suggestion.OK(); else ... } } using suggestions In reality, writing combinators is very tricky Software Security Monitors

36. Summary of Language Design • Polymer facilitates the implementation of program monitors by • encapsulating all elements (relevant actions, state, decision procedure) of atomic policies in a single place • providing mechanisms to compose policies in a well-defined manner • coming equipped with a formal semantics • we’re working on it Software Security Monitors

37. Conclusions • Technology for securing extensible systems is in high demand • Software security monitors are one part of the solution • For more information, see • Edit Automata: Enforcement Mechanisms for Run-time Security Policies. IJIS 2003. • Types and effects for non-interfering program monitors.  ISSS 2002 & LNCS 2609. • More Enforceable Security Policies. FCS 2002. • www.cs.princeton.edu/sip/projects/polymer/ Software Security Monitors

38. End Software Security Monitors

39. Realistic Monitors • Protect complex system interfaces • interfaces replicate functionality in many different places • method parameters communicate information in different forms • eg: Java file system interface • 9 different methods to open files • 4 different methods to close files • filename strings, file objects, self used to identify files Software Security Monitors

40. Abstract Action Definitions java.lang.io FileReader(String fileName); FileReader(File file); RandomAccessFile(...); ... FileReader.close(); RandomAccessFile.close(); ... fileOpen(String n); fileClose(); Software Security Monitors

41. Abstract Action Definitions class fileOpen extends ActionSig { boolean canMatch(Action a) { aswitch (a) { case FileReader(_) : return true; case RandomAccessFile () : return true; ... } String parameter1(Action a) { .... } } Software Security Monitors

42. Abstract Action Pattern Matching class limitFiles extends Policy { ... Suggestion step(Action a) { aswitch (a) { case fileOpen(String s) : ... case fileClose() : ... } } fileOpen.parameter1(a) fileOpen.canMatch(a) Software Security Monitors

43. Taxonomy of Precisely Enforceable Properties Software Security Monitors

44. Secure Application Untrusted application Host System (Java) Program Monitor Definition Polymer language extensions Java core Software Security Monitors

45. Policy Architecture: Simple Policies system interface Simple Policy Def. Host System (Java) Polymer language extensions Java core Software Security Monitors

46. Policy Architecture: Abstract Actions abstract system interface Host System (Java) Simple Policy Def. Abstract Action Def. Polymer language extensions concrete system interface Java core Software Security Monitors

47. Policy Architecture:Complex Policies Complex, System-specific Policy abstract system interface Simple Policy Def. Policy Comb. Def. Abstract Action Def. Host System (Java) Polymer language extensions concrete system interface Java core Software Security Monitors

48. Securing Extensible Systems • Many questions: • Our application requires property X. Can we enforce it precisely or will we have to get by with an approximation? • How do we write down our policy succinctly and unambiguously? • What specific mechanism will we need to enforce our policy? • How do we implement the mechanism? Software Security Monitors

49. Summary • A general framework for formal reasoning about security monitors • defined a hierarchy of security monitors • gave meaning to the word “enforceable” • developed rigorous proofs concerning enforceable properties • Polymer: A programming language for composing security monitors • techniques for modular monitor design & composition • formal semantics as an extension of FeatherWeight Java Software Security Monitors