Model-Checking In-lined Reference Monitors

1. Model-Checking In-lined Reference Monitors Meera Sridhar and Kevin W. Hamlen The University of Texas at Dallas January 17, 2010 Supported in part by grants from AFOSR VMCAI 2010 Madrid, Spain

2. Reference Monitors • Examples: • file system permissions • memory safety • Disadvantages: • changing the policy requires changing the OS/VM • difficult to enforce finer-grained policies such as • “No modifications to files ending in .exe” OS/VM Reference Monitor grant/deny event untrusted code Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

3. In-lined Reference Monitors [Schneider] • enforce safety policies by injecting runtime security guards directly into untrusted binaries • guards test whether the impending operation constitutes a policy violation, and if so some corrective action is taken • maintain history of security-relevant events • Advantages: • No need to modify the OS/VM • enforce richer policies: e.g., no network sends after file reads • more flexible: code recipient can specify security policy • Examples: SASI [Erlingsson, Schneider], Java-MAC [Kim et al], Java-MOP [Chen, Rosu], Polymer [Bauer, Ligatti, Walker], ConSpec [Aktug, Naliuka], MoBILe [Hamlen, Morrisett, Schneider] OS/VM Reference Monitor grant/deny event untrusted code Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

4. IRM Certification Problem Main Points: TCB reduction verifier must be light-weight (e.g. suitable for hand-held devices) untrusted code policy Rewriter verifier reject (rewriter failure) execute Related work: ConSpec [Aktug, Naliuka] (certification via contracts), MoBILe [Hamlen et al] (certification via type-checking) rewritten code Trusted Computing Base Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

5. Overview of Contributions • design and implementation of prototype IRM model-checkingframework for ActionScriptbytecode • ActionScript/AIR ubiquitous, existing security mechanisms center around code-signing/sandboxing which are inadequate • novel approach for constructing a state abstraction lattice from a security automaton • precise yet tractable abstract interpretation of IRM code • rigorous proofs of soundness and convergence • feasibility shown by enforcing a URL anti-redirection policy for ActionScriptbytecodeprograms Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

6. ActionScript IRM Example public class RUnsafeURL extends Sprite { // constructor for popup link public function RUnsafeURL(){ vartheField:TextField = new TextField(); theField.text = "Click me!"; addChild(theField); theField.addEventListener(MouseEvent.CLICK, clickListener); } // respond to user clicks by popping up infinite numbers of windows private function clickListener(e:MouseEvent):void { var url:String="javascript:window.open('http://popup.url.com');void(0);"; while(true) { navigateToURL(new URLRequest("http://popup.url.com")); } } } Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

7. ActionScript IRM Example Policy: At most 3 pop-ups public class RUnsafeURL extends Sprite { private varsecurity:Number = 0; // constructor for popup link public function RUnsafeURL(){ vartheField:TextField = new TextField(); theField.text = "Click me!"; addChild(theField); theField.addEventListener(MouseEvent.CLICK, clickListener); } // respond to user clicks by popping up infinite numbers of windows private function clickListener(e:MouseEvent):void { var url:String="javascript:window.open('http://popup.url.com');void(0);"; while(true) { if (security > 3) HALT; navigateToURL(new URLRequest("http://popup.url.com")); security++; } } } Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

8. Reified Security State • added by IRM (not in original program) • abstract the event history at some point in program’s execution • Example: event counter • two operations: set and test • both operations are statically identifiable • write-safety achieved via type-safety and object encapsulation (e.g., private fields) • verified by standard ActionScriptbytecode verifier (similar to Java) • dangerous operations conservatively rejected (e.g., untrackable aliasing) • values not necessarily current • stored state may reflect a past security state • Example: exhibit event, then increment counter later Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

9. Synchronizing Reified State Variables • in practice, reified state value temporarily deviates from actual trace • unavoidable in practice, must be supported • Solution: represent abstract state as a pair: • (S, τ ‘), literal event sequence subsequent to last s update (usually ε or some very short sequence) abstract state when s was last updated Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

10. Verifying IRMs with Model-Checking • introduce concrete trace meta-variable τ • tracks history of security-relevant events • implicitly modified by security-relevant operations • verification strategy: abstract interpret τ-enhanced program to statically verify τ⊆ P • deciding τ⊆ P is the model-checking step • implemented as simple regular language subset in our prototype, but more sophisticated models possible • main challenge: abstracting τ • too concrete: too slow to converge • too abstract: too much conservative rejection Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

11. Abstract State Lattice T e1 security automaton q2 e2 . . . e0 q0 q1 state lattice e3 q3 … Res({q0,q1}) e4 we assume automata are deterministic Res(q0) Res(q1) … ordered by subset relation Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

12. Security Policy Definition 1 (Security Policy) Let A = (Q,Σ, q0, δ) be a deterministic security automaton. The security policy PA for automaton A is PA= ResA(Q) ResA(Q) = ∪q∈QResA(q) where ResA(q) is the residual of state q in A. Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

13. Abstract Traces Definition 2 (Abstract Traces): The language SS of abstract traces is SS = {(Res(Q0), τ) | Q0 ⊆ Q, τ ∈ Σ∗, |τ| ≤ k} ∪ {⊤SS } where ⊤SS = Σ∗. Abstracttraces are ordered by subset relation ⊆, forming the lattice (SS,⊆). Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

14. A Simple Example Security Policy : ε + e Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

15. A Simple Example, contd. Security Policy : ε + e Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

16. Abstract Machine Language Syntax Abstract configurations Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

17. Abstract Small-step Operational Semantics Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

18. Soundness & Convergence • Proof of Soundness • concrete machine • defines semantics of ActionScript VM • security violations modeled as stuck states • abstract machine • defines semantics of abstract interpreter • rejection modeled as stuck state • Progress Lemma: if the abstract machine makes progress, so does the concrete machine • Preservation Lemma: whenever both machines make progress, the abstract machine abstracts the concrete machine • Proof of Convergence • abstract machine reaches a fixed point within O(n2) iterations in the worst case, where n is the size of the security automaton • proof bounds the height of the abstract state lattice Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

19. Example Security Policy • Policy Goal: • prohibit malicious URL-redirections by ABC ad applets • URL navigation in ActionScriptBytecode: • redirections are implemented at the bytecode level by navigateToURL system calls • our policy requires that method check_url(s) be called to validate s before any redirection to s • check_url has a trusted implementation provided by ad distributor and/or web host, and may incorporate dynamic information) such as ad hit counts or webpage context. Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

20. Example Security Policy (contd.) • Policy Enforcement: • our IRM enforces this policy by injecting calls to check_urlinto untrusted applets • for better runtime efficiency, it positions some of these calls early in the program's execution (to pre-validate certain URL's) and injects runtime security state variables that avoid potentially expensive duplicate calls by tracking the history of past calls Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

21. Example Security Policy (contd.) • we used Prolog to build our IRM system • declarative nature of Prolog predicates yields clean, concise code • reversible nature of Prolog predicates helps reuse parsing code for code-generation • some preliminary numbers: Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

22. Future Work • extend work to support • reified security state per-object • reified security state updated by IRM before actual security state changes at runtime • support for recursion and concurrency • extend work to other binary languages and IRM systems that have been implemented for them Sridhar and Hamlen: Model-Checking In-lined Reference Monitors

23. Selected Citations • P. Cousot and R. Cousot. Abstract Interpretation: A unified lattice model for static analysis of programs by construction or approximation of fixpoints. In Proc. Symposium on Principles of Prog. Languages, pages 234–252, 1977. • P. Cousot and R. Cousot. Abstract Interpretation Frameworks. J. Log. Comput., 2(4):511–547, 1992. • B. W. DeVries, G. Gupta, K. W. Hamlen, S. Moore, and M. Sridhar. ActionScriptBytecode Verification with Co-logic Programming. In Proc. of the ACM SIGPLAN Workshop on Prog. Languages and Analysis for Security (PLAS), 2009. • K. W. Hamlen, G. Morrisett, and F. B. Schneider. Certified In-lined Reference Monitoring on .NET. In Proc. ACM Workshop on Prog. Languages and Analysis for Security (PLAS), 2006. • K. W. Hamlen, G. Morrisett, and F. B. Schneider. Computability Classes for Enforcement Mechanisms. In ACM Trans. Prog. Languages and Systems, 2006. • F. B. Schneider. Enforceable Security Policies. ACM Trans. Information and System Security, 3:30–50, 2000. • M. Sridhar and K.W. Hamlen. ActionScript In-Lined Reference Monitoring in Prolog. In Proc. Intl. Symposium on Practical Aspects of Declarative Languages, 2010. Sridhar and Hamlen: Model-Checking In-lined Reference Monitors