CMV: Automatic Verification of Complete Mediation for Java Virtual Machines

1. CMV: Automatic Verification of Complete Mediation for Java Virtual Machines A. Prasad Sistla, V.N. Venkatakrishnan, M. Zhou, H. Branske University of Illinois at Chicago

2. Java applications Java compiler and other tools Java libraries (including java.io, java.net, jave.lang and other packages) Java Platform Java Virtual Machine (such as HotSpot, Harmony) OS (e.g. SolarisTM, Linux, Windows) Introduction • Java language and Java Platform • Java Language: With powerful features, gained considerable success • Java Platform: Enable users to write and run applications written in Java languages directly on top it • Java Applications and Platforms are Used Everywhere • Almost every browser runs JVM and supports running Java Applets • Java Applet: A Java program that is run from inside a web browser

3. An Example of Java Applet Internet Explorer browses a webpage containing a Java Applet Webpage content: <APPLET NAME="QuizMaster" CODE="QuizMaster.class" ….> …. </APPLET> - From http://www.realapplets.com/applets/quizmaster/

4. Risks • What can happen if a Java applet/Java program is allowed to access any system resources? • Compromise the security of the local host • Disclose the confidential information: Transmit information • Destroy system files • Display annoying pictures on a user screen • How the JVM prevents this from happening?

5. Java Fine-grained Access Control Model • Security Policies • Map code to sets of permissions to access security sensitive resources grant codeBase “URL” { permission java.io.FilePermission “/home/tmp/", "read"; } • Runtime Monitoring in JVM • Through a call to library class SecurityManager, before accessing any sensitive resources

6. A Typical Use of SecurityManager public FileInputStream(File file) throws FileNotFoundException { String name = (file != null ? file.getPath() : null); // Existing Solutions SecurityManager security = System.getSecurityManager(); // checks for SM if (security != null) { security.checkRead(name); permission check } …… open(name); sensitive operation implemented as a native method } Potential Problem: Without checking the permissions, any file specified by the application can be opened, leading to potential confidential information leaks. Challenge: Ensure that the SecurityManager is consulted on all paths that lead to sensitive operations

7. Our Goal • Ensure Java Standard Libraries are trustworthy • They satisfy Complete Mediation Property: • SecurityManager is consulted on all paths that lead to sensitive operations • This Assurance will benefit millions of Java users • Sensitive Operations • Implemented as native methods • Code that is specific to a hardware and operating system platform and are written in other languages, such as C and C++. Called by Java code. • Challenges to Our Goal: • Large code base - Thousands of classes in Java libraries • Existing Model Checking tools (e.g. SLAM, MOPS) are general purpose oriented, while our goal is specialized to verify complete mediation property. These generic tool may not be scalable to a large code base such as the Java libraries

8. Input: Java Standard Libraries An automated static analysis technique. Sound. Not Complete Security Property Verifier Output: Yes Some paths from some public methods to sensitive operations are missing security checks. No All paths from public methods to sensitive operations are guarded by security checks. Our Solution

9. Our Contributions • A scalable automated model checking technique that verifies satisfaction of complete mediation in open systems • A tool, Complete Mediation Verifier (CMV) • Efficiently analyzes JVM library classes • Experimental results in two shrink-wrapped JVM implementations -- HotSpot and Harmony VMs.

10. public void X () { // do some operations Y(); //sensitive-operation FileRd1(); } A Code Example to Walk through Various Verification Approaches private void Y() { if (….){ SecurityManager sm = System.getSecurityManager(); if (sm != null) { sm.checkPermission( FilePermission(pathname”read”)); } // sensitive-operation FileRd2(); } else { // other operations } } Risky A branch w/o a security check

11. Y0 Y1 Y5 Y2 Y3 Y4 safe RET Analysis : Naive First Approach - private void Y() { 0. if (….){ 1. SecurityManager sm = System.getSecurityManager(); 2. if (sm != null) { 3. sm.checkPermission( FilePermission(pathname, ”read”)); } // sensitive-operation 4. FileRd2(); } else { 5. // other operations } }

12. X0 Y Y0 CFG(X) Y1 Y5 X0 Y2 Invokes Y X1 Expand To Y3 X2 Y4 RET At risk X2 RET First Approach (Contd.)- Presence of Method Calls • Expand and analyze CFG inline to make Extended CFGs(ECFG) • Inline Expansion Drawbacks: • Recursion • Size of ECFG(M) – can be exponential public void X(int x) { 0: // do some operations 1: Y(); //sensitive-operation 2: FileRd1(); }

13. EN RT EN Summarization • High-level idea: Analyze methods once; “summarize” and reuse the results • All_Path_Secure • No paths to return node(s) w/o security check. • Otherwise, Insecure_Path • Good • No paths to unguarded sensitive operations. • Otherwise, Bad • We store negations of these two properties • Method Summary • A 2-tuple to store <Insecure_Path, Bad> • 4 kinds of values • <insecure_path, bad> • <insecure_path, ┴> • <┴, bad> • <┴, ┴>

14. To compute Bad attribute: A node that Invokes a bad method M1 A sensitive node equivalent Invokes a All_Path-Secure and Good method M1 A security check node equivalent To compute Insecure_path attribute: Invokes a Insecure_Path method M1 a neutral node equivalent Observations to Compute Method Summary

15. Y0 Step 1: Construct call graph Y1 Meth. X Y5 Y2 Meth. Y Y3 Y4 safe RET CFG(Y) Summary(Y) = <InSecure_Path, ┴ > Step 3: Analyze CFG(Y) to Compute Summary(Y) Second Approach –Computing Summaries of Methods Given two methods X & Y. X invokes Y. Step 2: Find out Y needs to be analyzed first; The summary of the method computed first can then be reused. In presence of multiple methods, list the methods in reverse topological order

16. Step 4: - Compute Summary(X) X0 At risk invocation Node X1 is equivalent to a neutral node X2 RET CFG(X) Summary(X) = <InSecure_Path, Bad> Second Approach (Contd.) Invokes Y, Summary(Y) = <InSecure_Path, ┴ > X0 X1 X2 RET CFG(X)

17. Second Approach - Algorithm Algorithm in summary: Construct call Graph; Get a list of methods in Reverse Topological Order(RTO); For each method M in the list ordered in RTO Compute Summary(M), reusing the method summaries of those invocation nodes; Runtime complexity: O(ΣM) • ΣM the sum of the sizes of the CFGs of all the methods.

18. Meth. M1 Meth. M2 A cycle exists in the call graph Expanded Second Approach- For Recursion • Let us say M1 and M2 are mutually recursive methods • Analysis based on reverse topological order will not work • Solution: • Repeat the previous algorithm, until all method summaries reach a fix point. • Worst case run time: • O(N* ΣM). Quadratic • N the number of methods analyzed • Quadratic algorithms still problematic for a large code base such as libraries • How can we make this more efficient?

19. Step 1: Add X0, Z0 to Q Q X0 Z0 Z0 Step 2: X0 is a neutral node. Add its successor X1 to Q Q Z0 X1 Z1 X0 Invokes Z Step 3: Z0 is a neutral node. Add its successor Z1 to Q Z1 Q X1 Z1 X1 Step 4: X1 is a method invocation node. Add X1 to WQ(Z). RET Q Z1 X1 WQ(Z) X1 RET Step 5: Z1 is a security check. Stop exploring the path. Q WQ(Z) X1 Step 6: Q is empty. Terminate the algorithm. Both X and Z are All_Path_Secure. Our Approach – A New Efficient Solution (Part 1 – Compute Insecure_Path) • Main Idea: • Compute method summaries for all methods, at the same time • For Insecure_Path summaries, process all the methods at the same time • Start with entry nodes in a queue, and keep adding successors to the queue as you explore • Linear run time complexity: O(ΣM) • ΣM sum of sizes of the CFGs of all methods public void X(int x) { 0: // do some operations 1: Z(); //sensitive-operation 2: FileRd1(); } private void Z() { 0: SecurityManager sm = System.getSecurityManager(); 1: sm.checkPermission( FilePermission(pathname, ”read”)); //sensitive-operation 2: FileRd2(); }

20. Step 1: Add X0, Z0 to Q Q X0 Z0 Z0 Step 2: X0 is a neutral node. Add its successor X1 to Q Q Z0 X1 Z1 X0 Invokes Z Step 3: Z0 is a neutral node. Add its successor Z1 to Q Z1 Q X1 Z1 X1 Step 4: X1 is a method invocation node. Insecure_Path(Z) is false. Add Method X to WQ(Z). RET Q Z1 X1 WQ(Z) X RET Step 5: Z1 is a security check. Stop exploring the path. Q WQ(Z) X Step 6: Q is empty. Terminate the algorithm. Both X and Z are Good. Our Approach - A New Efficient Solution (Part 2 – Compute Bad Summaries) • Main Idea (Similar to the computing of Insecure_Path ) • process all the methods at the same time. • Different in processing of each node in the queue Q • for a sensitive operation, label method summary as bad • for a node invoking a method M’, • If Insecure_Path(M’) is true, put all successors to the Q • Put the node to a waiting queue of M’ • Whenever the bad summary of M’ is computed, process the node based on summary of M’ • Linear run time complexity:O(ΣM) • O(ΣM) sum of sizes of the CFGs of all methods public void X(int x) { 0: // do some operations 1: Z(); //sensitive-operation 2: FileRd1(); } private void Z() { 0: SecurityManager sm = System.getSecurityManager(); 1: sm.checkPermission( FilePermission(pathname, ”read”)); //sensitive-operation 2: FileRd2(); }

21. CFG(Z) Z0 At risk Z1 Invokes Y Z2 RET Our Improvement to the New Approach • Current solution: • Constructs and explores CFG(Z) and CFG(Y) • Computes Summary(Y) and Summary(Z). • However, to identify a risky path in Z, • No need to explore the CFG(Y) and compute Summary(Y) • No need to compute Insecure_Path attribute of Z • Observations • Our goal is to identify paths from public methods to sensitive operations without security checks. • To achieve such a goal, sometimes there is no need to compute some method summaries. public void Z() { 0: // do some operations //sensitive-operation 1: FileRd1(); 2: Y(); // Y is a non-public method } How to make the algorithm even more efficient?

22. Our Approach: On-the-fly Approach - A Even More Efficient Solution • Start to process all public methods first • By putting their entry nodes of CFGs to a queue • While exploring the CFGs, if a method invocation node is encountered and is never visited before, add its entry node to the queue to explore its CFG. • Whenever its Bad attribute or Insecure_Path attribute is computed, replace the node with a non-method-invocation node based on its summary • For a return node, label the method as Insecure_Path • For a sensitive operation, label the method as Bad • For a security check, stop exploring that path

23. Step 1: Add Z0 to Q Q Z0 CFG(Z) Z0 Step 2: Z0 is a neutral node. Add its successors to Q At risk Z1 Q Invokes non public method Y Z1 Z2 Step 3: Z1 is sensitive, label method Z as Bad RET Q Step 4: Q is empty. Terminate the algorithm An Example to Illustrate the On-the-fly Approach public void Z() { 0: // do some operations //sensitive-operation 1: FileRd1(); 2: Y(); // Y is a non-public method }

24. Runtime of On-the-fly Approach • In comparison to the non-on-the-fly approach: • No need to construct the Call Graphs to get all methods! • Construct the CFG and compute the method summary of each method on demand. • In worst case, linear run time complexity O(ΣM) • ΣM sum of sizes of the CFGs of all methods

25. Method format: <Declaring class Name>:<Return Type>:<MethodName(param list)> Bold highlights risky methods java.net.URL: void <init>(URL, String) java.io.ObjectInputStream: java.lang.Object readObject represents other risky methods java.net.URL: java.net.URLStreamHandler getURLStreamHandler(String) java.net.DatagramSocket: void createImpl() java.io.ObjectInputStream: java.lang.Class resolveClass(ObjectStreamClass) java.lang.Class: java.lang.Class forName(String,boolean, ClassLoader) <java.lang.Class: java.lang.Class forName(String) java.lang.Class: java.lang.Class forName0(String, boolean, ClassLoader) Witness Generation & Analysis • Witness - A call chain to show a counter example, i.e. • An unguarded path to a sensitive operation, or • An insecure path.

26. Experiment & Results Risky method: A public and bad method. Real Risky: a risky method that has at least one feasible bad path in practice.

27. Experiment & Results (Contd.) • The average time taken by CMV to analyze each class was 74 seconds • The bulk of the time is spent on Call Graph construction • Preloaded method summary database storing summaries • Risky methods: Need further analysis • In HotSpot VM, only 61 methods to be analyzed, vs. code review on 1520 methods, a reduction in two orders of magnitude! • Automated witness analysis: further reduce human review efforts

28. Future Work #1 - Method Overriding • A method in base class can be overridden by child classes • For a method invocation node, method summaries from base and all child classes should be considered. • Our Solution • Transform the code of base class by adding the n-way branch statement before the first statement, to make a call to each overriding method of child classes. • Method summary in each branch is computed

29. Future Work #2- Exceptions • Code can throw exceptions. Control flow is changed • Issues • Throw statement • If catch block exists, add an edge from throw to catch block • Add an edge from throw to return node • Method invocation node • Similar to Throw statement • Rethrow statement, i.e. a throw in a catch block • Replace the node with a return node • Solutions: Add new edges

30. Future Work #3 - Check Non-native Sensitive Operations • Sensitive operations are not implemented as native methods. For examples: • Setting a Sensitive Private Data Member • Returning a Sensitive Private Data Member • Returning a Password • Returning Network IP Address • Solutions • Identify these sensitive operations, and then apply the same approach (to the handle of sensitive native methods), to compute method summaries.

31. Future Work #4 - Automated Sensitive Operation Detection • To automate the detections of sensitive operations in Java libraries. • Challenges- Hard to identify them due to: • Some native methods are non-sensitive • They do not access sensitive system resources • The parameters passed to a sensitive operation: another factor (in addition to the method name) to determine if a method is a sensitive operation and the sensitive type (e.g. read or write a file)

32. Future Work #5 - Automated Permission Checks Detection • Determine the types of permission checks for each sensitive operation. • Challenges • Needs to first identify the sensitive operations in JVM. • Should also determine the permission checks for those sensitive operations performed in Java libraries, but without permission checks, due to an undetected defect.

33. Future Work #6 - Certification • Given a method say M0 that is claimed to be Good or All_Path_Secure, verification is needed to ascertain its correctness. • Our Solution • We have initiated a novel and scalable verification technique to certify the correctness of method summaries. • Experiments are ongoing, and we will report our work in the future.

34. Related work • Model checking • Several general purpose: MOPS, SLAM, Bandera, C-Wolf • Ours is specialized for this problem; property-specific customization makes it scalable • Jensen et al [IEEE S&P99] • Algorithms for verification of closed systems • Our approach checks open systems (e.g., libraries) • Static analysis • Bug-finding [METAL, CQUAL] • Checking complete mediation [Zhang et al [Security’03], Fraser [PLAS06] • These techniques require non-trivial transformations to Java-like libraries • E.g. security checks and sensitive ops in different methods • Retrofitting code for authorization • Naccio, SASI, Ganapathy et al [IEEE S&P06] • One requires retrofitting only when verification reports unsafe methods

35. Tool & Paper • Tool: Complete Mediation Verifier (CMV) • Developed. To be released by Dec. 2008 • Paper: CMV: Automatic Verification of Complete Mediation for Java Virtual Machines • Accepted by ACM Symposium on Information, Computer and Communications Security (ASIACCS’08), to be held in Tokyo, Japan. March 18-20, 2008 • Collaborators: • Hilary Branske • Prof. Prasad Sistla • Prof. V.N.Venkatakrishnan

36. Thank you!