Securing Untrusted Code via Compiler-Agnostic Binary Rewriting

1. Securing Untrusted Code via Compiler-Agnostic Binary Rewriting Richard Wartell, Vishwath Mohan, Dr. Kevin Hamlen, Dr. Zhiqiang Lin The University of Texas at Dallas Supported in part by NSF, AFOSR, and DARPA

2. Software Fault Isolation (SFI) • Automatically rewrite binaries to make them safer • [Wahbe, Lucco, Anderson, Graham, SOSP 1993] Rewriter Untrusted code Safe code

3. Software Fault Isolation (SFI) • trusted & untrusted modules in common address space • Example #1: web browser plug-ins • Example #2: trusted system libraries inside untrusted application • Goal: protect trusted modules from untrusted ones • confine untrusted module behaviors • Example: Untrusted modules must obey trusted module interfaces • Blocks ROP attacks [Shacham, CCS 2007] kernel32.dll user.dll Trusted Untrusted eMule.exe

4. Inlined Reference Monitors (IRMs) • SFI foundation supports higher-level policies [Abadi, Budiu, Erlingsson, and Ligatti. CCS 2005] • Example: IRMs [Schneider, ISS 2000] • Enforces powerful policies: • program-specific (no other programs affected) • light-weight enforcement (minimize context switches) • Statefulness • Example: Adobe Reader may access the network (to check for updates) and may read my confidential files, but may not access the network after reading my confidential files. kernel32.dll user.dll Trusted IRM Untrusted reader.exe

5. A Brief History of SFI 1995 2000 2005 2010 XFI4 NaCl5 Wahbe1 PittSFIeld3 CFI / SMAC2 1: [Wahbe, Lucco, Anderson, and Graham. SOSP 1993] 2: [Abadi, Budiu, Erlingsson, and Ligatti. CCS 2005] 3: [McCamant and Morrisett. USENIX 2006] 4: [Erlingsson, Abadi, Vrable, Budiu, and Necula. SOSDI 2006] 5: [Yee, Sehr, Dardyk, Chen, Muth, Ormandy, Okasaka, Narula, and Fullagar. S&P 2009]

6. A Brief History of SFI 1995 2000 2005 2010 XFI4 Needs PDB NaCl5 Special GCC Wahbe1 RISC only PittSFIeld3 Special GCC CFI / SMAC2 Needs PDB All prior works require explicit code-producer cooperation 1: [Wahbe, Lucco, Anderson, and Graham. SOSP 1993] 2: [Abadi, Budiu, Erlingsson, and Ligatti. CCS 2005] 3: [McCamant and Morrisett. USENIX 2006] 4: [Erlingsson, Abadi, Vrable, Budiu, and Necula. SOSDI 2006] 5: [Yee, Sehr, Dardyk, Chen, Muth, Ormandy, Okasaka, Narula, and Fullagar. S&P 2009]

7. Reins: REwriting and IN-lining System • Main Discovery:  means of enforcing SFI for near arbitrary COTS binaries • no source code or debug info (assumed unavailable) • no disassembly listing • compiler-agnostic • real COTS binary features • interleaved code and data • computed control-flows • dynamic linking • event-driven callbacks • multithreading • Low overhead (~2%) • Formal machine-verification of policy enforcement

8. Binary Rewriting w/o metadata • Relocation information, debug tables and symbol stores not always available • Reverse engineering concerns • Perfect static disassembly without metadata is provably undecidable • Best disassemblers (IDA Pro) make many mistakes

9. Infeasibility of Perfect Disassembly • Disassemble this hex sequence • Undecidable problem FF E0 5B 5D C3 0F 88 52 0F 84 EC 8B

10. Separating Code from Data Original Memory Layout Rewritten Memory Layout Reins Binary Original Binary Rewritten Header Header IAT IAT .data .data .told (NX bit set) .text .tnew (NW bit set) Low Memory High Memory kernel32.dll user32.dll user32.dll kernel32.dll Denotes a section that is modified during static rewriting

11. De-Shingling Disassembly Byte Sequence: FF E0 5B 5D C3 0F 88 B0 50 FF FF 8B Disassembled Invalid

12. Aligning Instructions • Chunk instructions to 16 byte boundaries with targets at the beginning, and calls at the end [McCamant and Morrisett. USENIX 2006] Injected Instructions Alignment nops

13. Preserving Good Flows • Turn original code section into a dynamic lookup table Rewritten Binary Injected Instructions Alignment nops

14. Preserving Good Inter-module Flows • IAT data section locked non-writable Rewritten Code Original Code jmp [IAT:CreateWindow] jmp [IAT:CreateWindow] CreateWindow CreateWindow

15. Computed Inter-module Flows • computed jumps to trusted modules • dynamic linking (DLLs) • callbacks (event-driven programming) trusted library intermediary library (trusted) rewritten code caller callback stub callback callback_ret return trampoline

16. Results

17. IRM Synthesis • Enforced policies on Eureka email client (>1.6MB code): • Disallow creation of .exe, .msi, or .bat files • Disallow execution of Windows explorer as an external process • Disallow opening more than 100 SMTP connections • Malware policies: • Disallow creation of .exe, .msi, or .bat files • Successfully stopped virus propagation for real world malware samples Rewriter Policy Binary Policy-adherant binary

18. Formal Verification TCB • Formal verification of rewritten binaries • 1500 SLOC of 80-column OCamlcode • no shared code between verifier and rewiter • median verification time:  0.4 ms/KB code • Allows rewriter to remain completely untrusted! • rewriting deployable as an untrusted service Rewriter Policy Binary Verifier Policy-adherant binary

19. Compatibility Limitations • COM objects • Runtime code generation (JIT) • Undocumented OS callbacks

20. Conclusion • Reins finally opens the door to full-scale COTS native SFI for massively complex, real-world applications without source. • no source code, debug info, or disassembly (assumed unavailable) • compiler-agnostic • real COTS binary features • interleaved code and data, computed control-flows, dynamic linking, event-driven callbacks, multithreading • automated synthesis of monitor from policy specification • automated machine-verification • low runtime overhead (~2.4%) • successfully tested on real commercial applications (>3MB code) • Practical Applications: • safe reuse of untrusted commercial software in security-critical environments • rewriting on demand: rewriter deployable as an untrusted third-party service due to separate verifier

21. References • R. Wahbe, S. Lucco, T. E. Anderson, and S. L. Graham. Efficient software-based fault isolation. In Proc. ACM Sym. Operating Systems Principles, pages 203–216, 1993. • F. B. Schneider. Enforceable security policies. ACM Trans. Information and Systems Security, 3(1):30–50, 2000. • M. Abadi, M. Budiu, U. Erlingsson, and J. Ligatti. Control-flow integrity. In ACM Conference on Computer and Communications Security, pages 340-353, 2005. • S. McCamant and G. Morrisett. Evaluating SFI for a CISC architecture. In Proc. USENIX Security Sym., 2006. • Ú. Erlingsson, M. Abadi, M. Vrable, M. Budiu, and G. C. Necula. XFI: Software guards for system address spaces. In Proc. Sym. Operating Systems Design and Implementation, pages 75–88, 2006. • H. Shacham. The geometry of innocent flesh on the bone: Return-into-libc without function calls (on the x86). In Proc. ACM Conf. Computer and Communications Security, pages 552–561, 2007. • B. Yee, D. Sehr, G. Dardyk, J. B. Chen, R. Muth, T. Ormandy, S. Okasaka, N. Narula, and N. Fullagar. Native Client: A sandbox for portable, untrusted x86 native code. In Proc. IEEE Sym. Security and Privacy, pages 79–93, 2009.

22. Advantage over VMs • no air gap • IRM has controlled but direct access to system resources and other processes • no semantic gap • no dynamic instruction interpretation or translation • better performance • fewer context switches • light-weight VM logic essentially in-lined into code • formal verification • few VMs have been formally verified • each change to VM (e.g., to enforce new policy) requires re-verification of VM