IMPROVING THE RELIABILITY OF COMMODITY OPERATING SYSTEMS

1. IMPROVING THE RELIABILITY OF COMMODITY OPERATING SYSTEMS Michael M. Swift Brian N. Bershad Henry M. Levy University of Washington

2. Key Idea • Kernel extensions are a major source of system failures • Nooks isolates extensions within lightweight protection domains inside the kernel address space • Requires little or no changes to extension and kernel code • Still prevents most system failures

3. INTRODUCTION • Want to allow existing kernel extensions to execute safelyin commodity kernels because • Computer reliability remains anunsolved issue • Kernel extensions are increasingly popular in Windows and Linux • Account for most system failures (85% for Windows XP)

4. Why? • Programmers writing device drivers are often less experienced than kernel programmers • Number of variants prevent us from testing them completely

5. The Nooks approach • Works with commodity operating systems • Requires little or no changes to extension and kernel code • Prevents most but not all system failures • Lets kernel extensions reside in the kernel address space • Solution is practical, backward-compatible and efficient

6. PREVIOUS WORK (I) • Capability-based architectures, ring and segment architectures: • Enable construction and isolation of privileged subsystems. • Slow and do not address recovery issues

7. PREVIOUS WORK (II) • Microkernels: • Put extensions into separate address spaces • Slow and do not address recovery issues • Database recovery techniques: • Atomic transactions • Work well for file system • Often awkward and slow

8. PREVIOUS WORK (III) • Type-safe programming languages: • Must rewrite the kernel • Static analysis of extensions: • Can detect some errors

9. PREVIOUS WORK (III) • Virtual machines: • Can reduce the amount of code that can crash whole machine • Fails if extension executes inside the virtual machine monitor VM VM VM VM Monitor Hardware

10. REQUIRED MODIFICATIONS

11. NOOKS • Design for fault-resistance, not fault-tolerance: • System must prevent and recover from mostextension mistakes • Occupy middle ground between unprotected (Linux, Windows) and safe (SPIN, Java VM) • Design for mistakes, not abuse: • Exclude malicious behavior

12. Nooks goals • Isolation: • Isolate kernel from extension failures • Detect extension failures before they corrupt kernel • Automatic recovery from extension failures • Backward compatibility with existing systems and extensions

13. OS Kernel Isolation Interposition Object Tracking Recovery Kernel extensions Nooks functions • New system reliability layer:the Nooks Isolation Manager (NIM)

14. Nooks isolation mechanisms • Every extension executes within its own lightweight kernel protection domain • Communication between kernel and extensions must go through new kernel service, the Extension Procedure Call (XPC) • Similar to an RPC between two processes located on the same machine (Lightweight RPC)

15. Lightweight protection domains • Provide protection by having extensions execute with a different page table giving them • Read/write access to their own pages • Read only access to other kernel pages

16. Lightweight protection domains • Lightweight solution because extensions execute in kernel mode • A malicious extension could switch back to the kernel’s page table • Another could misuse DMA

17. Nooks interposition mechanisms • Ensure that • All extension-to-kernel and kernel-to-extension control flow goes through Nooks XPC mechanism • All data transfers between kernel and extension go through Nooks object tracking code • All interfaces are done through wrappers, similar to the stubs of an RPC package

18. Nooks object tracking functions • Maintain a list of kernel data structures accessed by each extension • Control all modifications to these structures • Provide object information for cleanup if object fails

19. Nooks object tracking functions • Extensions cannot directly modify kernel data structures • Object tracking code will: • Copy kernel data structures into extension address space • Copy them back after changes have been applied • Perform checks whenever possible

20. Nooks recovery functions • Detect and recover from various extension faults: • When an extension improperly invokes a kernel function • When processor raises an exception

21. Nooks recovery functions • Recovery helped by Nooks isolation mechanisms: • All access to kernel structures are done through wrappers • Nooks can successfully release extension-held kernel structures

22. IMPLEMENTATION • Inside Linux 2.4.18 kernel on Intel x86 architecture • Linux kernel provides • over 700 functions callable by extensions • over 650 extension-entry functions callable by the kernel • Most interactions between kernel and extensions go through function calls

23. Interposition Interposition Extension Extension Nooks Linux Kernel Nooks Isolation Manager Extensions are wrapped by Nooks wrapper stubs

24. Details • 22,000 lines of code • Linux kernel has 2.4 million lines • Makes no use of Intel x86 protection rings • Extensions execute at same protection level as kernel

25. Isolation • Two parts: • memory management:to implement lightweight protection domains • extension procedure call (XPC)

26. Memory management • All components share the same kernel address space • Each extension executes in a separate lightweight protection domain • Extension has: • Read-only access to kernel • Read-write access to its own domain

27. Lightweight protection domain • Each lightweight protection domain has • A synchronized copy of the kernel page table • Its own heap, a pool of stacks and other private data structures • Changing protection domains requires a change of page tables • Results in a TLB flush! • No protection against DMA misuse by extensions

28. Interposition (I) • Done by wrapper stubs all executing in kernel protection domain: • Before the call when kernel calls an extension • After the call when an extension calls the kernel • Wrappers: • Check parameters for validity • Implement call by value and result

29. Passing by value and result Variable i is increased after caller receivesserver’s reply Caller: … i = 0; abc(&i); … i = 0 abc(int *k){ (*k)++;} i = 1 i

30. Interposition (II) • Writing wrappers is not an easy task: • Requires knowing how parameters are used • Significant amount of wrapper sharing among extensions • Especially when extensions implement the same functionality

31. Object tracking • Records kernel objects and types manipulated by extensions

32. Recovery • Mostly through undoing

33. Limitations • Cannot prevent extensions from executing privileged instructions that would corrupt the kernel • Cannot prevent infinite loops inside an extension • Prevented by Linux semantics to do a thorough check of the parameters passed to the operating system • Current implementation of recovery assumes that extensions can be killed and restarted

34. Transparency • “Neither the extension not the kernel is aware of the Nooks layer” • In reality, must sometimes modify a few lines of extension code

35. Reliability • Tested eight extensions • Two sound card drivers • Four Ethernet drivers • A Win95 compatible file system (VFAT) • An in-kernel Web server • Injected 400 faults • 317 resulted in extension failures

36. Test results • Nooks eliminated 99% of the crashes observed with native Linux (313 out of 317) • Nooks is slower • VFAT benchmark spent 165 s in the kernel instead of 29.5 s for native Linux • Web server could only serve about 6,000 pages/s instead of 15,000 pages/s for native Linux Ouch!

37. PERFORMANCE • Ten percent performance penalty for sound and Ethernet drivers • Sixty percent performance penalty for in-kernel web server • (15,000 - 6,000)/15,000

38. Recovery errors • Big problems with VFAT extension • 90% of attempted recoveries resulted in on-disk corruption • VFAT extension cannot be safely killed and restarted We should expect that!

39. CONCLUSIONS • Nooks approach focuses on achievingbackward compatibility • Cannot provide complete isolation and fault-tolerance • Can still achieve “an extremely high level of operating system reliability” • Performance loss varies between 0 and 60%