Chapter 7

1. Securing Commercial Operating Systems Chapter 7

2. Chapter Overview • Retrofitting Security into a Commercial OS • History of Retrofitting Commercial OS's • Commercial Era • Microkernel Era • UNIX Era • IX • Domain and Type Enforcement • Recent Unix Systems • Summary

3. Retrofitting Security into a Commercial OS • Requires reference Monitor Concept • Complete Mediation • Tamperproofing • Verifiability

4. Complete Mediation Challenges • Identify all security-sensitive operations • Some embedded deep inside the kernel code. • Examples: • Open • Sockets • Shared memory, etc. • Covert channel identification is usually not even attempted

5. Tamperproofing Challenges • Obvious: place in ring 0, but • Kernel is often updated. • /dev/kmem, /proc, Sysfs, netlink sockets • → Every root process must STILL be part of the UNIX TCB

6. Verification Challenges • Must: • Mediation is implemented correctly, but • Mediation interface designed manually • Implemented in unsafe languages • Policy enforces required security goals • Large number of queries and processes. • Complicate policies. • Reference monitor implementation is correct • Rest of TCB is huge. • Rest of the TCB behaves correctly.

7. History of Retrofitting Commercial OS's • Commercial Era • Emulate system on security kernel • Retrofit security into OS • → UNIX MLS • Microkernel Era • Independent Server Processes → went to kernel • New security models addressing both confidentiality and integrity • Unix Era • Composed solutions from the two eras with focus on system integrity.

8. Commercial Era • Emulated Systems • Data Secure UNIX • KSOS • KVM/370 – 25% Performance overhead • VAX/VMS DEC/Sandia Labs: MLS • Secure Xenix (IBM) Access control and auditing • Added Compatibility • Retrofitted Unix services • Hidden subdirectories – Polyinstantiated file systems • Trusted Path (Secure attention sequence) • 1990 saw many secure Unix variants

9. Microkernel Era • Goal: minimal size kernel emphasizing system abstractions; no emphasis on security per se. • Source: Mach (1980's) • Trusted Mach (Tmach) • Distributed Trusted Mach (DTMach) • Distributed Trusted OS (DTOS) • Flask

10. Trusted Mach • Built by Trusted Information Systems (TIS) • Added MLS for files, memory. • Aim was to provide function for other systems like Unix and Windows. (Single level)

11. Distributed Trusted Mach • Secure Computing Corporation and NSA • Hybrid access control model: • MLS labels for confidentiality • Type Enforcement labels for integrity (TE) • Similar architecture to Tmach + servers for networking and general security policy server.

12. DTMach II • DTMach = Mach + security server • Security server = reference monitor outside the kernel • Each port access implies an authorization query • For example, opening a file opens a port to the file server, etc. • Security server used both MLS and TE rules. • TE rules: • code could only be modified by administrators • Limited code that could be executed • There were limitations: • For example, there was an arbitrary send right...

13. Distributed Trusted OS (DTOS) • AIM: True reference monitor in the Mach microkernel. • Richer set of operations for ports than just send. • Microkernel: • Managed labeling of subjects and kernel objects. • Mediated each kernel operation by querying security server. • Focus on verifiability of microkernel and TCB.

14. Flask • Fluke was a second generation microkernel developed at University of Utah, better than Mach. • Flask = DTOS – Mach / Fluke • More emphasis on TE.

15. UNIX Era • By early 1990's, many Unices had MLS support. • Search for adding integrity (very ad-hoc at this point). • Cover two systems: • IX • DTE

16. IX • AT&T prototype, enforces MLS and integrity. • Includes a reference monitor over file access • Mandatory access control policy providing both confidentiality and integrity protections. • Care has been taken to prevent tampering in the TCB. • Verification not a goal. • MLS was high water mark, for files and processes. However processes could not go beyond a certain ceiling.

17. IX (2) • Integrity was LoMac, with floors. • Since levels are dynamic, each data transfer must be checked/mediated. • No memory-mapped files. • Trusted paths/pipes between processes (pex); a pex includes a label for the process at each end so that only that process may work with it.

18. An assured pipeline in IX

19. Domain and Type Enforcement • Trusted Information Systems: • Problem: protecting TCB from vulnerable root processes • Runs on Tmach system, but reference monitor added to OSF/1.

20. DTE Policy Model • Subject types are now called Domains, object types are still types. • Each domain is a triple (access rights to objects, access rights to subjects in other domains (signals), entry point program) • A domain describes how a process accesses files, signals other processes and creates processes. • DTE Unix defines limited protection domains for root processes. Key point is “least privilege”. • Domain transitions are limited and their execution is limited also. • Labeled Networking.

21. Recent Unix Systems • BSD variants • Trusted BSD • MAC, auditing, authentication • Reference monitor interface similar to LSM • SEBSD is a version of SELinux for BSD • FreeBsd Jail • OpenBSD emphasizes correct coding and configuration • Code separation • Buffer overflow protection • Least privilege configurations • NetBSD • In-kernel authentication and verification of file execution • Veriexec

22. Summary