Security Patterns for Operating Systems by Ed Fernandez and Tami Sorgente

1. Security Patterns for Operating Systemsby Ed Fernandez and Tami Sorgente

2. Security and operating systems (OS) • OS act as an intermediary between the user of a computer and its hardware • OS supports the execution of all the applications • The OS architecture is fundamental in the organization its components and for utilization of these component services in a given application • It is the most critical of the software layers because compromise can affect all applications and persistent data • Most reported attacks occur through the OS • Security Patterns for Operating Systems

3. Security Patterns for Operating Systems • Published Patterns: • File access control • Controlled Virtual Address Space • Reference Monitor • Controlled Execution Environment • Authenticator • Controlled-Process Creator • Controlled-Object Factory • Controlled-Object Monitor • OS Architecture Patterns • Future Pattern Ideas: • Process structure • Thread structure • Administration roles • Execution Rings • Secure process communication • Protected entry points • Distributed directory • Isolated execution environments • Virtual Address Space Structure

4. O.S. Access Control Pattern Language

5. Patterns for Operating Systems Access Control • File access control. How do you control access to files in an operating system? Apply the Authorization pattern to describe access to files by subjects. The protection object is now a file component that may be a directory or a file. • Controlled Virtual Address Space. How to control access by processes to specific areas of their virtual address space (VAS) according to a set of predefined access types? Divide the VAS into segments that correspond to logical units in the programs. Use special words (descriptors) to represent access rights for these segments. • Reference Monitor. How to enforce authorizations when a process requests access to an object? Define an abstract process that intercepts all requests for resources and checks them for compliance with authorizations. • Controlled Execution Environment. How to define an execution environment for processes? Attach to each process a set of descriptors that represent the rights of the process. Use the Reference Monitor to enforce access. [Fer02] E.B.Fernandez, "Patterns for operating systems access control", Procs. of PLoP 2002, http://jerry.cs.uiuc.edu/~plop/plop2002/proceedings.html

6. Subject ID ProtectionObject ID AuthorizationRule Right accessType predicate copyFlag checkRights ( ) The Authorization pattern * *

7. Subject ID * AuthorizedOn * Workstation ID Authorization priority privileges startSession ( ) Authorized For ACLE accessMode FileComponent HasHomeDirectory File name size createFile ( ) save ( ) * Directory name The File Authorization pattern

8. Virtual Address Space Access Control pattern

9. The Execution Domain pattern

10. Reference Monitor pattern

11. Controlled Execution Environment pattern

12. More patterns for operating systems access control • Authenticator.How to verify that a subject is who it says it is? Use a single point of access to receive the interactions of a subject with the system and apply a protocol to verify the identity of the subject. • Controlled-Process Creator. How to define the rights to be given to a new process? Define their rights as part of their creation. • Controlled-Object Factory. How to specify rights of processes with respect to a new object? When a process creates a new object through a Factory, the request includes the features of the new object. Among these features include a list of rights to access the object. • Controlled-Object Monitor. How to control access by a subject to an object? Use a reference monitor to intercept access requests from processes. The reference monitor checks if the process has the requested type of access to the object. [Fer03]  E. B. Fernandez and J. C. Sinibaldi, “More patterns for operating system access control”, Proc. of  the 8th European conference on Pattern Languages of Programs, EuroPLoP 2003,  http://hillside.net/europlop, 381-398.

13. Authentication Pattern

14. Class model for PKI authentication

15. Controlled-Process Creator

16. Controlled Object Creator

17. Controlled Object Monitor

18. OS architectures • Patterns representing an abstract view of the four basic architectures of operating systems: • Modular Operating System Architecture • Layered Operating System Architecture • Microkernel Operating System Architecture • Virtual Machine (VM) Operating System Architecture

19. Operating System Architectures

20. The Modular Operating System Architecture pattern • An object oriented approach is used to dynamically load and link loadable modules to the core component of the OS

21. Modular Operating System Architecture • Known uses • Solaris version 10 • ExtremeWare from Extreme Networks [Ext]. • Related patterns • The Controlled Execution Environment pattern [Fer0] can be used to isolate executing modules.

22. Layered Operating System Architecture • Separate the OS into layers • OS has more control over separation of concerns • Overall features and functionality are separated into layers • Clearly defined interfaces between each kernel section of the OS and between user applications and the OS functions

23. Layered Operating System Architecture Pattern • Define a hierarchical set of layers and assign components to each layer. Each layer presents an abstract machine to the layer above it.

24. Layered Operating System Architecture Pattern

25. Layered Operating System Architecture Pattern Dynamics • Sequence diagram for opening and reading a disk file • A user sends an openFile( ) request to the OSInterface • The OSInterface interprets the openFile( ) request • The openFile( ) request is sent from the OSInterface to the FileManager • The FileManager sends readDisk( ) request to the DiskDriver

26. OS Layered Architecture

27. Layered Operating System Architecture • Known uses • OS/2 of IBM, Symbian OS [Sym01] • Related patterns • specialization of the Layers architectural pattern [Bus96]. • A security version of the layers pattern is presented in [Fer02] and in [Som05].

28. Microkernel Operating System Architecture • Move as much of the OS functionality from the kernel space • Very basic set of functions in microkernel • Use external and internal servers

29. Microkernel Operating System Architecture pattern

30. Microkernel Operating System Architecture • Known uses • PalmOS [PalmOS], QNX [QNX] • Related patterns • specialization of the microkernel pattern [Bus96].

31. Virtual Machine Operating System Architecture • Provides a set of replicas of the hardware architecture to separate operating systems • Strong isolation between each OS

32. Virtual Machine Operating System Architecture pattern • Define an architectural layer that is in control of the hardware and supervises and coordinates the execution of each OS environment. • This extra layer, usually called a Virtual machine Monitor (VMM) or Hypervisor presents to each operating system a replica of the hardware. • The VMM intercepts all system calls and interprets them according to the OS from where they came.

33. Virtual Machine Operating System Architecture pattern

34. Virtual Machine Operating System Architecture example resolved • Two companies using Unix and Linux in different virtual machines

35. OS Pattern Ideas/ Future Work • Process structure • Thread structure • Administration roles • Execution Rings • Secure process communication • Protected entry points • Distributed directory • Isolated execution environments • Virtual Address Space Structure

36. Process structure • Process Control Block (PCB), • a data structure containing its id, a reference to its code, program counter, stack, registers, status, and references to its resources (domain). • A process receives a separate address space for its execution. • Some architectures, e.g., the Intel X86 series, have hardware support for the type of PCB used by common operating systems (this accelerates context switching, changing execution from one process to another).

37. Thread structure • Lightweight process • Faster context switching than a process • shares its address space with other threads. • Typically, a thread includes a program counter, a register set, and a stack. • Because of its shared address space, an error or attack from another thread can corrupt its memory. • Thread stacks can be protected if they are kept in the system address space using separated segments or pages.

38. Thread structure • Most modern operating systems, e.g., Solaris, allow several threads to be bundled in one process; this protects the thread group as a whole from other processes. • User processes and threads can be created with special packages, e.g., Posix in Unix, or through the language, as in Java or Ada. • The operating system defines kernel threads as units of concurrent execution. • Kernel threads usually don’t have any protection against each other.

39. Administration roles • AIX reduces the privileges of the system administrator by defining five partially-ordered roles: Superuser, Security Administrator, Auditor, Resource Administrator, and Operator. • Windows Windows NT uses four roles for administrative privileges: standard, administrator, guest, and operator. A User Manager has procedures for managing user accounts, groups, and authorization rules. • Trusted Solaris ---This is an extension of Solaris 8. RBAC is used to restrict the rights of administrators, there is no superuser with all power. • Argus Pitbull [Arg]—Least privilege applied to all processes, including the superuser. The superuser is implemented using three roles: Systems Security Officer, System Administrator, and System Operator. [Arg] Argus Systems Group, “Trusted OS security: Principles and practice”, http://www.argus-systems.com/products/white_paper/pitbull [Cam90] N.A.Camillone , D.H.Steves, and K.C.Witte, “AIX operating system: A trustworthy computing system”, in IBM RISC System/6000 Technology, SA23-2619, IBM Corp., 1990, 168-172.

40. Execution Rings • Some hardware architectures define a set of protection rings (typically 4 to 32) with hierarchical levels of trust. • Rings are a generalization of the concept of mode of operation and define finer domains of execution. • They have been used in Multics and the Intel processors among others. • A combination (process, domain) corresponds to a row of the access matrix. • Crossing of rings is done through gates that check the rights of the crossing process. A process calling a segment in a higher ring must go through a gate.

41. Execution Rings

42. Execution Rings • In this case, processes are assigned to rings based on their level of trust; for example, if we had four rings we could assign them in decreasing order of privilege to: supervisor, utilities, trusted user programs, untrusted user programs. The Intel X86 architecture applies two rules: • Calls are allowed only in a more privileged direction, with possible restriction of a minimum calling level. • Data in a segment at level p can be accessed only by a program executing at a more privileged level (<= p).

43. Execution Rings • The Program Status word of the process indicates its current ring and data descriptors also indicate their assigned rings. These values are compared to apply the transfer rules. • Ring crossing applies protected entry points. A process calling a higher-privilege process can only enter this process at pre-designed entry points with controlled parameters. • Jumping many rings is suspicious. Multics defined a call bracket, where calls are allowed only within rings in the bracket. More precisely, for a call from procedure i to a procedure with bracket (n1, n2, n3) the following rules apply: if n2<i<=n3 the call is allowed to specific entry points; if i>n3 the call is not allowed, if i < n1 any entry point is valid. This extension only makes sense for systems that have many rings. • Rings don’t need to be strictly hierarchic, partial orders are possible and convenient for some applications. For example, a system including a secure database system could assign a level to this database equal but separated from system utilities; the highest level is for the kernel and the lowest level is for user programs. This was done in a design involving an IBM 370 [Fer78].

44. Secure process communication • Process communication also has an effect on security. • Systems that use explicit message passing have the possibility of checking each message to see if it complies with system policies.

45. Protected entry points • Process communication also has an effect on security. • Systems that use explicit message passing have the possibility of checking each message to see if it complies with system policies. • For example, a security feature that can be applied when calling another process is protected entry points. • A process calling another process can only enter this process at pre-designed entry points. This prevents bypassing entry checks. • The number and size of arguments in a gate crossing can also be controlled (this may protect against some types of buffer overflow attacks).

46. Distributed directory • Windows 2000 introduced the concept of Active Directory (AD) that provides an administrative structure for distributed domains. • The AD is a tree that acts as a repository or registry for all resource information. A domain is a group of systems that share the same directory; in this sense the AD is a directory of directories (the Composite pattern again !). • Each domain has a unique name and provides centralized administration for user and group accounts. Users can cross domains after being properly authenticated.

47. Distributed directory • The AD introduces the concept of dynamic inheritance of rights along the tree. This is a direct application of the concept of implied authorization that we proposed in 1975 [Fer75]. • Implied access rights are very convenient for system administration because the administrator can understand better the structure of rights in the system. • W2000 also improved and expanded the use of groups for security and introduced the use of Kerberos for distributed authentication

48. Isolated execution environments • Virtual machines • Tagged storage • Capabilities • Separation by encryption [Dal01] C. Dalton and T.H. Choo, “An operating system approach to securing services”, Comm. of the ACM, vol. 44, No. 2, February 2001, 58-64.