Secure Operating System Architectures Patterns

1. Secure Operating System Architectures Patterns

2. Security and operating systems (OS) • OS act as an intermediary between the user of a computer and its hardware • OS supports the execution of all the applications • The OS architecture is fundamental in the organization its components and for utilization of these component services in a given application • It is the most critical of the software layers because compromise can affect all applications and persistent data • Most reported attacks occur through the OS

3. OS architectures • Patterns representing an abstract view of the four basic architectures of operating systems: • Modular Operating System Architecture • Layered Operating System Architecture • Microkernel Operating System Architecture • Virtual Machine (VM) Operating System Architecture

4. Operating System Architectures

5. OS functional components • Process Management • Memory Management • File Management • I/O Management • Networking • Protection System • User Interface

6. Object-oriented model of OS Components

7. Modular Operating System Architecture • Example • Our group is building a new OS that should support various types of devices requiring dynamic services with a large variety of security requirements. We want to dynamically add OS components, functions, and services, as well as tailor their security aspects according to the type of application. For example, a media player may require support to prevent copying of the contents. Or we could remove a module for which a vulnerability alert has been issued. • Context • Operating systems are large systems that must accommodate a variety of diverse applications. • Problem • We need to be able to add/ remove functions in the easiest way. How do we structure the functions for this purpose?

8. Modular Operating System Architecture • The possible solution is constrained by the following forces: • OSs for PCs and other types of uses require a large variety of plug-ins. New plug-ins appear frequently and we need the ability to add and remove them without disrupting normal operation. • Some of the plug-ins may contain malware, we need to isolate their execution so they do not affect other processes. • We would like to hide security-critical modules from other modules to avoid possible attacks. • Modules can call each other, which is a possible source of attacks.

9. The Modular Operating System Architecture pattern • An object oriented approach is used to dynamically load and link loadable modules to the core component of the OS

10. Solaris 10 Operating System

11. Solaris 10 Operating System

12. Modular Operating System Architecture • Advantages: • Flexibility to add/ remove functions contributes to security in that we can add new versions of modules with better security. • Each module is separate and communicates with other modules over known interfaces. We can introduce controls in these interfaces. • It is possible to partially hide critical modules by loading them only when needed and removing them after use. • By giving each executing module its own address space we can isolate the effects of a rogue module. • Liabilities: • Any module can see all the others and potentially interfere with their execution. • Uniformity of call interfaces between modules makes it difficult to apply stronger security restrictions to critical modules.

13. Modular Operating System Architecture • Known uses • Solaris version 10 • ExtremeWare from Extreme Networks [Ext]. • Related patterns • The Controlled Execution Environment pattern [Fer0] can be used to isolate executing modules.

14. Layered Operating System Architecture • Separate the OS into layers • OS has more control over separation of concerns • Overall features and functionality are separated into layers • Clearly defined interfaces between each kernel section of the OS and between user applications and the OS functions

15. Layered Operating System Architecture • Example • Our system is very complex and we would like to separate different aspects to handle them in a more systematic way. We want to control the use of OS components and services. • Context • Operating systems are large systems that must accommodate a variety of applications. • Problem • Structure the components into hierarchical layers.

16. Layered Operating System Architecture • The possible solution is constrained by the following forces: • Interfaces should be stable and well defined. Going through any interface could imply authorization checks. • Parts of the system should be exchangeable or removable without affecting the rest of the system. For example, we could have modules that perform more security checks than others. • Similar responsibilities should be grouped to help understandability and maintainability. This contributes indirectly to improve security. • We should control module visibility to avoid possible attacks from other modules • Complex components need further decomposition. This makes the design simpler and clearer and also improves security.

17. Layered Operating System Architecture Pattern • Define a hierarchical set of layers and assign components to each layer. Each layer presents an abstract machine to the layer above it.

18. Layered Operating System Architecture Pattern

19. Layered Operating System Architecture Pattern Dynamics • Sequence diagram for opening and reading a disk file • A user sends an openFile( ) request to the OSInterface • The OSInterface interprets the openFile( ) request • The openFile( ) request is sent from the OSInterface to the FileManager • The FileManager sends readDisk( ) request to the DiskDriver

20. Layered Operating System Architecture • Implementation – • List all units in the system and define their dependencies. • Assign units to levels such that units in higher levels depend only on units of lower levels. • Once the modules in a given level are assigned, define a language for this level. This language includes the operations that we want to make visible to the next level above. Add well-defined operation signatures and security checks in these operations to assure the proper use of the level. • Hide in lower levels those modules that control critical security functions.

21. OS Layered Architecture

22. Layered Operating System Architecture • Advantages: • Clearly defined interfaces between each OS layer and the user applications • Control of information using layer hierarchical rules using enforcement of security policies between layers Each core component is separate • Each layer hides existence of certain data structures, operations and hardware from higher levels • Lower levels can be changed without affecting higher layers

23. Layered Operating System Architecture • Liabilities: • It is not clear what to put in each layer • There may be less efficiency as information needs to go through each layer • The layers are restricted to interface only with immediate neighboring layers. This restriction reduces flexibility and provides additional over head for to go through adjacent layers in order to use the services of layers i+2 and greater or layers i–2 and less.

24. Symbian layered OS architecture

25. Unix layered OS architecture

26. Layered Operating System Architecture • Known uses • OS/2 of IBM, Symbian OS [Sym01] • Related patterns • specialization of the Layers architectural pattern [Bus96]. • A security version of the layers pattern is presented in [Fer02] and in [Som05].

27. Microkernel Operating System Architecture • Move as much of the OS functionality from the kernel space • Very basic set of functions in microkernel • Use external and internal servers

28. Microkernel Operating System Architecture • Example We are building an OS for a range of applications with different reliability and security requirements and a variety of plug-ins. We would like to provide OS versions with different types of modules, some more secure, some less so. • Context • Operating systems are large systems that require decomposition for control of system functions • Problem • In general purpose environments we need to be able to add new functionality and security

29. Microkernel Operating System Architecture • The possible solution is constrained by the following forces: • The application platform must cope with continuous hardware and software evolution; these additions may have very different security or reliability requirements. • Strong security or reliability requirements indicate the need for modules with well-defined interfaces. • We may want to perform different types of security checks in different modules, depending their security criticality. • We would like a minimum of functionality in the kernel so we have a minimum of processes running in supervisor mode. A simple kernel can be checked and this is good for security.

30. Microkernel Operating System Architecture pattern • Separate all functionality into specialized services and provide an efficient way to route requests to the appropriate servers • microkernel is the central communication for the OS • one microkernel and several internal and external servers • an adapter is used between the client and the microkernel or an external server

31. Microkernel Operating System Architecture pattern

32. Microkernel Operating System Architecture pattern dynamics

33. QNX Microkernel Architecture • The QNX Microkernel responsibilities include the following [QNX]: • IPC (interprocess communication) • low-level network communication • process scheduling • first-level interrupt handling

34. QNX Microkernel Architecture source: [QNX ]

35. Microkernel Operating System Architecture • Advantages: • Provides a good degree of security because of the well defined interfaces between servers. • Can add even more security by putting fundamental functions in internal servers. • Liabilities: • Communication overhead due to message passing

36. Microkernel Operating System Architecture Variants • The Microkernel OS Architecture Pattern can be combined with the Layered OS Architecture pattern • Many applications in the PalmOS do not use the microkernel services; they are handled automatically by the system • microkernel functionality is provided for internal use by system software or for certain special purpose applications

37. Microkernel Operating System Architecture Variants source: [PalmOS ]

38. Microkernel Operating System Architecture • Known uses • PalmOS [PalmOS], QNX [QNX] • Related patterns • specialization of the microkernel pattern [Bus96].

39. Virtual Machine Operating System Architecture • Provides a set of replicas of the hardware architecture to separate operating systems • Strong isolation between each OS

40. Virtual Machine Operating System Architecture • Example • A web server is hosting applications for two competing companies. These companies use different operating systems. We want to ensure that there is no access to their files • Context • Mutually suspicious sets of applications that need to execute in the same hardware. Each set requires isolation from the other sets. • Problem • Sometimes we need to execute different operating systems in the same hardware. How do we keep those OSs isolated from each other?

41. Virtual Machine Operating System Architecture • The possible solution is constrained by the following forces • Each OS has its own set of machine dependent features • When an OS crashes or is penetrated by a hacker, the effects of this situation should not propagate to other OSs in the same hardware

42. Virtual Machine Operating System Architecture pattern • Define an architectural layer that is in control of the hardware and supervises and coordinates the execution of each OS environment. • This extra layer, usually called a Virtual machine Monitor (VMM) or Hypervisor presents to each operating system a replica of the hardware. • The VMM intercepts all system calls and interprets them according to the OS from where they came.

43. Virtual Machine Operating System Architecture pattern

44. Virtual Machine Operating System Architecture pattern dynamics

45. Virtual Machine Operating System Architecture example resolved • Two companies using Unix and Linux in different virtual machines

46. Virtual Machine Operating System Architecture • Advantages • Each environment (VM) does not know about the other VM(s). • Errors or attacks to a given VM have no way to propagate to other VMs • Liabilities • Extra overhead in use of privileged instructions • It is rather complex to let VMs communicate to each other (If this is needed).

47. Virtual Machine Operating System Architecture • Known uses • IBM VM/SP, Vmware • Related patterns