UNIX: Background and the Traditional Process and Kernel Abstractions

1. UNIX:Background and the Traditional Process and Kernel Abstractions Fred Kuhns fredk@cse.wustl.edu Applied Research Laboratory, Department of Computer Science and Engineering, Washington University in St. Louis

2. A Little History First: UNIX • Initial design: Ken Thompson, Dennis Ritchie and others at AT&T's Bell Telephone Laboratories (BTL) in 1969. • AT&T made the source available to Universities for research and educational use. • 1973 UNIX Version 4 is a C rewrite of previous version. • The C language originally developed for use on the UNIX system by Dennis Ritchie • C evolved from 'B', developed by Thompson. • AT&T sold UNIX to Novel; Novel passed the UNIX trademark to X/OPEN and sold source code to Santa Cruz Operation (SCO). • Plan 9 is AT&T's successor to UNIX • AT&T unable to market UNIX as a product, source code made available to Universities for use in Research and Education. • Influential variant: Berkeley Software Distributions (BSD) distributed by the Computer Systems Research Group, University of California at Berkeley • Berkeley obtained UNIX from AT&T in December of 1974 • UNIX ported to many different architectures • Microsoft and SCO collaborated to port UNIX to the Intel 8086: XENIX • AT&T purchased 20% of Sun: result resulting in joint effort to develop SVR4 • In 1982 AT&T broken up and able to market UNIX. Released System III in 1982 and System V the following year. • System V UNIX introduced virtual memory (different from BSD, called regions), IPC (shared memory, semaphores, message queues), remote file sharing, shared libraries and STREAMS. CSE522– Advanced Operating Systems

3. BSD UNIX • 2 BSD : text editor vi • 3 BSD : demand-paged virtual memory • 4.0BSD : performance improvements • 4.1BSD : job control, autoconfiguration • 4.2/4.3BSD : reliable signals, fast filesystem, improved networking (TCP/IP reference implementation), sophisticated IPC primitives • 4.4 BSD : stackable and extensible vnode interface, network filesystem, log-structured filesystem, other filesystems, POSIX support, and other enhancements. CSE522– Advanced Operating Systems

4. Commercialization • Interactive Systems first commercial (1977) • Microsoft and SCO release XENIX (1980) • 1982 Bill Joy left Berkeley and founded Sun Microsystems. • SunOS originally based on BSD 4.2 • SunOS 5 (Solaris 2.X) was a collaborative effort based on System V, release 4 (SVR4). • AIX from IBM • HP/UX from Hewlett Packard Corporation • ULTRIX from Digitial Equipment Corporation, followed by DEC OSF/1. DEC purchased by Compaq CSE522– Advanced Operating Systems

5. CMU and MACH • As UNIX grew, the kernel became large and complex - originally small and elegant. • Mid-1980 CMU researches began development of a new OS: • microkernel providing a small set of essential services. • Support UNIX API • Support uniprocessor and multiprocessors • supported distributed environments • collection of servers provide necessary functionality • New source so not encumbered by AT&T licenses • OSF1 and NextStep based on MACH CSE522– Advanced Operating Systems

6. Standards • Problem: Incompatible programming APIs and core service implementations across the different UNIX variants • Solution: Standard set of interfaces. Several standards exist that define the functions, syntax and semantics: • IEEE POSIX (Portable Operating System Interface for computing environments or portable OS for UNIX) specifications. • System V Interface Definition (SVID) from AT&T • X/Open Portability Guide (XPG) from the X/Open Consortium CSE522– Advanced Operating Systems

7. Standards • IEEE POSIX specifications. • 1003.1 (1990) system interfaces and headers • 1003.1b (1993) realtime extensions • 1003.1c (1995) threads extensions (pthreads) • 1003.1 (1996) includes 1003.1 (1990), 1003.1b (1993), 1003.1c (1995), 1003.1i (1995) • 1003.1g protocol independent interfaces • 1003.2 (1992) Shell and utilities • 1003.2a (1992) Interactive shell and utilities • SVID: SVID4 is for SVR4 • defines the System V programming interface. Must be conformant to call an OS SVR5. AT&T published the System V Verification Suite (SVVS) CSE522– Advanced Operating Systems

8. Standards • XPG - formed to develop the Common Applications Environment (CAE). • XPG3 - Superset of POSIX.1 1998 and utilities from SVID3 • XPG4 - Superset of POSIX.1 1990, POSIX.2 (1992), POSIX.2 (1992), and extension from XPG3 • XPG4v2 (SUS or single UNIX specification or spc 1170 or UNIX 95) - superset of XPG4 with widely used BSD interfaces • XNS4 (X/Open Network Services) - sockets and XTI extensions • SUSV2 (UNIX98)- superset of SUS, POSIX.1b (1993), POSIX.1c (1996) and ISO/IEC 9899 (C standard). Includes CDE specification • XNS5 - superset of LP64 - clean derivative of XNS4 CSE522– Advanced Operating Systems

9. char block Device drivers Traditional Kernel Architecture execution environment application trap libraries user System call interface kernel File subsystem IPC System Services Process control subsystem Buffercache scheduler memory hardware CSE522– Advanced Operating Systems

10. Basic Concepts and Terminology • Two privilege levels, or modes: user and system • System (Interrupt) versus process context • Kernel space protected • operates with system level privileges • requires special instruction sequence to change from user to kernel mode • reentrant • Each process has own "protected" address space • Per process state (u_area) in kernel space • All process share same kernel space • Per process kernel stack (since kernel is re-entrant) CSE522– Advanced Operating Systems

11. Mode, Space and Context Privileged user kernel mode context process Application (user code) System calls Exceptions system space kernel X not allowed Interrupts System tasks UNIX uses only two privilege levels CSE522– Advanced Operating Systems

12. Overview • Process • compete for system resources • blocks if can not acquire a resource • Kernel: Trusted software component, interfaces with hardware, implements system abstractions and interfaces • manages resource - allocate resource to processes "optimally" • implements policy for resource allocation (sharing) • provide high-level abstract interface to resources • centralized management simplifies synchronization and error recovery CSE522– Advanced Operating Systems

13. The Process • Fundamental abstraction • Executes a sequence of instructions • Competes for resources • Address space - memory locations accessible to process • virtual address space: memory, disk, swap or remote devices • System provides features of a virtual machine • shared resources (I/O, CPU etc) • dedicated registers and memory • Most created by fork or vfork • well defined hierarchy: one parent and zero or more child processes. The init process is at the root of this tree • different programs may be run during the life of a process by calling exec • processes typically terminate by calling exit CSE522– Advanced Operating Systems

14. Process states fork USER RUNNING INITIAL IDLE system call, interrupt return fork swtch() exit KERNEL RUNNING RUNNABLE ZOMBIE swtch() wait sleep() continue stop wakeup stop STOPPED stop ASLEEP continue wakeup STOPPED ASLEEP CSE522– Advanced Operating Systems

15. Process Context • Address Space • text, data, stack, shared memory ... • Control information (u area, proc) • u area, proc structure, maps • kernel stack • Address translation maps • Credentials • user and group ids • Environment variables • variable=value • typically stored at bottom of stack • Hardware context • program counter • stack pointer • processor status word • memory management registers • FPU registers • Machine registers saved in u area's process control block (PCB) during context switch to another process. CSE522– Advanced Operating Systems

16. Kernel stack stack Process address space Data Text (shared) 0x00000000 Process Address Space (one approach) 0xffffffff Kernel address space 0x7fffffff CSE522– Advanced Operating Systems

17. proc struct Stack Stack Stack kernel stack/u area kernel stack/u area kernel stack/u area Data Data Data Text (shared) Text (shared) Text (shared) Big Picture: Another look kernel memory CSE522– Advanced Operating Systems

18. Process Control Information • U area. • Part of user space (above stack). • typically mapped to a fixed address. • contains info needed while running. • Can be swapped • Proc • contains info needed when not running. • Not swapped out. • traditionally fixed size table CSE522– Advanced Operating Systems

19. U Area PCB - HW context pointer to proc real/effective ids args, return values or errors to current syscall. Signal info file descriptor table controlling terminal vnode Proc structure process ID and group u area pointer process state queue pointers - scheduler, sleep, etc. Priority memory management info flags U area and Proc structures CSE522– Advanced Operating Systems

20. User Credentials • Every user is assigned a unique user id (uid) and group id (gid). • Superuser (root) has uid == 0 and gid == 0 • every process both a real and effective pair of Ids. • effective id => file creation and access, • real id => real owner of process. Used when sending signals. • Senders real or effective id must equal receivers real id CSE522– Advanced Operating Systems

21. The Kernel • program that runs directly on the hardware • loaded at boot time and initializes system, • creates some initial system processes. • remains in memory and manages the system • Resource manager/mediator • Time share (time-slice) the CPU, • coordinate access to peripherals, • manage virtual memory. • Synchronization primitives. • Well defined entry points: • syscalls, exceptions or interrupts. • Performs privileged operations. CSE522– Advanced Operating Systems

22. Entry into Kernel • Synchronous - kernel performs work on behalf of the process: • System call interface (UNIX API): central component of the UNIX API • Hardware exceptions - unusual action of process • Asynchronous - kernel performs tasks that are possibly unrelated to current process. • Hardware interrupts - devices assert hardware interrupt mechanism to notify kernel of events which require attention (I/O completion, status change, real-time clock etc) • System processes - scheduled by OS • swapper and pagedaemon. CSE522– Advanced Operating Systems

23. Trap or Interrupt Processing • Hardware switches to kernel mode, using the per/process kernel stack. • HW saves PC, Status word and possibly other state on kernel stack. • Assembly routine saves any other information necessary and dispatches event. • On completion a return from interrupt (RFI) instruction is performed. CSE522– Advanced Operating Systems

24. Interrupt handling • Asynchronous event such as disk I/O, network packet reception or clock “tick”. • Must be serviced in system context. • Must not block. • This does take CPU time away from the currently running process • Multiple interrupt priority levels (ipl) traditionally 0-7. • User processes and kernel operate at the lowest ipl level. • Higher level Interrupts can preempt lower priority ones. • The current ipl level may be set while executing critical code. CSE522– Advanced Operating Systems

25. Exception handling • Synchronous to the currently running process for example divide by zero or invalid memory access. • Must run the the current processes context. • An Interrupt can occur during the processing of an exception or trap. CSE522– Advanced Operating Systems

26. Software Interrupts • Interrupts typically have the highest priority in a system. • Software interrupts are assigned priorities above that of user processes but below that of interrupts. • Software interrupts are typically implemented in software. • Examples are callout queue processing and network packet processing. CSE522– Advanced Operating Systems

27. System Call Interface • Implemented as an assembly language stub. • Trap into kernel dispatch routine which may save additional state and invoke the high-level system call. • User privileges are verified and any data is copied into kernel (copyin()). • On return, copy out any user data (copyout()), check for an Asynchronous System Trap, or AST (signal, preemption, etc). CSE522– Advanced Operating Systems

28. Synchronization • Kernel is re-entrant. • Only one process may be active within the kernel at any given time (others are blocked). • Nonpreemptive. • Blocking operations • Masking interrupts CSE522– Advanced Operating Systems

29. Blocking Operations • When resource is unavailable (possibly locked), process sets flag and callssleep() • sleep places process on a blocked queue, sets state to asleep and calls swtch() • when resource released wakeup() is called • all processes sleeping on this resource are woken: state set to runnable and placed on the runnable queue(s). • When running, process must verify resource is available. • Why? CSE522– Advanced Operating Systems

30. Process Scheduling • Preemptive round-robin scheduling • fixed time quantum • priority is adjusted by nice value and usage factor. • the more cpu time, the lower the priority => prevents starvation • Processes sleeping in the kernel are assigned a kernel priority which is higher than user priorities. • priority boost may depend on the reason for sleep • may give boost to interactive, I/O bound processes • Kernel 0-49, user 50-127. CSE522– Advanced Operating Systems

31. Signals: Event notification • Asynchronous events and exceptions • Signal generation using the kill() system call • Default operation or user specific handlers • sets a bit in the pending signals mask in the proc structure • All signals are handled before return to normal processing • System calls can be restarted CSE522– Advanced Operating Systems

32. Process creation • Fork • create a new process • copy of parents virtual memory • parent return value is child’s PID • child return value is 0 • exec • overwrite with new program CSE522– Advanced Operating Systems

33. Process Termination • exit() is called • closes open files • releases other resources • saves resource usage statistics and exit status in proc structure • sends SIGCHLD signal to parent • wakes parent • any children are inherited by the init processes • calls swtch() • Process is in zombie state • parent collects, via wait(), exit status and usage CSE522– Advanced Operating Systems