CPS 310 Processes and Unix Processes

1. CPS 310 Processes and Unix Processes Jeff Chase Duke University http://www.cs.duke.edu/~chase/cps310

2. The story so far: process and kernel • A (classical) OS lets us run programsas processes. A process is a running program instance (with a thread). • Program code runs with the CPU core in untrusted user mode. • Processes are protected/isolated. • Virtual address space is a “fenced pasture” • Sandbox: can’t get out. Lockbox: nobody else can get in. • The OS kernel controls everything. • Kernel code runs with the core in trusted kernel mode.

3. Processes and the kernel Each process has a private virtual address space and one or more threads. Programs run as independent processes. data data Protected system calls, and faults, ...and upcalls (e.g., signals) Protected OS kernel mediates access to shared resources. Threads enter the kernel for OS services. The kernel code and data are protected from untrusted processes.

4. Upcall example: Unix signals • Signals are asynchronous notifications to a user process that some event of interest to it has occurred. • A process may register signal handlersfor various events relating to the process. The signal handlers are procedures in user space. • To deliver a signal, the kernel redirects a user thread to execute a selected registered signal handler in user mode. • Unix signals take a default action if no handler is registered. • E.g., segmentation fault  die. Other actions: ignore, stop data data ...and upcalls (e.g., signals) Protected system calls

5. Processes and their threads stack main thread virtual address space other threads (optional) +… + STOP Each process has a virtual address space (VAS): a private name space for the virtual memory it uses. The VAS is both a “sandbox” and a “lockbox”: it limits what the process can see/do, and protects its data from others. wait Each process has a main thread bound to the VAS, with a stack. If we say a process does something, we really mean its thread does it. The kernel can suspend/restart a thread wherever and whenever it wants. On real systems, a process can have multiple threads. We presume that they can all make system calls and block independently.

6. The theater analogy virtual memory (stage) script Threads Program Address space Running a program is like performing a play. [lpcox]

7. The sheep analogy Address space “fenced pasture” Code and data Thread

8. The core-and-driveranalogy The machine has a bank of CPU cores for threads to run on. The OS allocates cores to threads (the “drivers”). Cores are hardware. They go where the driver tells them. OS can force a switch of drivers any time. Core #2 Core #1

9. Threads drive cores

10. What? • A process is a running program. • A running program (a process) has at least one thread (“main”), but it may (optionally) create other threads. • The threads execute the program (“perform the script”). • The threads execute on the “stage” of the process virtual memory, with access to a private instance of the program’s code and data. • A thread can access any virtual memory in its process, but is contained by the “fence” of the process virtual address space. • Threads run on cores: a thread’s core executes instructions for it. • Sometimes threads idle to wait for a free core, or for some event. Sometimes cores idle to wait for a ready thread to run. • The operating system kernel shares/multiplexes the computer’s memory and cores among the virtual memories and threads.

11. Process management • OS offers system call APIs for managing processes. • Create processes (children) • Control processes • Monitor process execution • “Join”: wait for a process to exit and return a result • “Kill”: send a signal to a process • Establish interprocess communication (IPC: later) • Launch a program within a process • We study the Unix process abstraction as an example. • Illustrative and widely used for 40+ years! • Use it to build your own shell.

12. Unix: A lasting achievement? “Perhaps the most important achievement of Unix is to demonstrate that a powerful operating system for interactive use need not be expensive…it can run on hardware costing as little as $40,000.” DEC PDP-11/24 • The UNIX Time-Sharing System* • D. M. Ritchie and K. Thompson • 1974 http://histoire.info.online.fr/pdp11.html

13. Unix shell(s)An early interactive programming environment • Classical Unix programs are packaged software components: specific functions with narrow, text-oriented interfaces. • Shell is a powerful (but character-based) user interface. • Shell is a user program that uses kernel syscalls to execute utility programs as child processes. • A typical shell command is just the name of a program to run. • Shell is also a programming environment for composing and coordinating programs interactively. • So: it’s both an interactive programming environment and a programmable user interface. • Both ideas were “new” at the time (late 1960s). • Its powerful scripting capabilities are widely used in large-scale system administration and services, for 40+ years!

14. The essence of Unix process “fork” fork Oh Ghost of Walt, please don’t sue me.

15. Unix fork/exec/exit/wait syscalls • intpid = fork(); • Create a new process that is a clone of its parent, running the same program. • exec*(“program”[argvp, envp]); • Overlay the calling process with a new program, and transfer control to it, passing arguments and environment. • exit(status); • Exit with status, destroying the process. • intpid = wait*(&status); • Wait for exit (or other status change) of a child, and “reap” its exit status. • Recommended: use waitpid(). fork child fork parent parent program initializes child context exec time wait exit

16. The Shell • Users may select from a range of command interpreter (“shell”) programs available. (Or even write their own!) • csh, sh, ksh, tcsh, bash: choose your flavor… • Shells execute commands composed of program filenames, args, and I/O redirection symbols. • Shell uses fork, exec, wait, etc., etc. • Can coordinate multiple child processes that run together as a process group or job. • Shells can run files of commands (scripts) for more complex tasks, e.g., by redirecting I/O channels (descriptors). • Shellbehavior is guided by environment variables, e.g., $PATH • Parent may control/monitor all aspects of child execution.

17. Postnote • The following slides were used in recitation on 1/27. • They fill in details on Unix fork/exec/exit/wait, variations and how they evolved, usage, and the power of combining these simple concepts. • They also illustrate fundamental threading concepts: thread create/exit/join, physical and logical concurrency, blocking/sleeping and wakeup, and parallelism. • You should be sure to understand all the concepts, but the details of the system calls and C/Unix programming model are for illustration only. • All of the C examples are available on the course web.

18. Unix fork/exit syscalls • intpid = fork(); • Create a new process that is a clone of its parent. Return child process ID (pid) to parent, return 0 to child. • exit(status); • Exit with status, destroying the process. • Status is returned to the parent. • Note: this is not the only way for a process to exit! parent fork parent child time data data exit exit p pid: 5587 pid: 5588

19. exit syscall (original concept)

20. fork The forksyscall returns twice: It returns a zero in the context of the new child process. It returns the new child process ID (pid) in the context of the parent. intpid; int status = 0; if (pid = fork()) { /* parent */ ….. } else { /* child */ ….. exit(status); }

21. A simple program: sixforks … int main(intargc, char* argv) { fork(); fork(); fork(); fork(); fork(); fork(); printf("Process %d exiting.\n", getpid()); } How many processes are created by these six forks? chase$ cc –o sixforkssixforks.c chase$ ./sixforks ??? chase$ getpidsyscall: Get processID of current process.

22. A simple program: sixforks Process 15220 exiting. Process 15209 exiting. Process 15232 exiting. Process 15219 exiting. Process 15233 exiting. Process 15223 exiting. Process 15210 exiting. Process 15234 exiting. Process 15228 exiting. Process 15192 exiting. Process 15230 exiting. Process 15211 exiting. Process 15227 exiting. Process 15239 exiting. Process 15231 exiting. Process 15242 exiting. Process 15243 exiting. Process 15240 exiting. Process 15236 exiting. Process 15241 exiting. Process 15244 exiting. Process 15247 exiting. Process 15235 exiting. Process 15245 exiting. Process 15250 exiting. Process 15248 exiting. Process 15249 exiting. Process 15204 exiting. Process 15238 exiting. Process 15251 exiting. Process 15237 exiting. Process 15252 exiting. Process 15253 exiting. Process 15246 exiting. Process 15254 exiting. chase$ cc –o sixforkssixforks.c chase$ ./sixforks Process 15191 exiting. Process 15200 exiting. Process 15195 exiting. Process 15194 exiting. Process 15197 exiting. Process 15202 exiting. Process 15193 exiting. Process 15198 exiting. Process 15215 exiting. Process 15217 exiting. Process 15218 exiting. Process 15203 exiting. Chase$ Process 15212 exiting. Process 15196 exiting. Process 15222 exiting. Process 15213 exiting. Process 15221 exiting. Process 15224 exiting. Process 15206 exiting. Process 15216 exiting. Process 15205 exiting. Process 15207 exiting. Process 15201 exiting. Process 15214 exiting. Process 15225 exiting. Process 15199 exiting. Process 15226 exiting. Process 15208 exiting. Process 15229 exiting. … int main(intargc, char* argv) { fork(); fork(); fork(); fork(); fork(); fork(); printf("Process %d exiting.\n", getpid()); }

23. sixforks: some questions • What if I want to create six children, but I don’t want my children to have children of their own? • What if I want the program to print the total number of processes created? How? (Other than by having the program do the math.) • How much memory does this program use? How many pages? • How does this test system assign process IDs? • Why do the process IDs print out of order?

24. fork (original concept)

25. fork in action today Fork is conceptually difficult but syntactically clean and simple. I don’t have to say anything about what the new child process “looks like”: it is an exact clone of the parent! The child has a new thread executing at the same point in the same program. The child is a new instance of the running program: it has a “copy” of the entire address space. The “only” change is the process ID and return code cpid! The parent thread continues on its way. The child thread continues on its way. void dofork() { intcpid = fork(); if (cpid < 0) { perror("fork failed: "); exit(1); } else if (cpid == 0) { child(); } else { parent(cpid); } }

26. A simple program: forkdeep int count = 0; int level = 0; void child() { level++; output pids if (level < count) dofork(); if (level == count) sleep(3); /* pause 3 secs */ } void parent(intchildpid) { output pids wait for child to finish } main(intargc, char *argv[]) { count = atoi(argv[1]); dofork(); output pid } We’ll see later where arguments come from. level==1 level==2

27. chase$ ./forkdeep 4 30866-> 30867 30867 30867-> 30868 30868 30868-> 30869 30869 30869-> 30870 30870 30870 30869 30868 30867 30866 chase$ chase$ ./forkdeep 3 11496-> 11498 11498 11498-> 11499 11499 11499-> 11500 11500 11500 11499 11498 11496 chase$

28. wait Process states (i.e., states of the main thread of the process) “sleep” “wakeup” Note: in modern systems the waitsyscall has many variants and options.

29. wait today • Parent uses wait to sleep until the child exits; wait returns child pid and status. • Wait variants allow wait on a specific child, or notification of stops and other “signals”. • Recommended: use waitpid(). int pid; int status = 0; if (pid = fork()) { /* parent */ ….. pid = wait(&status); } else { /* child */ ….. exit(status); }

30. But how do I run a new program in my child process? • The child, or any process really, can replace its program in midstream. • exec* system call: “forget everything in my address space and reinitialize my entire address space with stuff from a named program file.” • The exec system call never returns: the new program executes in the calling process until it dies (exits). • The code from the parent program runs in the child process and controls its future. The parent program selects the child program that the child process will run (via exec), and sets up its connections to the outside world. The child program doesn’t even know how those connection are set up! • But don’t forget to check error status from exec*! It returns an error to parent program if it fails.

31. Running a program data code (“text”) constants initialized data Process sections segments Thread Unix: fork/exec Program virtual memory When a program launches, the OS creates a process to run it, with a main thread to execute the code, and a virtual memory to store the running program’s code and data.

32. exec (original concept)

33. A simple program: forkexec … main(intargc, char *argv[]) { int status; intrc = fork(); if (rc < 0) { perror("fork failed: "); exit(1); } else if (rc == 0) { printf("Iam a child: %d.\n", getpid()); argv++; execve(argv[0], argv, 0); /* NOTREACHED */ } else { waitpid(rc, &status, 0); printf(“Child %d exited with status %d\n.", rc, WEXITSTATUS(status)); } } Always check return from syscallsand show any errors! Parent program running in child process A successful exec* never returns to calling program. Reap exitstatus returnvalue from child via exit/wait.

34. A simple program: prog0 … int main() { printf("Hi from %d!\n", getpid()); exit(72); } exitsyscall: Pass exitstatus returnvalue to parent via exit/wait. chase$ cc –o forkexecforkexec.c chase$ cc –o prog0 prog0.c chase$ ./forkexec prog0 I am a child: 11384. Hi from 11384! Child 11384 exited with status 72. chase$ getpidsyscall: Get processID of current process.

35. Exec setup (ABI) The details aren’t important. The point is: The exec system call sets up the VAS of the calling process to execute a named program. Exec passes two arrays of strings to the new program’s main(): an array of arguments and an array of named environment variables. It stages the argv/env arrays in the VAS before returning to user mode to start execution at main(). System V Application Binary Interface AMD64 Architecture Processor Supplement

36. Simple I/O: args and printf chase$ cc –o prog1 prog1.c chase$ ./forkexecprog1 arguments: 1 0: prog1 child 19178 exited with status 0 chase$ ./forkexec prog1 one 2 3 arguments: 4 0: prog1 1: one 2: 2 3: 3 Child 19181 exited with status 0. #include <stdio.h> int main(intargc, char* argv[]) { inti; printf("arguments: %d\n", argc); for (i=0; i<argc; i++) { printf("%d: %s\n", i, argv[i]); } }

37. Environment variables and property lists • The environment variable array is a property list. • The property list construct is very common and useful! • Also commonly used for configuration files. • It goes by various names: Java plist, Windows Registry, INI files • Each element of the list is a string: “NAME=VALUE”. • The standard library has primitives to look up the VALUE corresponding to a NAME. • In Unix systems: standard environment variables are handed down through the shell: they give programs lots of information about the environment. • The parent specifies them to the exec* syscall.

38. Environment variables (rough) #include <stdio.h> #include <stdlib.h> int main(intargc, char* argv[], char* envp[]) { inti; int count = atoi(argv[1]); for (i=0; i < count; i++) { printf("env %d: %s\n", i, envp[i]); } }

39. Environment variables (rough) chase$ cc –o env0 env0.c chase$ ./env0 Segmentation fault: 11 chase$ ./env0 12 env 0: TERM_PROGRAM=Apple_Terminal env 1: TERM=xterm-256color env 2: SHELL=/bin/bash env 3: TMPDIR=/var/folders/td/ng76cpqn4zl1wrs57hldf1vm0000gn/T/ env 4: Apple_PubSub_Socket_Render=/tmp/launch-OtU5Bb/Render env 5: TERM_PROGRAM_VERSION=309 env 6: OLDPWD=/Users/chase/c210-stuff env 7: TERM_SESSION_ID=FFCE3A14-1D4B-4B08… env 8: USER=chase env 9: COMMAND_MODE=unix2003 env 10: SSH_AUTH_SOCK=/tmp/launch-W03wn2/Listeners env 11: __CF_USER_TEXT_ENCODING=0x1F5:0:0 chase$

40. Environment variables (safer) #include <stdio.h> #include <stdlib.h> int main(intargc, char* argv[], char* envp[]) { inti; int count; if (argc < 2) { fprintf(stderr, "Usage: %s <count>\n", argv[0]); exit(1); } count = atoi(argv[1]); for (i=0; i < count; i++) { if (envp == 0) { printf("env %d: nothing!\n", i); exit(1); } else if (envp[i] == 0) { printf("env %d: null!\n", i); exit(1); } else printf("env %d: %s\n", i, envp[i]); } }

41. Where do environment variables come from? chase$ cc –o envenv.c chase$ ./env chase$ ./forkexecenv Usage: env <count> child 19195 exited with status 1 chase$ ./forkexecenv 1 env 0: null! child 19263 exited with status 1 chase$

42. forkexec revisited chase$ cc –o felforkexec-lala.c chase$ ./felenv 1 env 0: lalala child 19276 exited with status 0 chase$ char *lala = "lalala\n"; char *nothing = 0; … main(intargc, char *argv[]) { int status; intrc = fork(); if (rc < 0) { … } else if (rc == 0) { argv++; execve(argv[0], argv, &lala); } else { … }

43. forkexec revisited again … main(intargc, char *argv[], char *envp[]) { int status; intrc = fork(); if (rc < 0) { … } else if (rc == 0) { argv++; execve(argv[0], argv, envp); } else { … } chase$ cc –o fe forkexec1.c chase$ ./feenv 3 env 0: TERM_PROGRAM=Apple_Terminal env 1: TERM=xterm-256color env 2: SHELL=/bin/bash child 19290 exited with status 0 chase$

44. How about this? chase$ ./fefefefefefefefefefefefeenv3 <???>

45. How about this? chase$ ./fefefefefefefefefefefefeenv 3 env 0: TERM_PROGRAM=Apple_Terminal env 1: TERM=xterm-256color env 2: SHELL=/bin/bash child 19303 exited with status 0 child 19302 exited with status 0 child 19301 exited with status 0 child 19300 exited with status 0 child 19299 exited with status 0 child 19298 exited with status 0 child 19297 exited with status 0 child 19296 exited with status 0 child 19295 exited with status 0 child 19294 exited with status 0 child 19293 exited with status 0 child 19292 exited with status 0 chase$ It is easy for children to inherit environment variables from their parents. Exec* enables the parent to control the environment variables and arguments passed to the children. The child process passes the environment variables “to itself” but the parent program controls it.