Servers and Concurrency Threads and Synchronization

1. Serversand Concurrency Threads and Synchronization Jeff Chase Duke University

2. Server structure in the real world • The server structure discussion motivates threads, and illustrates the need for concurrency management. • We return later to performance impacts and effective I/O overlap. • Unix systems define various less-than-clean mechanisms to deal with all these problems. • We all know about signals and their “issues”… • Separation of poll/select and accept in Unix syscall interface: multiple workers wake up when a socket has new data, but only one can accept the request. This is an example of a thundering herd problem: multiple workers wake up and contend for an arriving request: one worker wins and consumes the request, the others go back to sleep – their work was wasted. Requires API changes, and other recent fixes in Linux. • Real servers (e.g., Apache/MPM) on Unix incorporate lots of complexity to overcome these problems. We skip this topic.

3. Event-driven programming • Event-driven programming is a design pattern for a thread’s program. • The thread receives and handles a sequence of typed messages or events. • Handle one event at a time, in order. • In its pure form the thread never blocks, except to get the next event. • Blocks only if no events to handle (idle). • We can think of the program as a set of handler routines for the event types. • The thread upcalls the handler to dispatchor “handle” each event. • A handler should not block: if it does, the thread becomes unresponsive to events. events Dispatch events by invoking handlers (upcalls).

4. But what’s an “event”? • A system can use an event-driven design pattern to handle any kind of asynchronous event. • Arriving input (e.g., GUI clicks/swipes, requests to a server) • Notify that an operation started earlier is complete • E.g., I/O completion • Subscribe to events published/posted by other threads • Including status of children: stop/exit/wait, signals, etc. • You can use an “event” to represent any kind of message that drives any kind of action in the receiving thread. • In Android: intents, binder RPC, UI events • But the system must be designed for it, so that operations the thread requests do not block; the request returns immediately (“asynchronous”) and delivers a completion event later.

5. Events vs. threading • Classic Unix system call APIs are blocking. Requires multiple processes/threads to build responsive/efficient systems. • Even so, kernel networking and I/O stacks are mostly event-driven (interrupts, callbacks, event queues, etc.).Example: Windows I/O driver stack is a highly flexible event-driven system. • Some system call APIs may be non-blocking, i.e., asynchronous I/O. E.g., polling APIs like waitpid() with WNOHANG. This is an event-driven model: notify thread by an event when op completes. • Modern systems combine events and threading • Event-driven model is natural for GUIs, servers. • But to use multiple cores effectively, we need multiple threads. And every system today is a multicore system. • Multi-threading also enables use of blocking APIs without compromising responsiveness of other threads in the program.

6. Threads • We now enter the topic of threads and concurrency control. • This will be a focus for several lectures. • We start by introducing more detail on thread management, and the problem of nondeterminism in concurrent execution schedules. • Server structure discussion motivates threads, but there are other motivations. • Harnessing parallel computing power in the multicore era • Managing concurrent I/O streams • Organizing/structuring processing for user interface (UI) • Threading and concurrency management are fundamental to OS kernel implementation: processes/threads execute concurrently in the kernel address space for system calls and fault handling. The kernel is a multithreaded program. • So let’s get to it….

7. Threads • A thread is a stream of control…. • Executes a sequence of instructions. • Thread identity is defined by CPU register context (PC, SP, …, page table base registers, …) • Generally “context” is the register values and referenced memory state (k/u stacks, page tables). • Multiple threads can execute independently: • They can run in parallel on multiple cores... • physical concurrency • …or arbitrarily interleaved on some single core. • logical concurrency • A “thread” is also an OS abstraction to spawn and manage a concurrent stream of control.

8. Portrait of a thread In an implementation, each thread is represented by a data struct. We call it a “thread object” or “Thread Control Block”. It stores information about the thread, and may be linked into other system data structures. “Heuristic fencepost”: try to detect stack overflow errors Thread Control Block (“TCB”) Storage for context (register values) when thread is not running. name/status etc 0xdeadbeef Stack ucontext_t Each thread also has a runtime stack for its own use. As a running thread calls procedures in the code, frames are pushed on its stack.

9. Threads and the kernel process • Modern operating systems have multi-threaded processes. • A program starts with one main thread, but once running it may create more threads. • Threads may enter the kernel (e.g., syscall). • Threads are known to the kernel and have separate kernel stacks, so they can block independently in the kernel. • Kernel has syscalls to create threads (e.g., Linux clone). • Implementations vary. • This model applies to Linux, MacOS-X, Windows, Android, and pthreads or Java on those systems. VAS user mode user space data threads trap fault resume kernel mode kernel space

10. A thread This slide applies to the process abstraction too, or, more precisely, to the main thread of a process. active ready or running User TCB sleep wait wakeup signal wait blocked kernel TCB kernel stack user stack When a thread is blocked its TCB is placed on a sleep queue of threads waiting for a specific wakeup event. Program

11. Android: threading model • An app is launched as a process when any of its components is first instantiated. • The process main thread is event-driven, e.g., by User Interface (UI) events. • Also called the “UI thread”. • UI toolkit code is not thread-safe, so it should execute only on the UI thread. • UI thread should never block, or app becomes unresponsive. • UI thread also receives incoming intents, launches and tears down components, and delivers component lifecycle upcalls. • An app may spawn other background threads (workers) for other uses. • Binder RPC manages a thread pool. events

12. Adapted from http://developer.android.com/guide/components/processes-and-threads.html Summary: By default, all components of the same application run in the same process and thread (called the "main" thread). The main thread controls the UI, so it is also called the UI thread. If the UI thread blocks then the application stops responding to the user. You can create additional background threads for operations that block, e.g., I/O, to avoid doing those operations on the UI thread. The background threads can interact with the UI by posting messages/tasks/events to the UI thread. Details: When an application component starts and the application does not have any other components running, the Android system starts a new Linux process for the application with a single thread of execution called the main thread. All components that run in the same process are initialized by its main thread, and system upcalls to those components (onCreate, onBind, onStart,…) run on the main thread. The main thread is also called the UI thread. It is in charge of dispatching events to user interface widgets and interacting with elements of the Android UI toolkit. For instance, when the user touches a button on the screen, your app's UI thread dispatches the touch event to the widget, to set its pressed state and redraw itself. If you have operations that might require blocking, e.g., to perform I/O like network communication or database access, you should run them in separate threads. A thread that is not the UI thread is called a background thread or "worker" thread.

13. Adapted from http://developer.android.com/guide/components/processes-and-threads.html Your app should never block the UI thread. When the UI thread is blocked, no events can be dispatched, including drawing events. From the user's perspective, the application appears to “hang” or “freeze”. Even worse, if the app blocks the UI thread for more than a few seconds, Android presents the user with the infamous "application not responding" (ANR) dialog. The user might then decide to quit your application and uninstall it. In a correct Android program the UI thread blocks only to wait for the next event, when it has nothing else to do (it is idle). If you have an operation to perform that might block for any other reason, then you should arrange for a background/worker thread to do it. Additionally, the Android UI toolkit is not thread-safe: if multiple threads call a module that is not thread-safe, then the process might crash. A correct app manipulates the user interface only from a single thread, the UI thread. So: your app must not call UI widgets from a worker thread. So how can a worker thread interact with the UI, e.g., to post status updates? Android offers several ways for a worker to post operations to run on the UI thread. Note: this concept of a single event-driven main/UI thread appears in other systems too.

14. Android: Looper/MessageQueue • A thread may attach a Looper object, and invoke a method to loop on it. • Internally, a Looper has exactly one MessageQueue, and is attached to exactly one thread. • Events are posted as Messages on the attached MessageQueue. • The attached thread handles one Message at a time from the MessageQueue, in order. • This is Android’s mechanism to deliver UI eventsand Intents to the UI thread. events • It has other uses as well. E.g. in binder RPC (but incoming binder calls execute on a pool of worker threads, and not on the UI thread).

15. Android: Handler • A thread may create one or more Handler objects to attach to its Looper. • Handler object has post/send API that other threads may use to enqueue events on the Handler’s attached queue. • The owning thread retrieves events from its queue and handles them, e.g., by invoking a handleMessage method on the Handler that posted the message. • Note: if the handler (handleMessage) blocks, then the thread cannot respond to other messages on its queue. E.g., if the thread is the main/UI thread, then the UI may “freeze”. events

16. Android Handler: some details There are two main uses for a Handler: (1) to schedule messages and runnables to be executed as some point in the future; and (2) to enqueue an action to be performed on a different thread than your own. …[Generating an event:] post(Runnable), postAtTime(Runnable, long), postDelayed(Runnable, long), sendEmptyMessage(int), sendMessage(Message), sendMessageAtTime(Message, long), and sendMessageDelayed(Message, long) methods. The post versions allow you to enqueueRunnable objects to be [run] when they are received; the sendMessage versions allow you to enqueue a Message object containing a bundle of data…to be processed by the Handler's handleMessage(Message) method ([in your implemented subclass of Handler])…. When a process is created for your application, its main thread is dedicated to running a message queue that takes care of managing the top-level application objects (activities, broadcast receivers, etc) and any windows they create. You can create your own threads, and communicate back with the main application thread through a Handler. This is done by calling the same post or sendMessage methods as before, but from your new thread…. [http://developer.android.com/reference/android/os/Handler.html]

17. Threads and RPC Q: How do we manage these “call threads”? A: Create them as needed, and keep idle threads in a thread pool. When an RPC call arrives, wake up an idle thread from the pool to handle it. On the client, the client thread blocks until the server thread returns a response. [OpenGroup, late 1980s]

18. Binder: thread pool details “The system maintains a pool of transaction threads in each process that it runs in. These threads are used to dispatch all IPCs coming in from other processes. For example, when an IPC is made from process A to process B, the calling thread in A blocks in transact() as it sends the transaction to process B. The next available pool thread in B receives the incoming transaction, calls Binder.onTransact() on the target object, and replies with the result Parcel. Upon receiving its result, the thread in process A returns to allow its execution to continue. …” [http://developer.android.com/reference/android/os/IBinder.html] Note: in this setting, a “transaction” is just an RPC request/response exchange.

19. Thread pool: idealized worker loop Magic elastic worker pool Resize worker pool to match incoming request load: create/destroy workers as needed. Handle one request, blocking as necessary. dispatch Incoming request queue When request is complete, return to worker pool. idle workers Workers wait here for next request dispatch. Workers could be processes or threads.

20. Ideal event poll API for thread pooling Poll() • Delivers: returns exactly one event (message or notification), in its entirety, ready for service (dispatch). • Idles: Blocks iffthere is no event ready for dispatch. • Consumes: returns each posted event at most once. • Combines: any of many kinds of events (a poll set) may be returned through a single call to poll. • Synchronizes: may be shared by multiple processes or threads ( handlers must be thread-safe as well).

21. Should you use a service or a thread? A service is simply a component that can run in the background even when the user is not interacting with your application. Thus, you should create a service only if that is what you need. If you need to perform work outside your main thread, but only while the user is interacting with your application, then you should probably instead create a new thread and not a service. For example, if you want to play some music, but only while your activity is running, you might create a thread in onCreate(), start running it in onStart(), then stop it in onStop(). Remember that if you do use a service, it still runs in your application's main thread by default, so you should still create a new thread within the service if it performs intensive or blocking operations. Also consider using AsyncTask or HandlerThread, instead of the traditional Threadclass. http://developer.android.com/guide/components/services.html

22. Android: AsyncTask [http://techtej.blogspot.com/2011/03/android-thread-constructs-part-3.html]

23. Thread APIs (rough/pseudocode) • Every thread system has an API for applications to use to manage their threads. Examples: Pthreads, Java threads, C#, Taos… • Create and start (“spawn” or “generate”) a thread. • Wait for a thread to complete (exit), and obtain a return value. • Break into the execution of a thread (optional). • Save and access data specific to each thread. • References to thread objects may be passed around in the API. Thread operations (parent) a rough sketch: t = create(); t.start(proc, argv); t.alert(); (optional) result = t.join(); Self operations (child) a rough sketch: exit(result); t = self(); setdata(ptr); ptr = selfdata(); alertwait(); (optional) Details vary.

24. Pthread (posix thread) example int main(intargc, char *argv[]) { if (argc != 2) { fprintf(stderr, "usage: threads <loops>\n"); exit(1); } loops = atoi(argv[1]); pthread_t p1, p2; printf("Initial value : %d\n", counter); pthread_create(&p1, NULL, worker, NULL); pthread_create(&p2, NULL, worker, NULL); pthread_join(p1, NULL); pthread_join(p2, NULL); printf("Final value : %d\n", counter); return 0; } volatile int counter = 0; int loops; void *worker(void *arg) { inti; for (i = 0; i < loops; i++) { counter++; } pthread_exit(NULL); } data [pthread code from OSTEP]

25. Java threads: the basics class RunnableTaskimplements Runnable { public RunnableTask(…) { // save any arguments or input for the task (optional) } public void run() { // do task: your code here } } … RunnableTask task = new RunnableTask(); Thread t1= new Thread(task, "thread1"); t1.start(); …

26. Java threads: the basics If you prefer, you may extend the Java Thread class. public void MyThread extends Thread { public void run() { // do task: your code here } } … Thread t1 = new MyThread(); t1.start();

27. A note on Java threads and Runnable. Java provides two alternatives to achieve the same end: starting a thread that runs a method that you designate. Either approach allows you to run code of your choice on your threads, and give each thread access to any data in your program that you want. Both ways to create and start a thread involve Runnable. Runnable is just an interface in Java. When you write a Java class, you may choose to have it implement the Runnable interface. Then every instance of your class is a Runnable object. The important part of the Runnable interface is a method called run(). The body of run() implements whatever you (the programmer) want the Runnable object to do when it runs. Simple, yes? Runnables are useful in various ways.  For example, in Android, you can post Runnables to another thread’s MessageQueuevia a Handler. The thread runs them as it receives them from its queue. In essence, Runnable allows you to send a message or event with code in it: the message tells the receiver what code to run. Thread is a system class that implements Runnable. Thus a Thread is a Runnable object. But not every Runnable object is a Thread. When a Java thread starts, it runs a designated Runnable object.  If that Runnable object is the Thread object itself, then the Thread object should be an instance of a subclass of Thread that overrides the method run() with some application-specific logic. If some other Runnable object is used, then its run() method should have the subprogram for the thread to run.

28. Example:Jabber class Jabber implements Runnable { String str; public Jabber(String s) { str = s; } public void run() { while (true) { System.out.print(str); System.out.println(); } } } class JabberTest{ public static void main(String[] args) { Jabber j = new Jabber("1"); Jabber k = new Jabber("2"); Thread t = new Thread(j); Thread u = new Thread(k); t.start(); u.start(); } } This came from http://www.cs.usfca.edu/~parrt/course/601/lectures/threads.html, which is a nice little page on Java threads.

29. Thread states and transitions revisited If a thread is in the ready state thread, then the system may choose to run it “at any time”. The kernel can switch threads whenever it gains control on a core, e.g., by a timer interrupt. If the current thread takes a fault or system call trap, and blocks or exits, then the scheduler switches to another thread. But it could also preempt a running thread. From the point of view of the program, dispatch and preemption are nondeterministic: we can’t know the schedule in advance. STOP wait running These preempt and dispatch transitions are controlled by the kernel scheduler. Sleep and wakeup transitions are initiated by calls to internal sleep/wakeup APIs by a running thread. yield preempt sleep dispatch blocked ready wakeup

30. Two threads sharing a CPU concept reality context switch

31. CPU Scheduling 101 The OS scheduler makes a sequence of “moves”. • Next move: if a CPU core is idle, pick a ready threadfrom the ready pool and dispatch it (run it). • Scheduler’s choice is “nondeterministic” • Scheduler and machine determine the interleaving of execution (a schedule). blocked threads If timer expires, or wait/yield/terminate ready pool Wakeup GetNextToRun SWITCH()

32. Non-determinism and ordering Thread A Thread B Thread C Global ordering Time Why do we care about the global ordering? Might have dependencies between events Different orderings can produce different results Why is this ordering unpredictable? Can’t predict how fast processors will run

33. Non-determinism example • y=10; • Thread A: x = y+1; • Thread B: y = y*2; • Possible results? • A goes first: x = 11 and y = 20 • B goes first: y = 20 and x = 21 • Variable y is shared between threads.

34. Another example • Two threads (A and B) • A tries to increment i • B tries to decrement i i= 0; Thread A: while (i < 10){ i++; } print “A done.” Thread B: while (i > -10){ i--; } print “B done.”

35. Example continued • Who wins? • Does someone have to win? Thread A: i = 0; while (i < 10){ i++; } print “A done.” Thread B: i = 0; while (i > -10){ i--; } print “B done.”

36. Pthread (posix thread) example int main(intargc, char *argv[]) { if (argc != 2) { fprintf(stderr, "usage: threads <loops>\n"); exit(1); } loops = atoi(argv[1]); pthread_t p1, p2; printf("Initial value : %d\n", counter); pthread_create(&p1, NULL, worker, NULL); pthread_create(&p2, NULL, worker, NULL); pthread_join(p1, NULL); pthread_join(p2, NULL); printf("Final value : %d\n", counter); return 0; } volatile int counter = 0; int loops; void *worker(void *arg) { inti; for (i = 0; i < loops; i++) { counter++; } pthread_exit(NULL); } data [pthread code from OSTEP]

37. Resource Trajectory Graphs Resource trajectory graphs (RTG) depict the “random walk” through the space of possible program states. Sm Sn So • RTG is useful to depict all possible executions of multiple threads. I draw them for only two threads because slides are two-dimensional. • RTG for N threads is N-dimensional. • Thread i advances along axis i. • Each point represents one state in the set of all possible system states. • Cross-product of the possible states of all threads in the system

38. Resource Trajectory Graphs This RTG depicts a schedule within the space of possible schedules for a simple program of two threads sharing one core. Every schedule ends here. Blue advances along the y-axis. EXIT The diagonal is an idealized parallel execution (two cores). Purple advances along the x-axis. The scheduler chooses the path (schedule, event order, or interleaving). context switch From the point of view of the program, the chosen path is nondeterministic. EXIT Every schedule starts here.

39. A race This is a valid schedule. But the schedule interleaves the executions of “x = x+ 1” in the two threads. The variable x is shared (like the counter in the pthreads example). This schedule can corrupt the value of the shared variable x, causing the program to execute incorrectly. This is an example of a race: the behavior of the program depends on the schedule, and some schedules yield incorrect results. x=x+1 x=x+1 start

40. Reading Between the Lines of C load x, R2 ; load global variable x add R2, 1, R2 ; increment: x = x + 1 store R2, x ; store global variable x Two threads execute this code section. x is a shared variable. load add store load add store Two executions of this code, so: x is incremented by two. ✔

41. Interleaving matters load x, R2 ; load global variable x add R2, 1, R2 ; increment: x = x + 1 store R2, x ; store global variable x Two threads execute this code section. x is a shared variable. load add store load add store X In this schedule, x is incremented only once: last writer wins. The program breaks under this schedule. This bug is a race.

42. Concurrency control • Each thread executes a sequence of instructions, but their sequences may be arbitrarily interleaved. • E.g., from the point of view of loads/stores on memory. • Each possible execution order is a schedule. • A thread-safe program must exclude schedules that lead to incorrect behavior. • It is called synchronization or concurrency control. • The scheduler (and the machine) select the execution order of threads

43. This is not a game • But we can think of it as a game. • You write your program. • The game begins when you submit your program to your adversary: the scheduler. • The scheduler chooses all the moves while you watch. • Your program may constrain the set of legal moves. • The scheduler searches for a legal schedule that breaks your program. • If it succeeds, then you lose (your program has a race). • You win by not losing.  x=x+1 x=x+1

44. The need for mutual exclusion  The program may fail if the schedule enters the grey box (i.e., if two threads execute the critical section concurrently). The two threads must not both operate on the shared global x “at the same time”. x=??? x=x+1 x=x+1

45. A Lock or Mutex Locks are the basic tools to enforce mutual exclusion in conflicting critical sections. • A lock is a special data item in memory. • API methods: Acquire and Release. • Also called Lock() and Unlock(). • Threads pair calls to Acquire and Release. • Acquire upon entering a critical section. • Release upon leaving a critical section. • Between Acquire/Release, the thread holds the lock. • Acquire does not pass until any previous holder releases. • Waiting locks can spin (a spinlock) or block (a mutex). A A R R

46. Definition of a lock (mutex) • Acquire + release ops on L are strictly paired. • After acquire completes, the caller holds (owns) the lock L until the matching release. • Acquire + release pairs on each L are ordered. • Total order: each lock L has at most one holder at any given time. • That property is mutual exclusion; L is a mutex.

47. OSTEP pthread example (2) load add store load add store pthread_mutex_t m; volatile int counter = 0; int loops; void *worker(void *arg) { inti; for (i = 0; i < loops; i++) { Pthread_mutex_lock(&m); counter++; Pthread_mutex_unlock(&m); } pthread_exit(NULL); } A A “Lock it down.” R R 

48. Portrait of a Lock in Motion The program may fail if it enters the grey box. A lock (mutex) prevents the schedule from ever entering the grey box, ever: both threads would have to hold the same lock at the same time, and locks don’t allow that.  R x=??? x=x+1 A R A x=x+1

49. Handing off a lock serialized (one after the other) First I go. release acquire Then you go. Handoff The nth release, followed by the (n+1)th acquire

50. Mutual exclusion in Java • Mutexes are built in to every Java object. • no separate classes • Every Java object is/has a monitor. • At most one thread may “own” a monitor at any given time. • A thread becomes owner of an object’s monitor by • executing an object method declared as synchronized • executing a block that is synchronized on the object public void increment() { synchronized(this) { x = x + 1; } } public synchronized void increment() { x = x + 1; }