Kernel Synchronization

1. Kernel Synchronization Examples From the Linux Kernel Michael E. Locasto

2. kernel control flow is a complicated, asynchronous interleaving Big Picture: How can the kernel correctly service requests?

3. Main Ideas / Concepts Atomic operations in x86 Kernel locking / synchronization primitives Kernel preemption Read-Copy-Update The “big kernel lock”

4. Kernel Preemption Kernel preemption is a concept in which the kernel can preempt other running kernel control paths (be they on behalf of a user or another kernel thread) Acquiring a spinlock automatically disables kernel preemption (as we will see in the code)

5. Synchronization Primitives Atomic operations Disable interrupts (cli/sti modify IF of eflags) Lock memory bus (x86 lock prefix) Spin locks Semaphores Sequence Locks Read-copy-update (RCU) (lock free)

6. Barriers Barriers are serializing operations; they “gather” and make operations sequential. Memory barrier: x86 in/out on I/O ports x86 lock prefix x86 writes to CReg, SReg/eflags, DReg

7. Barrier Implementation

8. Motivating Example: Using Semaphores in the Kernel what are: down_read, up_read, and mmap_sem

9. Let’s start with the data structure and see where that leads… start with the data structure: mm->mmap_sem

10. current->mm->mmap_sem structmm_struct: include/linux/mm_types.h

11. Primitive one: Atomic Type and Operations

12. On x86, these operations are atomic simple asm instructions that involve 0 or 1 aligned memory access read-modify-update in 1 clock cycle (e.g., inc, dec) anything prefixed by the IA-32 ‘lock’ prefix

13. atomic_t: include/linux/types.h

14. Example: Reference Counters Refcounts: atomic_t; associated with resources, but keeps count of kernel control paths accessing the resource

15. primitive two: SpinLocks

16. /include/linux/spinlock_types.h typedefstruct spinlock{ structraw_spinlockrlock; } spinlock_t; typedefstructraw_spinlock{ arch_spinlock_traw_lock; } raw_spinlock_t;

17. arch/x86/include/asm/spinlock_types.h#L10 slock=1 (unlocked), slock=0 (locked)

18. spinlock API (partial) /include/linux/spinlock.h /kernel/spinlock.c

19. include/linux/spinlock_api_smp.h

22. Linux Tracks Lock Dependencies @ Runtime

23. Here we mainly consider Read/Write Semaphores Primitive three: Semaphores

24. Important Caveats about Kernel Semaphores Semaphores are *not* like spinlocks in the sense that the invoking process is put to sleep rather than busy waits. As a result, kernel semaphores should only be used by functions that can safely sleep (i.e., not interrupt handlers)

26. might_sleep() leads (eventually) to:

29. rwsem_wake

30. __rwsem_do_wake On our way out, allow a writer at the front of the waiting queue to proceed. Then allow unlimited numbers of readers to access the critical region.

31. Advanced Techniques Sequence Locks Read-Copy-Update (RCU) Designed to protect data structures accessed by multiple CPUs; allows many readers and writers. Basic idea is simple (and in the name). Readers access data structure via a pointer; writers initially act as readers & create a copy to modify. “Writing” is just a matter of updating the pointer. A solution to the multiple readers-writer problem in that a writer is permitted to advance even if readers are in the critical section. Readers must check both an entry and exit flag to see if data has been modified underneath them.

32. RCU Only for kernel control paths; disables preemption. Used to protect data structures accessed through a pointer by adding a layer of indirection, we can reduce wholesale writes/updates to a single atomic write/update Heavy restrictions: RCU tasks cannot sleep readers do little work writers act as readers, make a copy, then update copy. Finally, they rewrite the pointer. cleanup is correspondingly complicated.

33. http://lxr.linux.no/#linux+v2.6.35.14/kernel/timer.c#L1354 RCU Example: Getppid(2)

36. Does synchronization impose a significant cost? (test at user level) Exercise: Time performance cost of synchronization

37. Code: Automatically Drawing Resource graphs