Compositional Verification of Interrupt and Thread Management in Realistic OS Kernels

1. Compositional Verification of Interrupt and Thread Management in Realistic OS Kernels Zhong Shao Yale University October 2012 Joint work with Yu Guo and Xinyu Feng at USTC

2. Review: OS kernel verification All of them are verified as if the kernel is sequential code How to verify kernels that are reentrant, concurrent, and run on multicore machines? AIM (Feng et.al., PLDI’08) seL4 (Klein et.al., SOSP’09) Verve Nucleus (Yang & Hawblitzel, PLDI’10)

3. Review: concurrent thread management User World #1 #2 #3 rq Concurrent Kernel World #2 #1 #3 Sequential Kernel World cswitch cswitch

4. Review: our work on concurrent thread management • Developed a new program logic for modular verification of concurrent kernelcode that operates on shared thread meta-data • Proved the soundness of the logic in Coq (14,000 loc) • Applied the logic to verify a variety of concurrent thread management routines, including all 3 patterns of schedulers • New paper to appear in APLAS 2012, Kyoto, Japan. • Next step: how to support interrupt management?

5. What is interrupt? CPU timeline Kernel thread Run a kernel thread External Event CPU Receive an interrupt signal and then begin ISR ISR Resume the kernel thread when ISR is finished

6. Verifying naive interrupt support(previous work) • By ownership (of memory) transfer • [Feng et.al., PLDI’08] • [Gotsman and Yang, ICFP’11] Kernel thread Thread Mem. ISR Mem. I ISR X Thread Mem. X I ISR Mem.

7. However … • In realistic OS kernels kernel threads (scheduling, interrupt management, thread management) interruptible part Boundary? code region with interrupt disabled (e.g. thread context switch, change critical kernel var.) Non-interruptible part

8. Realistic interrupt management • Many complicated features • Kernel preemption • Interrupt nesting • Interrupt thread, bottom halves, or deferred work • Many complicated concurrent interaction • ISR vs. ISR • ISR vs. kernel thread • kernel thread vs. kernel thread • code on different processors • deferred work vs. ISR • ...

9. Challenges • It is NOT easy to use a two-layer solution • Not clear where is the "boundary" • The non-interruptible code is difficult to abstract over • Some kernel variables are critical to kernel behaviors • Not easy to reason about them in a concurrent setting • Various kinds of concurrency

10. Our approach • One-layer solution • No fixed boundary btw interruptible and non-interruptible code • Fine-grained partition of shared resources • Verify concurrency via ownership transfer • Let abstract thread resources also carry abstract kernel variables • Provide enough information to reason about critical kernel variables • We can derive different program logics

11. What are thread resources • Thread resources specify thread meta data in the system • Can be thread local, shared (transferable) Kernel thread Kernel thread A B C D A B C D MA TA TB TC TD MA memory resources memory resources + thread resources MA TA TB TD MA C is Dead

12. What are abstract kernel variables? • Some kernel variables are critical to the kernel, e.g., • the variable recording the interrupt nesting level • the variable to switch on/off kernel preemption • the variable to indicate if the kernel is processing deferred work • ... • The code manipulating them needs to be verified. • Their values affect the programming logics • e.g., the ISR without nesting and the nested ISR need different programming rules

13. What are abstract kernel variables? • To verify kernel in a single layer • Let thread resources carry the abstract values of these critical kernel variables • The abstract kernel variables can be local • corresponding to per-thread kernel variables • Or, shared • corresponding to global (or, per-CPU) kernel variables A B C D Kernel thread MA TA TB TC TD K Abstract kernel variables

14. The role of abstract kernel vars. • Modification of kernel variables is guarded by • The program logic is designed according to K K A B C D Kernel thread MA TA TB TC TD K Abstract kernel variables

15. Feature (1): kernel preemption CPU timeline Kernel thread Run one of multiple kernel threads External Event CPU Receive an interrupt signal and then begin ISR ISR Learn that some thread should preempt to run Turn to the preempting thread when ISR is finished and if the preemption is enabled

16. Verifying kernel preemption • Partition the resources • Including memory resources and thread resources • Let K specify the kernel variable switching on/off the kernel preemption K IA IB K X I Shared resources Private resources of a kernel thread Resources used in ISR Kernel variable controlling the kernel preemption

17. X I K Ownership transfer (Kernel Preemption Enabled) Kernel threads A B C D Shared Resource IA K ISR Kernel Variables Context switch X I K IB K Shared Resource

18. Ownership transfer (Kernel Preemption Disabled) Kernel threads A B C D X K I IA K ISR Kernel Variables Shared Resource X K I IA K IB K

19. Feature (2): nested interrupts CPU timeline Kernel thread External Event Run one of multiple kernel threads CPU Receive an interrupt signal and then begin ISR External Event 2 ISR Receive another signal and reenter the ISR ISR Return to the interrupted ISR Turn to the preempting thread when ISR is finished

20. Verifying nested interrupts • Partition the shared resources • Let K specify the nesting level K Shared resources used in non-nested ISR I J IA IB K X Shared resources used in any ISR Private resources of a kernel thread Resources used in ISR Kernel variable recording the nesting level

21. Ownership transfer Shared Res. Shared Res. in Non-nested ISR Kernel threads X I J K Increase K Enable Intr IA K X K' X I K' ISR Increase K X I K'' ISR X I K'' Decrease K X I K' IA K X K' Disable Intr Decrease K X I J K

22. Feature (3): interrupt thread CPU timeline Kernel threads External Event Interrupt thread Run one of multiple kernel threads CPU Receive an interrupt signal and then begin ISR ISR Wakeup the interrupt thread to handle the event Top-half Turn to the interrupt thread when ISR is finished Bottom-half (deferred work)

23. Verifying interrupt thread • Partition the shared resources • Let K specify if the kernel is processing interrupt K Shared resources between ISR and intr. thread Private resources of the intr. thread K X I U IB IE IA Kernel variable indicating if in the context of interrupt processing Shared resources between ISR and kernel threads Private resources of a kernel thread Resources used in ISR

24. Ownership transfer Kernel threads Interrupt thread A B C D E X I U K' IA K ISR wakeup Intr thrd. Context switch X I U K' IE U K''

25. Contributions • Partition the shared resources to deal with realistic kernel features • Introduce abstract thread resources marked with interrupt information • Extend concurrent separation logic to reason about kernel code with realistic interrupt management

26. Progress • What we could verify so far • Interrupt bottom halves in Linux 3.1 • Work queue, SoftIRQ • Interrupt threads in FreeBSD 9.0 • Interrupt threads in FreeRTOS, uC/OS III • Next • How to apply these ideas to CertiKOS? • Combine interrupt management with thread management we've verified before • Support symmetric multiple processors • Tasklet in Linux • Inter-processor interrupts