Using Model-Checking to Debug Device Firmware

1. Using Model-Checking to Debug Device Firmware Sanjeev Kumar Microprocessor Research Labs, Intel Kai Li Princeton University

2. Programmable Devices D D CPU CPU Mem Mem D D Main CPU Main Memory Bus Network Card Disk Network Move functionality from main CPUs to devices Device firmware is more complex Using Model Checking to Debug Device Firmware

3. Firmware for Programmable Devices • Difficult to write and debug • Use concurrency • Inherently difficult to program correctly • Optimized for high performance • Tradeoff program simplicity for performance • Limited debugging support available • Firmware reliability is important • Firmware is trusted by the OS • Bugs in firmware can crash the entire machine • Model checking is a promising approach Using Model Checking to Debug Device Firmware

4. Model Checking Program Model Test Code Systematically verify properties of concurrent systems • Using State-space exploration • Try all possible scheduling options • Advantages • Automatic • Produces counter example • Disadvantages • Computationally expensive • Exponential search • Models cannot be too big Using Model Checking to Debug Device Firmware

5. Using Model Checking to Debug Firmware • Extracting models from programs • Manually (by the programmer) • Automatic (using the compiler) • Reduces programmer effort • Reduces mismatch between program and model • Extracting smaller models models • Support for abstraction • Discard irrelevant details in the program Using Model Checking to Debug Device Firmware

6. Our Work • Extract abstract models using a compiler • General compiler techniques • Programmer controls the abstraction process • Specifies what needs to be abstracted • Compiler performs the abstractions conservatively • Extract models for Spin model checker from programs written in the ESP language • Make practical choices • Goal: Debugging and not Verification • Used to debug VMMC firmware for a network card • Found 7 bugs that can cause the firmware to deadlock • Could not find these bugs without support for abstraction Using Model Checking to Debug Device Firmware

7. Related Work • Manual Model Extraction • Harmony, RUBIS, Plan 9, Fluke OS • Used model checking to debug a subsystem • Automatic Model Extraction • Teapot, Promela++, Esterel, Java Pathfinder • Domain-specific and general-purpose languages • Automatic Extraction + Support for abstraction • Feaver, Lie et. al., Bandera Using Model Checking to Debug Device Firmware

8. Outline • Background • Extracting Abstract Models using a Compiler • From ESP Language • For Spin Model Checker • Evaluation: Debugging VMMC Firmware • Conclusion and Future Work Using Model Checking to Debug Device Firmware

9. ESP: A Language for Programmable Devices Develop and Test using Model Checker pgm1.spin test1.spin pgmN.spin testN.spin pgm.ESP ESP Compiler Generate Firmware pgm.C help.C Goals • Easy to program • Allow extensive testing • Performance Using Model Checking to Debug Device Firmware

10. The ESP Language • Concurrent language: Processes & Channels • Pure message-passing communication • in, out, alt operations on channels • Channels are synchronous or unbuffered • Processes and channels are static • A number of interesting features • Explicit memory management scheme that uses model-checking to ensure safety • Supports dispatch on channels • Efficient and powerful interface to C Using Model Checking to Debug Device Firmware

11. Extracting Models for Spin • Step 1 : Detailed models [ PLDI’01 ] • Translate each language construct into Spin • Simple translation • int, bool, records, arrays, unions • If-then-else, while-loops • process, channel • Spin does not support • Dynamic memory allocation & Pointers • Additional bookkeeping necessary to support these • Can be used to check local properties • Debug subsystems (1-2 processes) separately • Too big to debug the entire system Using Model Checking to Debug Device Firmware

12. Extracting Abstract Models Conservatively • Step 2: Abstract models • Necessary to check global properties (like deadlocks) • Drop unnecessary details • Depending on the property being verified • Programmer controls the abstraction • Abstraction specified by the programmer • Drop variables • Drop fields from records and unions • Compiler used the abstraction specified conservatively • Could introduce fast-positive bugs • All bugs in the programs will be present in the extracted model • Involves dealing with a number of tricky cases Using Model Checking to Debug Device Firmware

13. Examples X if :: b1 = true :: b1 = false fi X typerecT =#record of{ int count; } X $r1: recT = {0}; if (b) { r2 = r1; } ... r1.count = 5; X if :: r2.count = 5 :: skip fi X Conservative: Use nondeterminism to broaden the state-space searched $b2: boolean = true; ... $b1:boolean= b2; Using Model Checking to Debug Device Firmware

14. Outline • Background • Extracting Abstract Models using a Compiler • Evaluation: Debugging VMMC Firmware • Conclusion and Future Work Using Model Checking to Debug Device Firmware

15. High-performance communication Bypass OS for data transfers Used Myrinet network cards Gigabit network 33 MHz CPU, 1 MB memory Original VMMC firmware Implemented in C Several man-years of debugging Still encounter bugs Some involve complex race conditions that are triggered only occasionally VMMC Application Data OS Network Card Network Using Model Checking to Debug Device Firmware

16. Debugging VMMC Firmware • Reimplemented VMMC firmware using ESP • Used model checking to debug • Global property of program (deadlocks) • Hard-to-find bugs • Found 7 bugs using abstract models • 4 Bugs would cause deadlock during normal operations • 3 Bugs would be triggered only by a malicious machine • Could not find these bugs without abstractions • No firmware bugs encountered on device • Microbenchmarks • SPLASH2 parallel application suite • On a 16-processor SMP cluster Using Model Checking to Debug Device Firmware

17. Resource used for Model Checking VMMC Firmware * Limiting Resource • Only partial search was possible • Even partial searches were effective Using Model Checking to Debug Device Firmware

18. Model extracted from VMMC Firmware • Programmer only write a small amount of Spin Code • Program can be rechecked with little effort Using Model Checking to Debug Device Firmware

19. Outline • Background • Extracting Abstract Models using a Compiler • Evaluation: Debugging VMMC Firmware • Conclusion and Future Work Using Model Checking to Debug Device Firmware

20. Conclusions • Use compiler to extract abstract models • Evaluation: Debugged VMMC firmware • Using compiler to extract models is good • Significantly reduces effort required to model check • Abstraction is required • To check global properties like deadlocks • Programmer can control the abstraction • Compiler is conservative • Does not require the programmer to be correct • Only partial search was possible • Still effective in finding bugs Using Model Checking to Debug Device Firmware

21. Future Work • Optimizations to reduce size of state space • Eliminating more redundancies • Quantify the effectiveness of a partial seach • Estimate the fraction of state-space searched • Use type systems to reduce the size of state space that has to be searched Using Model Checking to Debug Device Firmware

22. Questions? To find out more, Visit http://www.cs.princeton.edu/~skumar Using Model Checking to Debug Device Firmware

23. Debug not Verify • Several sources of incompleteness and unsoundness remain • Programmer supplied Spin code • Partial model checking • The goal is to isolate/reduce the unsound portions of the code Using Model Checking to Debug Device Firmware

24. ABCDEF Abcdef Ghijk ABCDEF Abcdef Ghijk Abcdef Ghijk Abcdef Ghijk Abcdef Ghijk ABCDEF Abcdef Ghijk Using Model Checking to Debug Device Firmware