1 / 42

Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

Log Based Dynamic Binary Analysis for Detecting Device Driver Defects. Olatunji Ruwase Thesis Proposal. Thesis Committee: Todd C. Mowry (Chair) David Andersen Onur Mutlu Brad Chen (Google) Michael Swift (U. Wisconsin). Device Drivers: The Good, The Bad, & The Ugly.

gizi
Download Presentation

Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Log Based Dynamic Binary Analysis for Detecting Device Driver Defects Olatunji Ruwase Thesis Proposal Thesis Committee: Todd C. Mowry (Chair) David Andersen OnurMutlu Brad Chen (Google) Michael Swift (U. Wisconsin)

  2. Device Drivers: The Good, The Bad, & The Ugly • Bad: Poor code quality [Chou01, Murphy04] • Written by non kernel experts • Poorly tested Detect bugs in production driver executions • Ugly: Major cause of system failures • System crashes • OS corruption • Application corruption • Device damage • Good: Enable use of hardware devices • Kernel module in commodity OS • Distributed in binary form Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  3. Program Monitoring Using Lifeguards … eax = X edx = eax Y = edx + 1 jmp ecx … Lifeguard program Can Lifeguards be used to catch Driver Bugs ? • Lifeguards: dynamic correctness checking tools • Dynamic binary analysis to work on unmodified binaries • Instruction grained analysis to catch subtle bugs • Versatility to catch broad range of bugs • Memory [Nethercote07] • Security [Newsome05, Castro05] • Concurrency [Savage97, Yu05, Flanagan09] • Multilingual program interface [Lee10] Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  4. Why Drivers Are Difficult To Write Correctly [Ryzhyk09_Dingo] • Concurrency issues • Reentrant interrupt handling User space SYSTEM CALL BOUNDARY • Interface issues • Network stack • Kernel resources • Hardware device Upper layers of network stack Kernel space Kernel resource mgmt • Generic C language issues • Memory management Synchronous: main memory & CPU registers Driver Lifeguards effectively detect similar spectrum of issues in applications Asynchronous: I/O memory & interrupts Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  5. Potential Uses of Driver Lifeguards • Diagnosing system failures • Test sites • Customer sites • Detecting “silent” faults • Test sites • Customer sites Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  6. Outline • Motivation • Overview of Lifeguard Deployment • Thesis Question • Related work • Research Challenges • Preliminary work • Current and Future work • Timeline Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  7. Lifeguard Deployment Approaches Memory Lifeguard Monitored program • Dynamic Binary Instrumentation [PIN, VALGRIND] • Fault isolation • Imprecise checking of parallel execution Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  8. Lifeguard Deployment Approaches check_store (p) *p = … p = NULL Memory Lifeguard Multithreaded program Monitored program • Logging [AFTERSIGHT, LBA, SPECK] • Monitor parallel execution [Pokam09,Vlachos10] • Accelerate lifeguard execution[Chen08,Nightingale08,Ruwase08,Ruwase10 • Require fault containment • Protect Lifeguard • Restrict damages to faulting program Log Based Lifeguards are more promising for monitoring kernel mode drivers Execution trace • Dynamic Binary Instrumentation [PIN, VALGRIND] • Fault isolation • Imprecise checking of parallel execution Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  9. Thesis Questions • Can Log Based Lifeguards precisely detect faults in the executions of device drivers ? • Can Log Based monitoring be adapted for drivers ? • Will the Lifeguards be efficient enough for production systems (Mobile, Desktop, Cloud) ? Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  10. Outline • Motivation • Overview of Lifeguard Deployment • Thesis Question • Related work • Research Challenges • Preliminary work • Current and Future work • Timeline Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  11. Eliminating Driver Faults During Development • Avoid overheads of runtime fault detection or isolation • Cannot find all faults in production drivers SYSCALL BOUNDARY • Testing [DDT] • Drivers have too many execution paths Upper layers of network stack • Synthesize driver code [Termite] • Cannot synthesize complex features e.g. multithreading Driver • Lifeguards to detect other faults • Customer sites • Testing sites • Static analysis [Metal, RacerX, SLAM] • Drivers are too complex Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  12. Using Existing Hardware to Isolate Driver Faults • Prevent system failures due to driver faults • Little information on driver faults SYSCALL BOUNDARY Upper layers of network stack Driver • Lifeguards on customer systems • Pinpoint fault location to aid debugging • Detect “silent” driver faults Page table permissions [Nooks] User space drivers [Microdrivers, SUD] Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  13. Checking Driver Execution to Isolate Faults • Pinpoint fault location • Detect “silent” faults SYSCALL BOUNDARY Upper layers of network stack • Hardware breakpoints [DataCollider] • Sampling approach misses real faults • Lifeguard finds all faults in execution Driver • Instrumented software checks [SafeDrive,XFI,BGI] • Imprecise on parallel execution • Only memory faults studied • Logging works for parallel execution • Lifeguards for high level faults Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  14. Related Work Summary • Eliminating Driver faults during development • Static analysis [Metal, RacerX, SLAM] • Testing [DDT] • Synthesizing driver code [Termite] • Using existing hardware to isolate Driver faults • Page table permissions [Nooks] • User space drivers [Microdrivers, SUD] • Checking Driver execution to isolate faults • Instrumented software checks [SafeDrive, XFI, BGI] • Hardware breakpoints [DataCollider] Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  15. Outline • Motivation • Overview of Lifeguard Deployment • Thesis Question • Related work • Research Challenges • Preliminary work • Current and Future work • Timeline Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  16. Research Challenges • Preliminary work • Adapting Log Based Monitoring for Drivers • Understanding Device Drivers • Current and Future work • Detecting Common Driver Faults (Driver Lifeguards) • Efficiency of Driver Lifeguards Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  17. Log Based Architectures (LBA) [Chen 08] Program Lifeguard Operating System Hardware Log Simulated LBA Design • Execution logging • Toggle when monitored thread (de)scheduled • Fault containment • Lifeguard as separate process • Block program at system calls until Lifeguard catches up Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  18. Adapting Execution Logging for Driver Monitoring SYSTEM CALL BOUNDARY Network stack Upper layers of network stack Driver • Toggle point • Difficulty • Complete information for precise fault detection • Efficient • Modest storage and bandwidth costs • No lifeguard filtering costs Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  19. Adapting Execution Logging for Driver Monitoring [AFTERSIGHT] DIIFICULTY SYSTEM CALL BOUNDARY Network stack Upper layers of network stack Driver Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  20. Adapting Execution Logging for Driver Monitoring DIIFICULTY SYSTEM CALL BOUNDARY Network stack Upper layers of network stack Driver Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  21. Adapting Execution Logging for Driver Monitoring DIIFICULTY SYSTEM CALL BOUNDARY Network stack Upper layers of network stack Identify driver entry points at load time Driver Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  22. Adapting Fault Containment for Driver Monitoring Lifeguard Operating System Hardware Log Driver • Execution logging • Toggle when monitored thread (de)scheduled • Fault containment • Lifeguard as separate process • Block program at system calls until Lifeguard catches up Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  23. Adapting Fault Containment for Driver Monitoring Lifeguard OS OS Hardware Log Driver • Virtual Machine (VM) separation to protect Lifeguard [AFTERSIGHT] • Rest of system remain vulnerable to driver faults • Overhead of VM is high Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  24. Understanding Device Drivers Network Functions hard_start_xmit() irq_handler() SYSCALL BOUNDARY open() stop() get_stats() ... Upper layers of network stack PCI Bus Functions probe() remove() Driver PCI Required Functions module_init() module_cleanup() Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  25. Adapting Data Race Lifeguard for Network Drivers Thread 1 Write (X) Thread 2 Read (X) Lock (Mx) Fork (Thread2) Unlock (Mx) Lock (Mx) Unlock (Mx) Lockset + kernel synch (interrupts, spinlocks) =KernelEraser • Data race on X • Two access on X where at least one access is a write • No explicit synchronization between the accesses • Lockset algorithm for detecting races in applications [Eraser] • Shared data protected with consistent set of locks • Happens-before relation for non-lock synch. (e.g fork) [RaceTrack] Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  26. Network Driver Races Reported by KernelEraser Classification of Races * Fixed in versions 2.6.18 & 2.6.21 Workload • Load driver • Enable Ethernet • Transfer file over network • Disable Ethernet • Unload driver Simulated LBA environment Kernel version: Linux 2.6.17.1 Drivers: tg3 & tulip Driver class: Network Bus: PCI Driver VM : 2 CPU Lifeguard VM : 1 CPU Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  27. False Alarms due to Unobserved Invariants Lock(rtnl_lock); driver->open(); Unlock(rtnl_lock); … Lock(rtnl_lock); driver->stop(); Unlock(rtnl_lock) SYSCALL BOUNDARY Upper layers of network stack stop () { … while(tptg3_flags & …) … } tg3 PCI open () { … tptg3_flags &= … … } • Synchronizations in upper layers of I/O stack • Synchronizations due to device states Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  28. probe() open() ready for pkt rx/tx connected to pci bus inactive False Alarms due to Unobserved Invariants SYSCALL BOUNDARY Upper layers of network stack probe() { … tptg3_flags |= … … } tg3 PCI open () { … tptg3_flags &= … … } • Synchronizations in upper layers of I/O stack • Synchronizations due to device states Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  29. Preliminary Work Summary • Adapted Log Based Monitoring for Drivers • Identify driver code region to log only driver execution • VM separation to protect Lifeguard • Adapted Lockset (KernelEraser) to detect races in network drivers • Found 2 known but serious data races in tg3 • False alarms due to external synchronizations Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  30. Outline • Motivation • Overview of Lifeguard Deployment • Thesis Question • Related work • Research Challenges • Preliminary work • Current and Future work • Timeline Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  31. Eliminating False Alarms in KernelEraser SYSTEM CALL BOUNDARY stop () { Lock(rtnl_lock); … while(tptg3_flags & …) … Unlock(rtnl_lock); } Network stack Upper layers of network stack open () { Lock(rtnl_lock); … tptg3_flags &= … … Unlock(rtnl_lock); } Driver • External synchronizations • Network stack • Log network stack • Emulate interface invariants Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  32. probe() open() ready for pkt rx/tx connected to pci bus inactive Eliminating False Alarms in KernelEraser SYSTEM CALL BOUNDARY probe () { (INACTIVE) … tptg3_flags |= … … (CONNECTED TO BUS) } Network stack Upper layers of network stack open () { (CONNECTED TO BUS) … tptg3_flags &= … … (READY FOR TX/RX) } Driver • External synchronizations • Network stack • Log network stack • Emulate interface invariants • Device • Model finite state machine Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  33. Eliminating False Alarms in KernelEraser SYSTEM CALL BOUNDARY Network stack Upper layers of network stack Driver • External synchronizations • Network stack • Log network stack • Emulate interface invariants • Device • Model finite state machine • Other driver classes • SCSI disk • SOUND • USB • GRAPHICS Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  34. Lifeguards for Common Driver Faults [Ryzhyk09_Dingo] User space • Concurrency faults • Data Races SYSTEM CALL BOUNDARY • Memory faults • Illegal memory access • Memory leaks • Uninitialized memory use Upper layers of network stack Kernel space Kernel resource managers Network driver • Interface violations • Device protocol • Kernel protocol • I/O stack protocol Scalability ? Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  35. Efficiency of Driver Lifeguards • Accelerating Lifeguard analysis • Static analysis • Dynamic optimizations • Parallel Lifeguards • Hardware accelerators • Reduce overhead of VM fault containment • Hardware enforced fault isolation in same VM Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  36. Accelerating Driver Lifeguards Lifeguard OS OS Hardware Log Driver • Reduce analysis workload • Static analysis [XFI] Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  37. Accelerating Driver Lifeguards Lifeguard OS OS Hardware Log • Run analysis faster • Dynamic compiler optimizations [Qin06,Ruwase10] • Parallel Lifeguards [Nightingale08,Ruwase08] • Hardware accelerators [Vlachos10] Driver • Reduce analysis workload • Static analysis [XFI] Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  38. Avoid Overhead of VM Fault Containment User space Lifeguard SYSTEM CALL BOUNDARY Upper layers of network stack Kernel space Kernel resource managers Network driver • Hardware enforced fault isolation [Nooks, SUD] • Issues to consider • Protection quality • Lifeguard using Driver (e.g. disk) Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  39. Current and Future Work Summary • Detecting common driver faults • Data races • Memory • Interface violations • Efficiency of Driver Lifeguards • Accelerating Lifeguard analysis • More efficient fault containment Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  40. Outline • Motivation • Overview of Lifeguard Deployment • Thesis Question • Related work • Research Challenges • Preliminary work • Current and Future work • Timeline Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  41. Timeline Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

  42. Questions ? Thanks to members of the LBA Group for their contributions • Shimin Chen • Babak Falsafi • Phillip Gibbons • Michelle Goodstein • Michael Kozuch • Onur Mutlu • Todd Mowry • Gennady Pekhimenko • Vivek Seshadri • Theodoros Strigkos • Evangelos Vlachos Log Based Dynamic Binary Analysis for Detecting Device Driver Defects

More Related