Shimin Chen LBA Reading Group Presentation

1. Rx: Treating Bugs As Allergies – A Safe Method to Survive Software Failures.Qin, Tucek, Sundaresan, Zhou (UIUC). SOSP’05 Shimin ChenLBA Reading Group Presentation

2. Motivation • High availability is important • Critical applications: process control, etc. • Financial company: an hour of downtime costs $6 million • SW defects account for up to 40% of system failures • Common: memory-related bugs and concurrency bugs • Bugs still occur in production runs • Even after SW company spends enormous effort on testing  Ask for mechanisms for surviving software bugs

3. Previous Work on Surviving SW Failures • Four categories: • Rebooting • Checkpointing and recovery • Application-specific mechanisms • Recent proposals: • Failure-oblivious computing • Reactive immune system

4. Previous Work 1: Rebooting • Schemes: • Whole program restart • Micro-rebooting of partial system components • SW rejuvenation (proactively restart processes) • Problem: • Cannot deal with deterministic bugs • Restart time

5. Previous Work 2: General checkpointing and recovery • Schemes: • Checkpoint, rollback, re-execute • Or use a backup server • Problems: • Cannot deal with deterministic bugs • Progressive retry in distributed systems: • Reorder messages to get around SW bugs, but not bugs on single system • N-version programming: • Too expensive

6. Previous Work 3: Application-Specific Recovery Mechanisms • Multi-process model (MPM) • Kill a request-handling process and start a new one • Problems: • Cannot handle deterministic bugs • What if shared data structure is corrupted?

7. Previous Work 4: Recent Non-Conventional Proposals • Failure-oblivious computing • Manufacture values for out-of-bound reads • Discard out-of-bound writes • Reactive immune system • Detect failures of function calls • Forcefully return from the function with a manufactured error return value (e.g. -1 for int, 0 for unsigned int etc.) • Problem: • Unsafe for correctness-critical applications (e.g. banking)

8. New Proposal: Rx • Rollback the program to a recent checkpoint when a bug is detected • Dynamically change the execution environment based on the failure symptoms • Re-execute the buggy code in the new environment • Features: • Comprehensive: can deal with deterministic bugs • Safe: do not speculatively “fix” bugs, but change environment • Noninvasive: no changes to app source code • Efficient • Informative: help locating the bugs

9. Outline • Introduction • Main Idea of Rx • Rx Design & Implementation • Evaluation • Summary

10. Main Idea Record the changes for offline diagnosis

11. Useful Execution Environmental Changes • Must be safe and may avoid bugs • Memory management based • Buffer overflows, dangling pointers, etc. • Timing based • Concurrency bugs • User request based • Dropping unexpected (malicious) user request • As a last resort

13. Outline • Introduction • Main Idea of Rx • Rx Design & Implementation • Evaluation • Summary

14. Rx Components Overview 4 1 2 3 5

15. Sensors for Detecting SW Failures • OS-raised exceptions: • Assertion failures, segfault, divide-by-zero, etc. • Fine-grain detection: • buffer overflow, accesses to freed memory etc. • Only implemented OS-raised exceptions

16. Checkpoint and Rollback (Flashback) • Memory state: fork-like operation • Files: keep a copy of each accessed files and file pointers for a checkpoint • Checkpoint management: • Equal intervals or exponential landmarks • Limit oldest checkpoint by considering recovery time goal • Multi-threaded process checkpointing • Send a signal to all threads to make them exit from blocked syscalls with EINTR • Take checkpoint • Library wrapper in Rx retries syscalls • High cost so cannot be frequent

17. Environment Wrappers • Memory wrapper: (intercepting library calls) • Delaying free: • keep a freed buffer for a threshold (process) time • FIFO recycling • Padding buffers: • adds two fixed-size padding to both ends of allocated buffers • Allocation isolation: • put allocated buffers to isolated locations • Zero-filling • Do the above during re-execution for failed code region only

18. Other Wrappers • Message wrapper (in proxy) • Randomly shuffle message orders of different connections while keeping the message order of the same connection • Randomize packet sizes • Process scheduling: change process’ priority • Signal delivery: randomize hw interrupt delivery time while preserving order • Dropping user requests • Binary search for bad requests • Drop at most 10% of requests

19. Proxy

20. Control Unit • Coordinate checkpoint/roll back, environment changes etc. • Failure vector <S1, S2, …, Sm> per failure symptom (exception type, PC adderss, call chain etc.) • Si is the score for environmental change #i • If change #i is successful, Si++; if failed, Si - - • Try the changes with scores greater than a certain threshold first

21. Outline • Introduction • Main Idea of Rx • Rx Design & Implementation • Evaluation • Summary

22. Setup • A client machine and a server machine • 2.4GHz x86 CPU, 512KB L2 cache, 1GB DRAM • 100Mbps Ethernet Injected bugs

23. Overall Results

24. Checkpoint Overhead • Time: with checkpoint interval of 200ms, 5% overhead (MySQL) • Workloads: • apache, squid: 5 threads, GET files with size uniform [1KB, 512KB] • CVS: client exports a 30KB file • MySQL: 5 client threads, transactions on a small table

25. Summary • Rx: re-executing the buggy program region in a modified execution environment • Not panacea: • Semantic bugs, resource leaks • Latent bugs (long delay from bug to symptom)