Transient Fault Tolerance via Dynamic Process-Level Redundancy

1. Transient Fault Tolerance via Dynamic Process-Level Redundancy Alex Shye, Vijay Janapa Reddi, Tipp Moseley and Daniel A. Connors University of Colorado at Boulder Department of Electrical and Computer Engineering DRACO Architecture Research Group Workshop on Binary Instrumentation and Applications San Jose, CA 10.22.2006

2. Outline • Introduction • Background/Terminology • Software-centric Fault Detection • Process-Level Redundancy • Experimental Results • Conclusion

3. Introduction • Process technology trends • Single transistor error rate expected to stay close to constant • Number of transistors is increasing exponentially with each generation • Transient faults will be a problem for microprocessors! • Hardware Approaches • Specialized redundant hardware, redundant multi-threading • Software Approaches • Compiler solutions: instruction duplication, control flow checking • Low-cost, flexible alternative but higher overhead • Goal: Leverage available hardware parallelism in SMT and CMP machines to improve the performance of software transient fault tolerance

4. Background/Terminology • Types of transient faults (based upon outcome) • Benign Faults • Silent Data Corruption (SDC) • Detected Unrecoverable Error (DUE) • True DUE • False DUE • Sphere of Replication (SoR) • Indicates the scope of fault detection and containment • Input Replication • Output Comparison

5. Software-centric Fault Detection Hardware SoR • Most previous approaches are hardware-centric • Even compiler approaches (e.g. EDDI, SWIFT) • Software-centric able to leverage strengths of a software approach • Correctness is defined by software output • Ability to see larger scope effect of a fault • Ignore benign faults Software SoR Processor Application Libraries Cache Operating System Memory Devices Software-centric Fault Detection Hardware-centric Fault Detection

6. Process-Level Redundancy (PLR) System Call Emulation Unit • Creates redundant processes • Barrier synchronize at all system calls • Enforces SoR with input replication and output comparison • Emulates system calls to guarantee determinism among all processes • Detects and recovers from transient faults • Master Process • only process allowed to perform system I/O • Redundant Processes • identical address space, file descriptors, etc. • not allowed to perform system I/O App App App Libs Libs Libs Watchdog Alarm SysCall Emulation Unit Operating System • Watchdog Alarm • occasionally a process will hang • set at beginning of barrier synchronization to ensure that all processes are alive

7. Enforcing SoR and Determinism Redundant Processes Master Process • Input Replication • All read events: read(), gettimeofday(), getrusage(), etc. • Return value from all system calls • Output Comparison • All write events: write(), msync(, etc. • System call parameters • Maintaining Determinism at System Calls • Master process executes system call • Redundant processes emulate it • Ignore some: rename(), unlink() • Execute similar/altered system call • Identical address space: mmap() • Process-specific data: open(),lseek() Barrier Write cmd line parameters and syscall type to shmem Compare syscall type and cmd line parameters read() Write resulting file offset and read buffer to shmem Copy the read buffer from shmem lseek() to correct file offset Example of handling a read() system call

8. Fault Detection and Recovery • PLR supports detection/recovery from multiple faults by increasing number of redundant processes and scaling the majority vote logic Type of Error Detection Mechanism Recovery Mechanism

9. Experimental Methodology • Use a set of the SPEC2000 benchmarks • PLR prototype developed with Pin • Intercept system calls to implement PLR • Fault Injection • Gather an instruction count profile • Use profile to generate a test case • Test case: an instruction and a particular execution of the instruction to fault • Run with Pin in JIT mode and use IARG_RETURN_REGS to alter a random bit of the instructions source or destination registers • Fault Coverage • Use fault injector on test inputs generating 1000 test cases per benchmark • specdiff in SPEC2000 harness determines output correctness • PLR Performance • Run PLR (in Probe mode using Pin Probes) on reference inputs with two redundant processes • 4-way SMP machine, each processor is hyper-threaded • Use sched_set_affinity() to simulate various hardware platforms

10. Fault Coverage • Watchdog timeout very rare so not shown • PLR detects all Incorrect and Failed cases • Effectively detects relevant faults and ignores benign faults • Floating point correctness question (ex. 168.wupwise, 172.mgrid) • Actually different results but tolerable difference for specdiff

11. Performance • Performance for single processor (PLR 1x1), 2 SMT processors (PLR 2x1) and 4 way SMP (PLR 4x1) • Slowdown for 4-way SMP only 1.26x • Should be better on a CMP with faster processor interconnect

12. Conclusion • Present a different way to use existing general purpose SMT and CMP machines for transient fault tolerance • Differentiate between hardware-centric and software-centric fault detection models • Show how software-centric can be effective in ignoring benign faults • PLR on a 4-way SMP executes with only a 26% slowdown, a 36% improvement over the fastest compiler technique • Future Work • Implementation in a run-time system allows for dynamically altering amount of fault tolerance • Simple PLR model is presented; work on handling interrupts, shared memory, and threads (the tough one) Questions?