Enhancing Availability and Security Through Failure-Oblivious Computing

1. Enhancing Availability and Security Through Failure-Oblivious Computing Martin Rinard, Cristian Cadar, Daniel Dumitran, Daniel Roy, and William Beebee, Jr.

2. Introduction • Memory errors are a common source of program failures • ML and Java use dynamic checks to eliminate such errors • Assumption: • Invalid memory access  unsafe to continue the execution

3. Failure-Oblivious Computing • Instead of throwing an exception or terminating • Ignores any memory access errors and continue • Read (an out of bounds array element) • Just read a manufactured value • Write (an out of bounds array element) • Discard the value

4. Wrong Results? • Many programs can continue to run • As long as errors do not corrupt the program’s address space or data structures • Failure-oblivious computing can improve the availability, robustness, and security of such programs

5. Shouldn’t We Stop at the First Error? • Debugging may not be an option • No source code • Not enough time • Failure-oblivious computing can still provide acceptable service • Better than no service

6. Servers and Buffer-overrun Attacks • When a program allocates a fixed-size buffer • Then fails to check if input string fits in the buffer • A long input string containing executable code can overwrites the stack contents • Can coerce the server into running arbitrary code

7. Servers and Buffer-overrun Attacks • Failure-oblivious computing discards the excess characters, preserving the integrity of the stack • Server detects invalid request and returns an error • Converts a dangerous attack into an invalid input

8. Multiple Items or Outputs • Many programs (e.g. mail readers) process multiple items • Some applications generate multiple outputs • Some outputs are more important than others • Without failure-oblivious computing • Failure to process one can prevent the program to process the rest

9. Benefits and Drawbacks + Increased resilience • Graceful degradation and continue to operate successfully on most of its inputs + Increased security • Can survive stack overruns + Reduced development costs • Pressured to find and eliminate all disruptive bugs + Reduced administration overhead • Reduce the success rate of attacks

10. Benefits and Drawbacks + Safer integration • Lowers the risks to use foreign components - May generate unacceptable results • Inevitable consequence for better resiliency • Need to convert unanticipated states into anticipated error states

11. Scope • Interactive computing environments • Mailers • Servers • System administration tools • Operating systems • Document processing systems • Mission critical applications • Halting is not an option

12. Scope • Less appropriate for programs • Hard to determine whether the output is correct • Safety-critical applications • Safer to terminate the computation

13. Example • A Mutt procedure • Takes an input string • Returns an encoded output string • Fails to allocate sufficient space • With standard compilers • Writes succeed, corrupt the address space, and program segfaults • With safe-C compilers • Mutt exits before presenting the GUI

14. Example • With the failure-oblivious compiler • The returned string is incorrect • Server responds with an error • Failure oblivious approach works • Mostly correct programs • With subtle errors

15. Implementation • Failure oblivious compiler • Generate two kinds of additional code • Checking code • Discard erroneous writes • Manufactures values for erroneous reads • Continuation code • Executes when checking code detects an attempt to perform illegal access

16. Checking Code • Jones and Kelly’s Scheme • Track the locations to structs, arrays, variables • Each data item is padded with an extra byte • Initialized to ILLEGAL • Check the status of each pointer before dereferencing it

17. Continuation Code • Write continuation code • Discards the value • Read continuation code • Redirects the read to a preallocated buffer of values • Iterates through all small integers • Increasing the chance to exit loops • To avoid nontermination • Mostly 0s and 1s

18. Continuation Code • Optional logging • Can be used to track down errors • Failure-oblivious computing • Can also reduce the incentive to eliminate errors

19. Case Studies • Recompiled widely-used open-source programs with known memory errors • Pine (mail user agent) • Midnight commander (file manager) • Sendmail (mail transfer agent) • Mutt (mail user agent) • Samba (file server) • WsMp3 (mp3 server) • Apache (http server)

20. Methodology • Compare each program compiled differently • By a standard C compiler • By the CRED safe-C compiler • By the failure-oblivious compiler • Workloads • Contain inputs that exploit known security vulnerabilities

21. Pine 4.44 • Fails to parse certain legal From fields • Possible to execute arbitrary code • Standard version: crashed • Safe version: terminated with an error • Failure oblivious version: continued to run • Was able to forward the read and forward the message with the problematic From field

22. Midnight Commander • Problems with symbolic links in tgz files • Standard version: segfaulted • Safe version: terminated with an error message • Failure-oblivious version: continued to run

23. Sendmail 8.11.6 • Allows root privilege to execute arbitrary code on the machine running the Sendmail server • Standard version: vulnerable to an attack to gain the root shell • Safe version: exited with an error message • Failure-oblivious version: not vulnerable to the attack

24. Mutt 1.4 • Memory error in the conversion from UTF-8 to UTF-7 string formats • Standard version: crashed • Safe version: exited with an error message • Failure oblivious version: continued to run • 6x slow down • Took about 1 second to load 3,000 messages

25. Samba 2.2.5 • Memory corruption error • A remote user can obtain the root shell • Standard version: vulnerable to an attack to gain the root shell • Safe version: functional until the attack • The child process exited • Failure oblivious version: continued to run • Similar performance compared to the safe version

26. WsMp3 0.0.5 • Memory-error vulnerability • Standard version: segfaulted • Safe version: crashed the entire server • Single threaded • Failure-oblivious version: survived the attack

27. Apache 2.0.47 • mod_alias contains a memory-error vulnerability • Standard version: child process segfaulted • Safe version: child process exited properly • Failure-oblivious version: child process redirected the attacking request to a nonexistent URL • The child process stayed alive and processed subsequent requests correctly

28. Gzip 1.2.4a • Memory error in its file name processing code • An attacker can run arbitrary code • Standard version: segfaulted • Remaining files were not processed • Safe version: exited at the problematic file • Failure-oblivious version: prompted an error message for the problematic files • Proceeded to process all remaining files • 10x slow down (1.2 MB/sec)

29. Discussion • Failure oblivious versions survived all memory-corruption attempts • Work well for this class of applications • One input has a minimal effect on the next input • Unless it corrupts the data structures or address space • Little performance degradation for interactive programs • Safe versions are prone to DoS attacks • Tend to terminate prematurely

30. Related Work • Any safe-C compiler can be modified to implement a failure-oblivious compiler • Discard writes • Manufacture values for unsafe reads • Typically < 2x slow down • Occasionally 8x slow down • Does not perceptibly degrade the response times of interactive programs • Also I/O-bound programs

31. Safe Languages • Java and ML • Modify the exception handling code • Discard illegal writes • Return manufactured values for illegal reads

32. Traditional Error Recovery • Traditional approaches • Reboot • Checkpointing • Partial system restarts • Hardware redundancy • Failure-oblivious computing reduces down time and vulnerabilities to persistent errors • Restarting Pine will not solve the problem

33. Other Approaches • Data structure repair • Failure-oblivious approach is preventive • Statically detect all buffer-overrun errors • May conservatively reject almost working code • Buffer-overrun detection tools • Detect overwriting the return address • Detect overwriting function pointers • Failure-oblivious approach prevents the attack from corrupting the address space

34. Conclusion • Failure-oblivious computation enhances availability, resilience, and security • Converts dangerous unknown system states to known error cases