An Automated Approach for Software Reliability and Security

1. An Automated Approach for Software Reliability and Security Zhen Xiao Senior Technical Staff Member AT&T Labs – Research Joint work with Christof Fetzer

2. Motivation • Software reliability is becoming increasingly important. • Financial transactions, virtual office environment, life critical applications, etc. • Does the software function correctly under exceptional or stressful settings? • Field experience indicates that the error handling paths in the software typically contain most bugs. • Testing all boundary conditions before the official release of the software can be prohibitively expensive.

3. Example: Robustness Violation strcpy(destination, source) source length destination unmapped pages detectable with help of signal handler

4. Security Problems strcpy(dst, attack_string) attack_string length destination shell script attack code We want to avoid all buffer overwrites!

5. Why C/C++? What is a good approach for increasing software reliability?

6. Challenges • Transparency • Cannot assume access to the source code for applications & libraries. • Cost effectiveness & scalability • Large number of applications, shared libraries, and functions. • Applications and libraries may change often. • Flexibility • Different applications may need different levels of protection

7. Most Popular Libraries

8. Distribution of #Libraries vs. #Applications

9. Number of shared libraries 933 Number of applications 15431 Number of undefined func/appl ~49 Linux SuSE 8.0 Automation is essential for protecting a large number of libraries and applications!

10. The HEALERS Approach • Fault-containment wrappers • Intercept function calls into the dynamic link libraries. • Provide transparent protection for software without source code access. • Automated fault-injection experiments • Find all functions defined in a library • Detect arguments that cause a function to crash • Derive safe argument types for each function. • Prevent heap and stack buffer overflows and a large set of robustness violations.

11. The HEALERS Approach (Cont’d) • Micro-generator architecture • Generate a variety of wrappers through a set of micro-generators. • Applications only pay the overhead they actually need.

12. Extracting Libraries and Undefined Functions

13. Example: Unix Wrappers • Overwrite “exit” by “abort” • Wrapper: void exit(int status) { abort(); } • Start wrapper: > setenv LD_PRELOAD `pwd`/wrapper.so • No need to change existing programs! > date Wed Oct 18 13:06:41 EDT 2000 Abort

14. Wrapper check d and s strcpy(d,s) Approach: Wrap Functions Application calls strcpy(d,s) calls Shared Library strcpy(d,s)

15. Keeping Track of the Heap • Wrap malloc, calloc, realloc, free • Use Red/Black trees to keep meta-data for each allocated block • Update Red/Black tree whenever such a function is called • Red/Black node contains address range of each block • Check if the destination buffer is sufficiently large: • Search Red/Black tree to find pointer • returns #bytes between pointer and the end of the block • Complexity: • Insert, Remove, Find: O(log(entries))

16. Wrapper Structure

17. Original Function w Wrapper needs to call the “original” function w Access to original function by “ ” dlsym w Example: dlsym(RTLD_NEXT, “strcpy”); – returns address of original “strcpy” function –

18. Wrapper Structure

19. Fault-Injection Experiments Wrapper Generation Process Function Specification Shared Library Wrapper Generation Wrapper Phase 1: Phase 2: Function Spec. Wrapper generation Generation

20. Run-time checks [SRDS2001] • Example: function asctime(d) • wrapper checks that d is buffer of sufficient size • if not, return error code • otherwise, call original function

21. Asctime Wrapper char* asctime (const struct tm* a1) { char* ret; if (!check_R_ARRAY_NULL(a1,44)) { errno = EINVAL ; ret = (char*) (nil); goto PostProcessing; } asm("movl %ebp,%esp"); asm("popl %ebp"); asm("jmp *libc_asctime_0"); PostProcessing: ; return ret; }

22. Function Specification for asctime <function> <name>asctime</name> <argument>const struct tm* <robust_type>R_ARRAY_NULL[44]</robust_type> </argument> <return_type>char *</return_type> <error_value>NULL</error_value> <errors> <errno>EINVAL</errno> </errors> <attribute>unsafe</attribute> </function> Robust Argument Type

23. Function Declaration Function Declaration Function Declaration Function Declaration Generation of Function Specifications Fault - Header Injector Files Fault - Injector Manual Generator Pages Fault - Injector Shared Fault - Library Injector

24. Generator Algorithm For each function in library • Find prototype of function: • Parse include files given in manual pages, or • Parse all header files that might contain prototype • Generate fault-injector using prototype • Define sequence of hypotheses • Perform automated fault injection experiments to test each hypothesis • Select non-rejected hypothesis

25. Prototype Extraction for glibc2.2 Total: 1278 global functions

26. Errors in Manual

27. crash no crash Computation of Robust Argument Types • Ideal Goal: determine set of argument values that crash a function set of all argument values wrapper accepts these values, rejects others

28. Idea Goal Not Realistic • Revised Goal: Accurate but not necessarily complete checks: crash no crash set of all argument values wrapper accepts these values, rejects others

29. Approach • Divide argument value set in disjoint subsets set of all argument values Fundamental Type

30. Classify Fundamental Types Using Fault-Injections • Divide argument values in disjoint subsets no value crashes f wrapper accepts these values, rejects others all values crash f some values crash f

31. not mapped read only 3 Example: Fixed Arrays set of all argument values WONLY_FIXED RW_FIXED RONLY_FIXED UNMAPPED INVALID RONLY_FIXED[1], RONLY_FIXED[2], RONLY_FIXED[3], … NULL

32. asctime: injection results • Crashes for all test cases in: • RONLY_FIXED[i] for i < 44 • WONLY_FIXED[i] for any i • RW_FIXED[i] for i < 44 • INVALID crashes • Does not crash for test cases in: • RONLY_FIXED[i] for i >= 44 • RW_FIXED[i] for i >= 44 • NULL

33. Type Hierarchy • Need to be able to compute union of value sets • Define type hierarchy: • fundamental types: value sets of any two fundamental types is non-overlapping • union types: value set of this type is the union of the value set of its “subtypes”

34. R_ARRAY_NULL[s] W_ARRAY_NULL[s] s≤t s≤t RW_ARRAY_NULL[t] s≤t R_ARRAY [t] W_ARRAY[t] t≤v t≤v RW_ARRAY [u] u≤v Fixed Array Type Hierachy UNCONSTRAINED INVALID NULL t≤u t≤u RONLY_FIXED[v] WONLY_FIXED[v] RW_FIXED[v]

35. R_ARRAY_NULL[s] W_ARRAY_NULL[s] s≤t s≤t s≤t asctime: Robust Argument Type UNCONSTRAINED RW_ARRAY_NULL[t] R_ARRAY [44] W_ARRAY[t] 44≤44 t≤u 44≤v RW_ARRAY [44] RONLY_FIXED[v] 44≤v RW_FIXED[v]

36. asctime: Robust Argument Type UNCONSTRAINED R_ARRAY_NULL[44] W_ARRAY_NULL[s] 44≤44 44≤44 RW_ARRAY_NULL[44] s≤t R_ARRAY [44] W_ARRAY[t] NULL 44≤44 t≤u 44≤v RW_ARRAY [44] RONLY_FIXED[v] 44≤v RW_FIXED[v]

37. Phase 2: Generation of Wrappers Retry Wrapper Function Security Wrapper Specification Wrapper Generator Flags Robustness Wrapper …

38. Micro-Generators Generator

39. Wrapper Types Data Collection Wrappers: w collect failure data and usage data – Security Wrapper: [SRDS2001] w detect buffer overflows on stack and heap – Robustness Wrappers: [DSN2002] w prevent segmentation failures – try to keep applications running – Retry Wrapper: [ISSRE 2002] w – retry failed function calls … w

41. Wrapper Types Data Collection Wrappers: w collect failure data and usage data – Security Wrapper: [SRDS2001] w detect buffer overflows on stack and heap – Robustness Wrappers: [DSN2002] w prevent segmentation failures – try to keep applications running – Retry Wrapper: [ISSRE 2002] w – retry failed function calls … w

42. Creating Profiling Wrappers • Create Profile Wrapper: > profile_app ls • Run wrapped ls: > wrapped_ls/ls • Output of profile wrapper in XML

43. Profiling Wrapper

44. Wrapper Types Data Collection Wrappers: w collect failure data and usage data – Security Wrapper: [SRDS2001] w detect buffer overflows on stack and heap – Robustness Wrappers: [DSN2002] w prevent segmentation failures – try to keep applications running – Retry Wrapper: [ISSRE 2002] w – retry failed function calls … w

45. Security Wrapper • Currently the wrapper can detect • heap smashing attacks (caused by C-library func.) • stack smashing attacks • Stack Smashing Detection: • based on an approach by LibSafe • uses gcc frame pointers to check that return address is not overwritten

46. Creating Security Wrappers • Create Wrapper for individual library: > protect_library /lib/libc.so • Create Wrapper for all libraries: > protect_all_libraries • Info about Security Wrappers in XML

47. Wrapper Types Data Collection Wrappers: w collect failure data and usage data – Security Wrapper: [SRDS2001] w detect buffer overflows on stack and heap – Robustness Wrappers: [DSN2002] w prevent segmentation failures – try to keep applications running – Retry Wrapper: [ISSRE 2002] w – retry failed function calls … w

48. Robustness Wrapper tested with Ballista

49. Performance Measurements • Pentium 3, 864Mhz, 384 Mbytes RAM • Linux 2.4.4 kernel, SuSE 7.2 • Performance data: • each data point is 10% trimmed mean of 100 executions

50. Performance