Scalable Network - based Buffer Overflow Attack Detection

1. Scalable Network-basedBuffer Overflow AttackDetection Tzi-cker Chiueh Computer Science Department Stony Brook University Stony Brook, NY, U.S.A. chiueh@cs.sunysb.edu Fu-Hau Hsu Department of Computer Science and Information Engineering National Central University Taoyuan, Taiwan, R.O.C. hsufh@csie.ncu.edu.tw Fanglu Guo Symantec Research Laboratory Cupertino, CA, U.S.A. fanglu_guo@symantec.com ANCS 2006

2. Virulence of Buffer Overflow Attacks • Buffer overflow attack is arguably the most widely used and thus most dangerous attack method used today. • Most Internet Worms use it to proliferate themselves. • It accounts for more than 50% of all the security vulnerabilities recorded by CERT. ANCS 2006

3. Proposed Solutions • Compiler Transformation • Stack Guard, RAD, Address Obfuscation • Library Rewriting • OS • Non-executable Stack • Instruction Set • Hardware • AMD Athlon-64 ANCS 2006

4. Discrepancy between Theory and Practice • In theory, these efforts have largely solved the buffer overflow attack problem. • In practice, however, new buffer overflow vulnerabilities are still discovered and reported on a routine basis. ANCS 2006

5. substantial modification • substantial resistance Why? ANCS 2006

6. A Solution to the above Dilemma --Nebula • Nebula • A network-based buffer overflow attack detection mechanism • Observe the network traffic only to detect BOAs • Currently version is developed for Linux paltforms. ANCS 2006

7. Existing Network-based Intrusion Detection System (NIDS) • Misuse intrusion detection • Zero-day BOAs • Labor-Intensive • Solution: automatically signature-generating approaches • Anomaly intrusion detection • False Positive ANCS 2006

8. Generalized Signature ANCS 2006

9. Two Factors for a Successful Buffer Overflow-style Attack • A successful buffer overflow-style attack should be able to overflow the right place (e.g. the place to hold a return address with the correct value (e.g. the address of injected code entry point)). ANCS 2006

10. Non-predicable Offset and Entry Point Address return address buffer where the overflow start injected code address of injected code entry point. offset between the beginning of the overflowed buffer and the overflow target. The offset and the entry point address are non-predicable. They can not decided by just looking the source code or local binary code. ANCS 2006

11. Non-predicable Offset • For performance concerns, most compilers don’t allocate memory for local variables in the order they appear in the source code, sometimes some space may be inserted between them. (Source Code doesn’t help) • Different compiler/OS uses different allocation strategy. (Local binaries don’t help) • Address obfuscation insert random number of space between local variables and return address. (Super good luck may help) ANCS 2006

12. Function main()’s stack frame Non-predicable Entry Point Address webserver –a –b security [fhsu@ecsl]# system data 0xbfffffff environment variables argument strings command line arguments and environment variables env pointers argv pointers argc ANCS 2006

13. Strategies Used by Attackers to Increase Their Success Chance • Repeat address patterns. • Insert NOP (0x90) operations before the entry point of injected code. ANCS 2006

14. Indispensable Elements of BO-style Attacks ‘ The Address ’ • For buffer overflow attacks, it is the address of the entry point of injected code. ANCS 2006

15. Linux Process Memory Layout 0xffffffff kernel address space 0xc0000000 address space of addresses of injected code and frame pointers (Stack Address Zone) user stack %esp 8M for Shared libraries, including libc functions 0x40000000 brk run-time heap data and code ANCS 2006

16. Size of Stack Address Zone • The default maximum size of a process’s user space stack is 8 Mbytes. • However, according to Ditzel et al., the average function frame size is 28 bytes. • Therefore, the majority of program are not supposed to use a 2Mbyte stack. • In our test, a 8k stack is enough to identify all 10 remote exploit strings. ANCS 2006

17. Repeating Times and Values of Return Addresses 2k stack --- 0xbffffffff ~ 0xbfffe000 ANCS 2006

18. A Property of Stack Addresses • The leading byte of any words that contain a stack address corresponds to a non-printable ASCII character. ANCS 2006

19. Generalized Signature • Signature of a stack smashing buffer overflow attack : If a sub-string of a traffic string could be interpreted as a stack address that repeats 3 or more times, it is alarmed as a buffer overflow attack string. ANCS 2006

20. Contextual Analysis ANCS 2006

21. Bypassing Detection • Patient attackers could bypass detection based on repeating address signature by repeating addresses no more than 2 times. • PS: All the 10 remote exploit code we tested repeat at least 4 times. • Attackers repeat the addresses to increase their chance to success. In other words, it is very likely that without the repeat, attackers will fail many times before getting a successful one. ANCS 2006

22. Unsuccessful Attacks • Buffer overflow-style attacks will destroy targeted process’s address space which in turn usually will crash the attacked process. • In order to recycle valuable system resources, OS will close the sockets opened by crash processes automatically. • On both Linux and Windows, when a program is crashed, the OS will terminate all the program’s pending socket connections by sending out an RST packet to the communicating hosts on its behalf. ANCS 2006

23. Server Termination Signature • After forwarding a sub-string which could be interpreted as a single stack address, Nebula detects that the server closes the TCP connection without sending any data, then the traffic string is deemed as a buffer overflow attack string. • Future traffic coming from the same hosts will be blocked or examined thoroughly. ANCS 2006

24. Will Normal Traffic Behavior the Same Way? • HTTP Protocol (RFC 2616) works in the request-reply way. (After the request, there will be a reply before the server close the connection) • SMTP protocol (RFC 2821), for e-mail, and FTP protocol(RFC 959) use QUIT command to close a connection. (QUITcan not be interpreted as a stack address.) ANCS 2006

25. Payload Bypassing ANCS 2006

26. Payload Bypassing • Payload bypassing tries to avoid packet analysis for as much traffic as possible. • Because most buffer overflow attacks take place during the exchange of control messages, it is safe to ignore the bulk of data that is downloaded as uninterpreted bytes. • For example, in an FTP session, data transferred over the data connection can never be used to mount a buffer overflow attack against the FTP program because the FTP program does not interpret them. ANCS 2006

27. Internet Traffic Statistic • From CacheLogic’s measurement on USA, Europe, and Asia backbone in June 2004, HTTP and P2P packets accounted for more than 70% of the total traffic. ANCS 2006

28. Percentage of Payload • Percentage of payload in the traffic when each of the four protocols that Nebula can recognize is used to transfer files of a total size of 1.22 Gbytes. ANCS 2006

29. Number of False Positives without Payload Bypassing • Number of false positives under the our sample as reported by Nebula. • The minimal number of times the attack pattern is repeated is assumed to be 1, 2, 3 or 10, and the stack size tested is 2Mbytes, 16Kbytes, or 8Kbytes. In each entry the left is the number of false positives for RTL attacks, whereas the right is the number of false positives for CI attacks. • The sample includes 134966 TCP connections and about 1.582 Gbytes of data. ANCS 2006

30. Number of False Positives with Payload Bypassing • The number of false positives in the test traffic associated with different protocols after applying payload bypassing is negligible even when the attack pattern repetition count is 1. ANCS 2006

31. Throughput Comparison • The throughput of Nebula under a test HTTP connection when different options are turned on. With payload bypassing, Nebula can perform buffer overflow attack detection and still achieve a throughput higher than a generic Linux router. ANCS 2006