1 / 37

Comprehensive Shellcode Detection using Runtime Heuristics

Comprehensive Shellcode Detection using Runtime Heuristics. Michalis Polychronakis (Columbia University,USA ), Kostas G. Anagnostakis ( Niometrics , Singapore), Evangelos P. Markatos (FORTH-ICS, Greece) ACSAC,2010. Outline. Introduction Architecture Runtime Heuristics Implementation

malcolm-kirby
Download Presentation

Comprehensive Shellcode Detection using Runtime Heuristics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Comprehensive Shellcode Detection using Runtime Heuristics MichalisPolychronakis(Columbia University,USA), Kostas G. Anagnostakis(Niometrics, Singapore), Evangelos P. Markatos(FORTH-ICS, Greece) ACSAC,2010

  2. Outline • Introduction • Architecture • Runtime Heuristics • Implementation • Experimental Evaluation • Conclusion

  3. Introduction • The injected code, known as shellcode, carries out the first stage of the attack, which usually involves the download and execution of a malware binary on the compromised host.

  4. Introduction • Identify thepresence of shellcode in network inputs using static code analysis- advanced obfuscation-self-modifications • Dynamic codeanalysis using emulation- quite effective

  5. Introduction • But these are confined to the detection of a particular class of polymorphic shellcode that exhibits self-decrypting behavior • Metamorphism

  6. Introduction • In this paper, present a comprehensive shellcode detection technique based on payload execution • Detection method relies on several runtime heuristics tailored to the identification of different shellcode types • Gene, a network level detector based on passivenetwork monitoring

  7. Architecture

  8. Architecture • Shellcode is meant to be injected into a running process and it usually accesses certain parts of the process' address space • Gene is equipped with a fully blown virtual memory subsystem that handles all user-level memory accesses

  9. Runtime Heuristics

  10. Runtime Heuristics - Resolving kernel32.dll • Windows API is divided into several dynamic load libraries (DLLs) • In order to call an API function, the shellcode must firstfind its absolute address in the address space of the process • Searching for the Relative Virtual Addresses (RVAs) of the function in the Export Directory Table (EDT) of the DLL- LoadLibrary- GetProcAddress

  11. Runtime Heuristics - Resolving kernel32.dll • No matter which method is used, a common fundamental operation in all methods is that the shellcode has to first locate the base address of kernel32.dll

  12. Runtime Heuristics - Process Environment Block • Process Environment Block[PEB]- a user-level structure that holds extensive process specific information

  13. Runtime Heuristics - Process Environment Block • Condition P1. • (i) the linear address ofFS:[0x30]is read • (ii) the current or any previous instruction involved the FS register, then this input may correspond to ashellcode that resolves kernel32.dllthrough the PEB

  14. Runtime Heuristics - Process Environment Block • Condition P2. • PEB_LDR_DATA structure- holds the list of loaded modules • (P2): the linear address of PEB.LoaderDatais read.

  15. Runtime Heuristics - Process Environment Block • Condition P3. • Walk through theloaded modules list and locate the second entry (kernel32.dll) • (P3): the linear address of any of theFlinkor Blinkpointers in the one of the three list records of the PEB_LDR_DATA structure isread.

  16. Runtime Heuristics - Backwards Searching • Structured Exception Handling (SEH) - provides a unified way of handling hardware and software exceptions

  17. Runtime Heuristics - Backwards Searching

  18. Runtime Heuristics - Backwards Searching • Condition B1. • (i) any of the linear address between FS:[0]–FS:[0x8]is read • (ii) the current or any previous instruction involved theFS register.

  19. Runtime Heuristics - Backwards Searching • Condition B2. • (B2): the linearaddress of the Handlerfield of the default SEH handler is read.

  20. Runtime Heuristics - Backwards Searching • Condition B3. • (B3): at least one memory read formthe address space of kernel32.dll

  21. Runtime Heuristics – Process Memory Scanning • Egg-hunt shellcode

  22. Process Memory Scanning-SEH • Installing a custom exception handler that is invoked in case of a memory accessviolation • Create a new SEH frame and adjust the currentSEH frame pointer of the TIB to point to it • Directly modify the Handlerpointer of the current SEH frame to point to theattacker's handler routine

  23. Process Memory Scanning-SEH • Condition S1. • (i) the linear addressof FS:[0]is read or written • (ii) the current or any previousinstruction involved the FSregister.

  24. Process Memory Scanning-SEH • Condition S2. • (S2): thelinear address of the Handlerfield in the custom SEH frame is orhas been written

  25. Process Memory Scanning-SEH • Condition S3. • (S3): starting from FS:[0], all SEH frames should reside on the stack, andthe Handlerfield of the last frame should be set to 0xFFFFFFFF

  26. Process Memory Scanning–System Call • Another way to safely scanning the process address space is to check whether a page is mapped—before actually accessing it—using a system call • Some Windows system calls accept as an argument a pointer toan input parameter. If the supplied pointer is invalid, the system callreturns with a return value of STATUS_ACCESS_VIOLATION.

  27. Process Memory Scanning–System Call

  28. Process Memory Scanning–System Call • Condition C1. • (C1): the execution of an int 0x2einstruction withthe eaxregister set to one of the following values: NtAccessCheckAndAuditAlarm(0x2), NtAddAtom (0x8), NtDisplayString(0x39 in Windows 2000, 0x43 in XP, 0x46in 2003 Server, and 0x7F in Vista)

  29. Process Memory Scanning–System Call • Condition C2. • (C2): (C{N}):C1 holds true N times

  30. Runtime Heuristics –SHE-based GetPC Code • The most widely used types of GetPCcode for this purpose rely on some instruction from the call orfstenv instruction groups • The shellcode can register a custom exception handler, triggeran exception

  31. Implementation • Gene, a network-level attack detector that uses a custom IA-32 emulator to identify the presence of shellcode in network streams • Scan the client-initiated part of each TCP connection using the runtime heuristics • The virtual memory of the emulator is initialized with an image ofthe complete address space of a typical Windows XP process takenfrom a real system

  32. Experimental Evaluation – Detection Effectiveness

  33. Experimental Evaluation – Heuristic Robustness • False Positives Evaluation-using a large and diverse set of benign inputsto test • Heuristic Analysis

  34. Experimental Evaluation – Runtime Performance • Running on a system with a Xeon 1.86GHz processor and 2GB of RAM

  35. Experimental Evaluation – Real-world Deployment • Deployed Gene in two University networks, where it has been operational since 25 November 2009. • As of 17 April 2010, Gene has detected 116,513 code injection attacks against internal and external hosts in these two networks

  36. Conclusion • Present a comprehensive shellcode detection method based on code emulation • Expands the range of malicious code types that can be detected • Detection of plain and metamorphic shellcode • Without any false positives

  37. TIB and Thread Stack

More Related