Detection of Injected, Dynamically Generated, and Obfuscated Malicious Code (DOME)

1. Detection of Injected, Dynamically Generated, and Obfuscated Malicious Code (DOME) Subha Ramanathan & Arun Krishnamurthy Nov 15, 2005

2. Presentation Outline • Introduction to Malicious Code and DOME • Areas of Coverage • Detection Techniques • Proof of Concepts Study • Deployment Options • Related Work • Conclusion

3. Introduction to Malicious Code (MC) • Definition • Any type of code that can potentially harm computers and networks. • Classes of MC • Injected Code – Code that is introduced into a process’ address space at runtime. • Worms using buffer-overflow attacks • Dynamically Generated Code – Code that is created by a process at runtime. • Polymorphic Viruses • Obfuscated Code – Code that is present in the process original code but whose true intentions are hidden with obscure calculations and data manipulation.

4. Introduction to DOME • A host-based technique for detecting several general classes of malicious code in software executables. • Uses static analysis to identify the locations of system calls within the software executables, and monitors the executables at runtime to verify the location. • Simple to understand and implement.

5. Limitations of DOME • Cannot detect unobfuscated viruses and trojans whose code is embedded in the software executable, prior to the pre-processing by DOME • Limited to executable MC that uses Win32 functions • Does not work for worms that spread using techniques other than code injection, such as script-based worms or worms that infiltrate through drive sharing • So, to ensure full protection from the MC threat, a system based on DOME should be deployed in conjunction with other detection-response systems

6. DOME Areas of Coverage • DOME is designed to detect the three classes of MC. • Five Assumptions • Any code that can be classified in one of the three classes is assumed to be malicious. • MC interacts with the Operating System. • When interacting with the OS, MC uses the Win32 APIs. • If MC hides itself from detection by using dynamic code generation and obfuscation, its Win32 API usage is hidden as well. • Software executables that are to be protected has to be easily disassembled, and their Win32 APIs can be effectively monitored at run time.

7. DOME Detection Technique • Preprocessing • Disassemble software executables and analyze them to identify the instructions that call into Win32 APIs. • Virtual addresses of these intrusions and API names are recorded into an output file. • Monitoring and Detection • Monitor Win32 API calls at runtime. • During the call, identify the instruction that produced the call and the address within the executable. • Validate the instruction address and the API name against the model generated during the preprocessing step. • If it doesn’t match, then signal that a malicious code was detected.

8. Pre-processing Step • Software executables are disassembled and analyzed to identify the instructions that call into Win32 APIs. • The virtual addresses of these instructions and the API names are then recorded. Also record the addresses of instructions that occur immediately after the identified Win32 API calls – these are the return addresses for the Win32 API calls • The identification mechanism should be designed to differentiate between normal compiler-generated code, but none of the Win32 API calls that are intentionally hidden.

9. Monitoring Win32 API calls • Number of methods can be used to monitor the Win32 API calls made by processes. • Proof-of-concept study uses direct patching method implemented by the Detours package • Detours instruments the DLLs containing the Win32 APIs at load time. • By directly patching the entry point of each Win32 API, all Win32 APIs can be monitored.

10. DOME Detection Technique(Monitoring using Wrappers Example) • The process makes a call into the API function (1). • The first instruction is an unconditional jump to the wrapper (2). • The wrapper may execute pre-stub code before returning control to the Win32 API body (3 & 4). • After the API body finishes executing, control is returned to the wrapper (5), which may execute post-stub code before returning to the caller (6). • The pre-stub code is where DOME validates the Win32 API call against the information identified in the preprocessing step.

11. DOME Detection Technique(Handling Bypassability) • DOME Bypassability • MC can forge return address on top of runtime stack, making it appear like the address came from a legitimate program • Solution: Perform runtime stack verification. • MC can use legitimate software’s own instructions and then supply its own malicious arguments. • Solution: Identifying and recording static Win32 API arguments during preprocessing and then validating them at runtime. • Wrapper Bypassability • If a MC detects a wrapper, it can manipulate the memory and disable the wrappers before calling the APIs. • MC can also directly call into the kernel, avoiding the Win32 API calls and wrappers. • Solution: Add a kernel level authentication mechanism that verifies that the APIs are reached only after the execution has passed through the unmodified wrappers.

12. Proof of Concept Study (Goals) • It is possible to identify API calls in real-world software using static analysis. • It is possible to monitor API calls at runtime and to identify the instruction responsible for the observed API calls. • DOME is able to accurately distinguish between the normal code and the malicious code. • Provided that the above 2 assertions are true.

13. Proof of Concept Study (Tools Used) • IDA PRO Disassembler (Preprocessing Step) • Disassembling executables. • Identifying and annotating instructions making Win32 API calls. • Detours Wrapper Package (Monitoring Step) • Monitors Win32 API calls by directly patching each entry point.

14. Proof of Concept Study (Sample Data – Benign)

15. Proof of Concept Study (Sample Data – Malicious)

16. Proof of Concept Study (Sample Output File)

17. Proof of Concept Study (Results) • Successful in determining benign executables. • Did not find any unexpected API calls. • Only false positives noticed were due to dynamic binding of APIs. • Successful in determining malicious executables. • Found Win32 API calls during runtime of executables, but not found during preprocessing. • Gives users better understanding of how malicious codes work. • Even detected the W32-Simile Virus

18. Proof of Concept Study (W32-Simile Virus Detection)

19. DeploymentOptions • Online Detection and Blocking • Offline Software Scanning • Online and Offline Analysis

20. DeploymentOptions(Online Detection and Blocking) • Blocking simple and complex viruses at real-time. • Legitimate code that uses obfuscation can still use API calls by giving DOME early notification. • New software that was not preprocessed can still use API calls by allowing DOME to read the code and use local static analysis to determine whether that API call can be identified.

21. DeploymentOptions(Offline Software Scanning) • Similar to anti-virus scanning. • Preprocesses designated executables, then launches them to see if they produce Win32 API calls that were not identified. • Blocks the Win32 API calls if not identified. • Executives need to be driven through all execution paths. • Not many practical application-independent solutions.

22. DeploymentOptions(Online and Offline Analysis) • Not only detects malicious code, but also pinpoints instructions belonging to it. • Can be used to generate detection signatures, predict propagation vectors, or identify code lineage and perform attribution. • Allows users to understand how malicious code works.

23. Related Work(Types of MC Detection) • Misuse Detection • Attempts to find code that have malicious behaviors. • Anomaly Detection • Determines the normalcy of the behavior. DOME is an anomaly detection technique!!!

24. Conclusion • DOME uses static analysis to identify locations of Win32 API calls within executables. It then uses these locations as a model of which Win32 calls are allowed to occur at run-time. • Very effective in finding injected, dynamically generated, and obfuscated MC. • Future implementation will try to detect MC that bypasses DOME.

25. Works Cited • Jesse C. Rabek, Roger I. Khazan, Scott M. Lewandowski, Robert K. Cunningham, "Detection of injected, dynamically generated, and obfuscated malicious code", Proceedings of the 2003 ACM workshop on Rapid Malcode, Washington, DC, October, 2003, pp. 76-82.