Automatic Reverse Engineering of Malware Emulators

1. Automatic Reverse Engineering of Malware Emulators Sharif, M., Lanzi, A., Giffin, J., and Lee, W. Georgia Institute of Technology, S&P 2009. Presented by WANG Zhi

2. Outline • Emulation Obfuscation • Emulator Model • Abstract Variable Binding • Identifying Candidate VPC • Identifying Emulation Behavior • Extracting Syntax and Semantics

3. Emulation Obfuscation • The term emulation generally expresses the support of a binary instruction set architecture (ISA) that is different from that supported by CPU. • Jave(JVM), Javascript, Perl, Python…

4. Emulation Obfuscation Using emulation for obfuscation

5. Emulation Obfuscation • Without knowledge of the bytecode language, even the location of bytecodes in the obfuscated program. • The key challenges in analyzing emulation obfuscation are how to extract the bytecode trace and the syntax and semantics of the bytecode instructions from the emulated program trace.

6. Emulator Model A decode-dispatch based emulator

7. Emulator Model • The decode phase fetches opcode. • The dispatch phase selects appropriate handling routine. • The execution phase perform the dispatched handling routine

8. Reverse Engineering of Emulation • In this paper, they developed algorithms to extract the syntax and semantics of unknown bytecode based upon the execution behavior of the decode-dispatch emulator.

9. Abstract Variable Binding • The compiler translate source code into low-level CPU instructions while the high-level variables are assigned to memory location or registers. • In the x86 instruction trace, there are no high-level language variables. • To identify variables in the execution trace of the emulator is the first work.

10. Abstract Variable Binding • In this paper, memory locations are used to represent high-level variables, which are called abstract variables.. • In the x86 architecture, memory read and write operations should use register indirect addressing. • Source Code: instruction = bytecode[VPC] • Pseudo-x86 Instructions: eax <= [VPC] instruction <= [eax]

11. Abstract Variable Binding • The first instruction loads the value of the abstract variable VPC to eax. In other word , binds eax to VPC, and this binding is propagated in the next instruction. • It use a forward and backward data-flow analysis to identify the abstract variables and their propogation.

12. Forward Binding Forward binding identifies abstract variable from memory read operations.

13. Forward Binding Forward binding propagates abstract variable bindings when operations compute an update to its previous value. The outputs is a mapping describing bindings from memory read operations to abstract variables.

14. Backward Binding • Backward binding operates in the reverse order of forward bindings. It determines abstract variables from memory write operations

15. Identifying Dependent Abstract Variables • It identify dependencies among abstract variables by tracking the data flow from one abstract variable to another.

16. Lifetime Abstract Variables • It use memory locations as abstract variables, but the same memory location may be used for different variables at different points of execution. • The abstract variables on the stack or heap have shorter lifetime than those in the static data region • The Lifetime depends on the allocation and deallocation operations.

17. Identifying Candidate VPCs • VPC is a virtual program counter, like program counter or instruction pointer register. • The emulator fetches bytecode instructions from memory address specified by VPC

18. Identifying Candidate VPCs • It clusters all read operations to n clusters using a simple similarity metric that check whether they have common abstract variables. • The fetching behavior’s read operations should be contained within one of the n cluster because VPC is the common abstract variable.

19. Identifying Emulation Behavior • Decode-dispatch emulators have fundamental execution properties: a main loop with a bytecode fetch through the VPC, decoding of the opcode within the bytecode, dispatch to an opcode handler, and a change to the VPC value.

20. Identifying Emulation Behavior • They use a standard loop detection methods to detect the emulator’s main loop. • They analyze the abstract variable propagation to find decoding, dispatching and execution of bytecode behaviors.

21. Identifying Emulation Behavior • They use multi-level dynamic tainting to analyze the data flow. • There are four taint label: fetch, decode, dispatch and execute.

22. Identifying Emulation Behavior • If a abstract variable is read from memory, they mark the instruction taint label fetch. • Once the analyzer detects a dispatch-like behavior which is a control-flow transfer instruction and its target address is a taint variable, it will mark the instruction taint label dispatch, and mark the target of the contol-transfer as a probable execute routine

23. Extracting Syntax and Semantics • Once the analyzer identifies the emulation behavior, it reverse engineers each iteration of the emulator loop to extract the syntax and semantics of the bytecode instruction executed on that iteration.

24. Extracting Syntax and Semantics • The syntactic information shows how bytecode instructions are parsed into opcodes and operands. • The semantic information consisting of native instructions that carry out the actions of the bytecode instructions.

25. Extracting Syntax and Semantics • To identify the opcode part, they use taint analysis to determine which portion of the bytecode instruction was used for dispatch behavior. • The execution routine invoked by emulator for the opcode encodes the semantics.

26. Extracting Syntax and Semantics • By determining how the parse the bytecode and by locating control-flow transfer opcodes, a CFG for the bytecode can be constructed. • The CFG structure provides a foundation for subsequent malware analysis.

27. Thank you