SPLIT PERSONALITY MALWARE DETECTION AND DEFEATING IN POPULAR VIRTUAL MACHINES

1. SPLIT PERSONALITY MALWARE DETECTION AND DEFEATING IN POPULAR VIRTUAL MACHINES Alwyn Roshan Pais Alwyn.pais@gmail.com Department of Computer Science & Engineering National Institute of Technology, Karnataka

3. Objective • To study the VM detection techniques in popular Virtual machines. • Develop strategy to counter the detection. • Prevent analysis aware malwares from detecting VM.

4. Plan of Action • Introduction • VM detection techniques • Detection techniques in VMware, VirtualBox and VirtualPC. • Related Work • Prevent Analysis aware malwares from detecting VM. • VMDetectGuard – Tool to mask VM detection : Windows • Optimization of VMDetectGuard • Results

5. Introduction

6. Malware • Malware: It is a collective term for any malicious software which enters the system without the authorization of the user of the system. • Anti-virus/anti-malware products do not guarantee complete protection.

7. Present Scenario • Security researchers use malware analysis tools to build defenses against the unknown malware forms. • They then build patches for the newly discovered vulnerabilities and exploits. • Virtualization has emerged as a very promising technology. • Malware analyst use Virtual Machine Environment (VME), debuggers and sandboxes in their analysis work.

8. Virtualization • A software based representation of a computer that executes programs in the same way as a real computer. • Examples, VMware, Virtual PC, VirtualBox. • Advantages • Reduced capital and operational costs through more efficient use of hardware resources. • Simplifies maintenance . • Improves scalability and deployment agility. • Improves reliability.

9. Benefits of Virtualization to Security Researchers • Researchers can intrepidly execute potential malware samples without having their systems affected. • If a malware destabilizes the OS, analyst just needs to load in a fresh image on a VM. • Reduces time and cost. • Increases productivity.

10. Analysis Awareness Functionality • Malware developers have added a new functionality to malware. • Detect the presence of analysis tools such as VMs, debuggers and sandboxes. • Hide their malicious behavior on detection. • Analysis Aware / Split Personality malware.

11. Related Work • Carpenter (Carpenter et al., 2007) proposes two mitigation techniques. • They aim at tricking the malware by • Changing the configuration settings of the .vmx file present on the host system and, • Altering the magic value to break the guest-host communication channel.

12. Drawbacks of the First Approach • The configuration options break the communication channel between guest and host not just for the program trying to detect the VM, but for all the programs. • Moreover the authors claim that these are undocumented features and that they are not aware of any side effects.

13. Related Work • The work by Guizani (Guizani et al., 2009) provides an effective solution for Server-Side Dynamic Code Analysis. • Small part of the solution deals with tricking the Split Personality malware that employ Memory Detection and VM Communication Channel Detection techniques.

14. Related Work • Kalpa Vishnani et. al. 2011: Masks all the detection techniques used in Vmware.

15. Related Work • Other works concentrate • Detecting this category of malwares • Running in host machine • Save the current state • quickly restore to previous state • Virtual machines in the order of market share • VMware, Virtual PC, and Virtual Box.

16. VM Detection Techniques • Hardware fingerprinting • Registry Check • Process and File Check • Memory Check • Timing Analysis • Communication Channel Check • Invalid Instruction Check

17. Hardware Fingerprinting • Involves looking for specific virtualized hardware. • VMs give an abstracted view of many hardware components. • Querying for such components reveals VM presence. • For Example: BIOS, Motherboard, SCSI Controllers, USB Controllers, etc.

18. Hardware Fingerprinting Results

19. Registry Check • The registry entries contain hundreds of references to the string containing the name of the VM, Ex. “Vmware”, VirtualPC and VirtualBox. • Checking the registry values for certain keys clearly reveals the VM presence.

20. Registry Check • For Example: HKEY_LOCAL_MACHINE\HARDWARE\DEVICEMAP\Scsi\Scsi Port1\Scsi Bus 0\Target Id 0\Logical Unit Id 0\Identifier VMware, VMware Virtual S1.0 HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Class\{4D36E968-E325-11CE-BFC1-08002BE10318}\0000\DriverDesc  VMware SCSI Controller HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Class\{4D36E968-E325-11CE-BFC1-08002BE10318}\0000\ProviderName  VMware, Inc.

21. Process and File Check • Check - VM specific processes and files presence • Eg. • VBoxService.exe : In VirtualBox for synchronization with host • drivers like “vboxhook.dll” and “vpcbus” driver present in %SYSDIR%/drivers

22. Memory Check • This involves looking for values of critical operating system data structures. • These data structures are relocated on a virtual machine so that they do not conflict with the host system's copies. • Store Interrupt Descriptor Table (SIDT), Store Local Descriptor Table( SLDT), Store Global Descriptor Table (SGDT), Store Task Register (STR), Store Machine Status Word (SMSW) • Redpill.exe, ScoopyNG.exe use this method.

23. Timing Analysis • Obvious yet rare attack. • Involves looking at a local Time Stamp Counter (TSC) value. • By noting down the time difference VM presence is detected.

24. VM Communication Channel Check • This check involves detecting the presence of a host-guest communication channel. • IN instruction and a magic number ‘VMXh’ • VmDetect.exe uses this check. • Not applicable to VirtualPC and VirtualBox. • Runs in VMware without exception.

25. Invalid Opcode Check • Specific to VirtualPC • Uses certain opcodes for guest host communication • In host system raise exception and no exception in VirtualPC.

26. Vmware Detection Hardware fingerprinting • hardware details • motherboard serial number, graphics card and network adapter captions • Windows Management Instrumentation (WMI) contains classes • hardware, display, registry etc. • Check for VM specific strings

27. Registry Check • Windows Registry stores • configuration settings • low-level operating system components • Applications running • Check for • Strings like “VirtualPC”, “VBOX”, “VirtualBox” • value that is specific to the corresponding virtual machine being testing on.

28. Process and File Check • Check - VM specific processes and files presence • Eg. • VBoxService.exe : In VirtualBox for synchronization with host • drivers like “vboxhook.dll” and “vpcbus” driver present in %SYSDIR%/drivers

29. Memory Check • involves looking at the values of specific memory locations • STR (Store Task Register) • stores the selector segment of the TR register (Task Register) in the specified operand (memory or other general purpose register). • Value specific in Virtual Machine

30. Invalid Opcode Check • Specific to VirtualPC • Uses certain opcodes for guest host communication • In host system raise exception.

31. Detection of VM running Linux • Techniques: (tested on Vmware) • Hardware Fingerprinting • Dmesg check - prints the message buffer of the kernel • /proc file system check - interface to internal data structures in the kernel. • Communication channel check

32. Dmesg and /proc file system check • Dmesg - prints the message buffer of the kernel • Shows diagnostic message showing presence of hardware during boot • contain strings like “VMware”, • /proc file system - an interface to internal data structures in the kernel • Contains system dependent information

33. Communication Channel Check • IN instruction • Raises exception ““EXCEPTION PRIV INSTRUCTION” in host • Runs in VMware without exception • initiates guest to host communication by calling the “IN” instruction.

34. VMwareDetect • Is the proof of concept tool. • It employs the various VM detection techniques to detect the presence of VMware virtual machine. • Memory Check • VM Communication Channel Check • Hardware Fingerprinting • Registry Check • Timing Analysis

35. VMwareDetect

36. VirtualMachineDetect - VirtualPC • Check using all the methods

37. VirtualMachineDetect - VirtualBox

38. Virtual Machine Detect In VB

39. Remote Detection • Scenario • There is access to the terminal of a system • need not be administrator access • WMIC ( Windows management instrumentation command line) is used

40. Masking Detection of VM • Using PIN API provided by Pin tool. • Can get all the instructions, the arguments and return value • Steps followed for masking • Get each call made by binary. • Check if matches a predefined list of calls. E.g. • RegEnumValueA • Str • LoadLibraryA • __emit

41. Masking Detection of VM • Provide false values if • VM specific values are read (matched from predefined list) • Eg. • Registry read returns the value “VBOX” • Pin Tool gets the return value and modifies it in runtime. • Registry read function returns modified value

42. Masking Detection of VM • Binary does not detect – manipulated value received. • This currently supports • 64 and 32 bit OS • 64 and 32 bit applications

43. Detect if the binary is 64 or 32 bit. Display the detection and give option to user to change it. Load Binary Detect the Underlying VM Detect the OS as 64/32 bit. VirtualBox Virtual PC Instruction Check Masking Register Check Masking Invalid Opcode Check Masking Register Check masking File Check Masking File Check Masking Execution of loaded binary completed Feedback Save to db for further analysis Masking Detection of VM

44. Our Approach

45. Our Approach STEP 1: Maintain a list of all the hardware as well as registry querying API calls. Also maintain a list of all the VM specific instructions such as SIDT, SLDT, SGDT, STR, IN.

46. Our Approach • Following is a partial list of API calls to be monitored. • Hardware Querying APIs • SetupDiEnumDeviceInfo • SetupDiGetDeviceInstanceId • SetupDiGetDeviceRegistryProperty • Registry Querying APIs • RegEnumKey • RegEnumValue • RegOpenKey • RegQueryInfoKeyValue • RegQueryMultipleValues • RegQueryValue

47. Our Approach Step 2: Perform dynamic binary instrumentation of the sample under test in order to obtain its low level information as well as to intercept all the API calls made by it. • We hook into the sample under test by means of .dll injection. • This is achieved using the pin framework.

48. Our Approach Step3: Check to see if the sample under test makes a call or executes any of the monitored API calls or instructions respectively. If a match is found, set the OUTPUT to “Split Personality Malware Detected”. Also, log the activity and provide fake values to the sample so as to make it feel that it is running on a host system.

49. Implementation • Designed, implemented and tested VMDetectGuard. • Implemented in the framework provided by the Pin tool released by Intel Corporation. • Pin is a tool for the instrumentation of programs. • We made use of its framework to intercept the various API calls and low level instructions executed by the sample under test.

50. COUNTERING HARDWARE FINGERPRINTING • Hardware emulation. • APIs that query for BIOS, Motherboard, Processor, Network Adapter. • Ex. VM returns a value “none” for motherboard serial number. VMDetectGuard returns a more appropriate string such as “.16LV3BS.CN70166983G1XF” instead.