1 / 23

Stealthy Malware Detection Through VMM-based “ Out-of-the-Box ” Semantic View Reconstruction

CCS’07, Alexandria, VA, Oct 29 – Nov 2, 2007. Stealthy Malware Detection Through VMM-based “ Out-of-the-Box ” Semantic View Reconstruction. Xuxian Jiang , Xinyuan Wang, Dongyan Xu. George Mason University Purdue University. Motivation. Internet malware remains a top threat

verena
Download Presentation

Stealthy Malware Detection Through VMM-based “ Out-of-the-Box ” Semantic View Reconstruction

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CCS’07, Alexandria, VA, Oct 29 – Nov 2, 2007 Stealthy Malware Detection Through VMM-based “Out-of-the-Box” Semantic View Reconstruction Xuxian Jiang, Xinyuan Wang, Dongyan Xu George Mason University Purdue University

  2. Motivation • Internet malware remains a top threat • Malware: viruses, worms, rootkits, spyware, bots…

  3. Motivation • Recent Trend on Rootkits Viruses/worms/bots, PUPs, … 700%growth 400%growth Q1 of 2005 Source: McAfee Avert Lab Report (April 2006)

  4. Existing Defenses (e.g., Anti-Virus Software) • Running inside the monitored system • Advantages • They can see everything (e.g., files, processes,…) • Disadvantages • Once compromised by advanced stealthy malware, they may not see anything! IE Firefox VirusScan … OS Kernel

  5. VirusScan IE Firefox … OS Kernel Existing Defenses • Key observation • Both anti-virus software and vulnerable software are running inside the same system • Hard to guarantee tamper-resistance • Solution: “Out-of-the-box” defense ? Virtual Machine Monitor (VMM)

  6. VirusScan The “Semantic-Gap” Challenge Semantic Gap Guest OS Virtual Machine Monitor (e.g., VMware, Xen, QEMU) • What we can observe? • Low-level states • Memory pages, disk blocks,… • Low-level events • Privileged instructions, • Interrupts, I/O access, … • What we want to observe? • High-level states w/ semantic info. • Files, processes,… • high-level events w/ semantic info. • System calls, context switches, …

  7. Main Contribution • VMwatcher: A systematic approach to bridge the semantic gap • Reconstructing semantic objects and events from low-level VMM observations Capability I: “Out-of-the-box” execution of commodity anti-malware software IE Firefox … VMwatcher OS Kernel Capability II: View comparison-based stealthy malware detection Virtual Machine Monitor (VMM)

  8. VMwatcher: Bridging the Semantic Gap • Step 1: Procuring low-level VM states and events • Disk blocks, memory pages, registers, … • Traps, interrupts, … • Step 2: Reconstructing high-level semantic view • Files, directories, processes, and kernel modules,… • System calls, context switches, … VM Introspection Guest View Casting

  9. Step 1: VM Introspection VM Disk Image VM Physical Memory VM Hardware State (e.g., registers) VM-related low-level events (e.g., interrupts) VMware Academic Program

  10. VirusScan VMwatcher Disk Step 2: Guest View Casting Cross-view Semantic Gap Guest OS Virtual Machine Monitor (VMM) Key observation: The guest OS already contains all necessary semantic definitions of data structures as well as functionalities to construct the semantic view

  11. Guest View Casting Device drivers, file system drivers VM Disk Image Memory translation, task_struct, mm_struct VM Physical Memory VM Hardware State (e.g., registers) CR3, MSR_SYSENTER_CS, MSR_SYSENTER_EIP/ESP Event-specific arguments… Syscalls, Context switches, .... VM-related low-level events (e.g., interrupts) Event semantics Demo clip (3.5mins): http://www.ise.gmu.edu/~xjiang/

  12. Guest View Casting on Memory State (Linux) Process List Process Memory Layout

  13. Guest Memory Addressing • Traditional memory addressing • Given a VA, MMU translates VA to PA • OSes used to map with known PA • Linux: VA 0xc0000000 == PA 0x0 • Windows: VA 0x80000000 == PA 0x0 • VM complicates the translation • Guest virtual -> guest physical • Guest physical -> host physical Emulated Address Translation VM Introspection Reverse Address Translation

  14. Evaluation • Effectiveness • Cross-view malware detection • Exp. I: Cross-view detection on volatile state • Exp. II: Cross-view detection on persistent state • Exp. III: Cross-view detection on both volatile and persistent state • Out-of-the-box execution of commodity anti-malware software • Exp. IV: Symantec AntiVirus • Exp. V: Windows Defender • Performance • Difference between internal scanning & external scanning

  15. Exp. I: Cross-view detection on volatile memory state • Experiment Setup • Guest VM: Windows XP (SP2) • Windows Fu Rootkit • Host OS: Scientific Linux 4.4 • VMM: VMware Server 1.0.1 Diff VMwatcherview “Inside-the-box” view

  16. Exp. II: Cross-view detection on persistent disk state • Experiment Setup • Guest VM: A Redhat 7.2-based honeypot • Linux SHv4 rootkit • Host OS: Windows XP (SP2) • VMM: VMware Server 1.0.1 Diff VMwatcher view “Inside-the-box” view

  17. Experiment (IV) • Experiment Setup • Both guest OS and host OS run Windows XP (SP2) • VMM: VMware Server 1.0.1 • Running Symantec AntiVirus Twice • Outside • Inside Hacker Defender NTRootkit

  18. Internal Scanning Result Diff External Scanning Result

  19. Performance • Internal scanning time vs. external scanning time Internal scanning takes longer to complete !

  20. Related Work • Enhancing security with virtualization (Livewire[Garfinkel03], IntroVirt[Joshi05], HyperSpector[Kourai05]) • Focusing on targeted attacks with specialized IDSes • Cross-view detection (Strider GhostBuster[Wang05], RootkitRevealer/ Blacklight/IceSword/…) • Either destroying the volatile state or obtaining two internal views • Secure monitors • CoPilot[Petroni04], Terra[Garfinkel03], sHype[Sailer05], SecVisor[Perrig07],TRANGO,…

  21. Conclusions • VMwatcher – A systematic approach that bridges the semantic gap and enables two unique malware detection capabilities: • Cross-view malware detection • “Out-of-the-box” execution of commodity anti-malware software

  22. Thank you! For more information: Email:xjiang@ise.gmu.edu URL:http://www.ise.gmu.edu/~xjiang

More Related