1 / 39

Hunting for Metamorphic Engines

Hunting for Metamorphic Engines. Wing Wong Mark Stamp. In This Paper…. Analyze metamorphic malware Hacker-produced metamorphic code Measure similarity of software Based on n -gram analysis Compute scores Based on n -grams and Based on HMMs This paper is baseline for future work.

coby-anthony
Download Presentation

Hunting for Metamorphic Engines

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Hunting for Metamorphic Engines • Wing Wong • Mark Stamp Hunting for Metamorphic Engines

  2. In This Paper… Hunting for Metamorphic Engines • Analyze metamorphic malware • Hacker-produced metamorphic code • Measure similarity of software • Based on n-gram analysis • Compute scores • Based on n-grams and • Based on HMMs • This paper is baseline for future work

  3. Motivation Hunting for Metamorphic Engines • Many virus construction kits available • Many can produce metamorphic code • So anybody can create “new” version of existing malware • Virtually no technical expertise required • How “effective” is the resulting metamorphic code? • Can we detect metamorphic malware?

  4. Background Hunting for Metamorphic Engines • Encrypted, polymorphic, metamorphic • Metamorphic == body polymorphic • Metamorphic vs cloned software • Clone is the norm, but metamorphic could offer advantages to the good guy too… • From the theory, we know malware detection is NP-complete • And metamorphic is at least as hard • But what about practical situation?

  5. Metamorphism Hunting for Metamorphic Engines • Metamorphic code changes it “shape” • Well-known examples • W95/Regswap • W32/Ghost • W95/Zperm • MetaPHOR

  6. Metamorphism Hunting for Metamorphic Engines • General techniques available • Insertion • Substitution • Transposition • Deletion • Some easier than others • Some more effective against certain detection strategies

  7. Virus Construction Kits Hunting for Metamorphic Engines • In this paper, we consider • PS-MPC (Phalcon/Skism Mass Produced Code generator) • G2 (Second Generation virus generator) • MPCGEN (Mass Produced Code GENerator) • NGVCK (Next Generation Virus Construction Kit) • VCL32 (Virus Creation Lab for Win32)

  8. Virus Construction Kits Hunting for Metamorphic Engines • Did not consider MetaPHOR • Difficult to work with, finicky • All of these claim to be metamorphic • Are they really? • How can we measure “metamorphism”? • If they are highly metamorphic, can we still detect them?

  9. Brief Review of Malware Detection Hunting for Metamorphic Engines • First generation • Signature scanning, wildcards OK • Second generation • Approximate signature scanning; e.g., ignore NOP instructions • Code emulation • Heuristic analysis • Static or dynamic, false positives…

  10. Machine Learning Hunting for Metamorphic Engines • Consider the following • Data Mining, Neural Networks, HMMs • Data Mining • Malware-related previous work • Generic approach • Neural Networks • Previous work based on byte trigrams • Developed and used at IBM

  11. Hidden Markov Models Hunting for Metamorphic Engines • Train HMM on metamorphic family • Then we can score any file to see how “close” it is to the family • What to use to train such an HMM? • Raw bytes in exe? • Disassembled code? • Opcode sequence? • More on this later…

  12. Software Similarity Hunting for Metamorphic Engines • How to quantify metamorphism? • In general, how to measure similarity of software? • Given program 1 and program 2.. • We develop a score • Score of 0 means “no similarity” • Score of 1 means “virtually identical”

  13. N-gram Similarity Hunting for Metamorphic Engines • Given executable files X and Y • Extract opcode sequences from each • Suppose X has nopcodes • Suppose Y has mopcodes • How to compare the sequences? • Many possible ways --- here we use n-gram analysis • That is, we compare subsequences

  14. N-gram Similarity Hunting for Metamorphic Engines • Extracted opcode sequences • X=(x0,x1,…,xn-1) and Y=(y0,y1,…,ym-1) • Compare subsequences of length k • Then xi,xi+1,…,xi+k-1 matches yj,yj+1,…,yj+k-1 if they are the same in any order • For each such match, plot the point (i,j) • Remove any segments less than p points • Then score = (x axis covered + y axis covered) / 2

  15. N-gram Similarity Example Hunting for Metamorphic Engines

  16. N-gram Similarity Hunting for Metamorphic Engines • Score is between 0 and 1 • If program X identical to program Y • Main diagonal is a solid line • And score = 1 • Minimum score is 0 • The smaller the score, the less similar are the programs

  17. Typical N-gram Similarity Hunting for Metamorphic Engines Normal (cygwin utility) files

  18. Typical N-gram Similarity Hunting for Metamorphic Engines NGVCK

  19. Typical N-gram Similarity Hunting for Metamorphic Engines G2

  20. N-gram Similarity Hunting for Metamorphic Engines Compare members of a “family” with each other

  21. N-gram Similarity Hunting for Metamorphic Engines In graphical form…

  22. N-gram Similarity Conclusion? Hunting for Metamorphic Engines • G2 more similar to each other than expected • So, they are not very metamorphic • Ditto for most of the other generators • But, NGVCK viruses more different from each other than expected • So, they are highly metamorphic • Implication wrt signature detection?

  23. NGVCK Similarity Hunting for Metamorphic Engines Compare NGVCK to other families…

  24. NGVCK Similarity Conclusion? Hunting for Metamorphic Engines • NGVCK viruses very different from each other • Implies highly metamorphic… • …so, signature detection will fail • But NGVCK viruses are even more different from normal files • Then what about detection?

  25. Aside: Similar Similarity Measures to Consider? Hunting for Metamorphic Engines • Given opcode sequences • Edit distance • Other sequence comparison techniques • Statistical measures • Considering raw bytes • Statistical measures • Entropy and other “structural” measures

  26. Hidden Markov Models Hunting for Metamorphic Engines Generic view of HMM

  27. HMM Notation Hunting for Metamorphic Engines

  28. HMM for Metamorphic Detection Hunting for Metamorphic Engines • Train HMM • Extract opcodes from family executables • Append opcode sequences • Train a model, i.e., determine matrices • Use trained HMM to score files • Given an file, extract opcode sequence • Score sequence against the model • Compare to predetermined threshold

  29. HMM Scoring: Fine Points Hunting for Metamorphic Engines • Score computed as log likelihood of the scored sequence • We mormalize score to “log likelihood per opcode” (LLPO) • Why? • How to quantify effectiveness? • ROC curves are very useful • Specifically, area under ROC curve (AUC)

  30. Results Hunting for Metamorphic Engines HMM scoring for NGVCK family

  31. HMM Scoring: Bottom Line Hunting for Metamorphic Engines • Signature detection for metamorphic families, except NGVCK • For NGVCK, we can use HMM • Classification is 100% when compared to normal (benign) files • Some misclassifications of other malware (is that good or bad?) • Should include ROC curves, AUC, …

  32. HMM States: 3 State Model Hunting for Metamorphic Engines

  33. N-gram Score Hunting for Metamorphic Engines • Can also score files using N-grams • Randomly select NGVCK file • Extract its opcode sequence • Given a file we want to score • Extract its opcode sequence • N-gram similarity to NGVCK sequence • Higher similarity, classify as NGVCK • Lower similarity, classify as “not NGVCK”

  34. N-gram Score Results? Hunting for Metamorphic Engines • For NGVCK, obtain ideal separation • There exists a threshold for which… • …we can separate NGVCK from normal • Surprisingly strong results • For such a simple similarity score • Why does this work? • We come back to this at the end…

  35. Compare to Commercial AV Hunting for Metamorphic Engines • Tested following on our virus sets • eTrust, avast!, AVG • These scanners detected most of the viruses from weak families • That is, G2, VCL32, etc. • But none of the NGVCK viruses detected by any of the 3 scanners

  36. Conclusion Hunting for Metamorphic Engines HMM effective at detecting a highly metamorphic NGVCK malware family N-gram similarity also effective NGVCK not detected by commercial AV So, this detection improves the state of the art Practical considerations?

  37. Lessons Learned? Hunting for Metamorphic Engines Why can we detect NGVCK family? In spite of high metamorphism, code is statistically different from normal “Improved” metamorphic malware? Metamorphism must be sufficient to evade signature detection But, metamorphic family must be statistically similar to normal

  38. Future Work Hunting for Metamorphic Engines • Build a better metamorphic generator • Some progress here, but still detectable using other detection methods • Still need better generators… • Develop and test other detection strategies • Lots of work done here too • But lots more to do

  39. References Hunting for Metamorphic Engines W. Wong and M. Stamp, Hunting formetamorphic engines, Journal in Computer Virology 2(3):211-229, 2006 M. Stamp, A revealing introduction tohidden Markov models

More Related