Normalizing Metamorphic Malware Using Term Rewriting

1. A. Walenstein, R. Mathur, M. R. Chouchane, and A. Lakhotia Software Research Laboratory The University of Louisiana at Lafayette Sixth IEEE International Workshop on Source Code Analysis and Manipulation 27th-29th September 2006 Philadelphia, PA, USA Normalizing Metamorphic Malware Using Term Rewriting

2. SCAM'06 About this Work The core of the paper's work formed the Master's thesis of Rachit Mathur. He has since graduated and is now working at McAfee.

3. SCAM'06 Malware Identification Malware are malicious programs such as viruses, worms, and Trojans. Virus Form - A Antivirus scanners use extracted patterns, or “signatures” to identify known malware. Anti-Virus Signature Signature

4. SCAM'06 Metamorphic Malware Virus Virus Virus M M Form - A Form - B Form - C • Metamorphic malware change as they propagate • They create multiple variants of themselves

5. SCAM'06 Metamorphic Malware Challenge Virus Virus Virus M M Form - A Form - B Form - C Using different signatures for most variants cannot scale. Anti-Virus Signature Too many signatures challenge the AV Scanner

6. SCAM'06 Proposed approach: normalizer Virus Virus Virus M M Form - A Form - B Form - C N N N Virus Normalizer Construction Problem:Reduce the number of signatures needed to detect all variants. NormalForm Anti-Virus Signature

7. SCAM'06 Inspiration: “undo” transformations push ecx mov ecx, [ebp + 10] mov ecx, ebp push eax add eax, 2342 mov eax, 33 add ecx, eax pop eax mov eax, esi push eax mov esi, ecx push edx xor edx, 778f mov edx, 34 sub esi, edx pop edx mov [esi-2], eax pop esi pop ecx push ecx mov ecx, ebp push eax mov eax, 33 add ecx, eax pop eax push esi mov esi, ecx push edx mov edx, 34 sub esi, edx pop edx mov [esi - 2], eax pop esi pop ecx push ecx mov ecx,ebp add ecx,33 push esi mov esi,ecx sub esi,34 mov [esi-2],eax pop esi pop ecx push ecx mov ecx,ebp add ecx,33 mov [ecx-36],eax pop ecx mov [ebp - 3], eax

8. SCAM'06 Problem 1: “naïve” undo is naïve 1. push ecx mov ecx, 0x04 mov edi, ecx pop ecx mov edi, 0x04 2. push eax mov eax, 0x04 push eax 3. mov eax, 0x04 push eax push 0x04 push 0x04 mov eax, 0x04 mov eax, 0x04 push eax mov eax, 0x04 mov eax, 0x04 push eax push 0x04

9. SCAM'06 Problem 2: conditional transformations mov edi, 0x04 push ecx mov ecx, 0x04 mov edi, ecx pop ecx unconditional push eax mov eax, 0x04 push eax eax not live push 0x04 mov eax, 0x04 push eax eax not live Q: how to reorient rules while guaranteeing termination?

10. SCAM'06 Term rewriting approach Adopted term-rewriting framework Model the metamorphic engine as TRS Modify it to create normalizing rule set and engine apply completion procedure, which reorients rules Can guarantee needed properties (termination, confluence)

11. SCAM'06 Completion procedure sketch push 0x04 mov eax, 0x04 mov eax, 0x04 push eax mov eax, 0x04 Critical Pairs mov eax, 0x04 push eax push 0x04

12. SCAM'06 Completion procedure sketch push 0x04 mov eax, 0x04 mov eax, 0x04 push eax mov eax, 0x04 Reorient New Rule mov eax, 0x04 push eax push 0x04

13. SCAM'06 What to do when completion procedure fails? Successful completion guarantees a unique normal form for all variants: The “perfect” normalizer but Completion procedure may not terminate! Number of rules in the normalizer may be too high to be practical Does not take into account conditions  Need alternative scheme

14. SCAM'06 Priority Scheme Partition N into NU and NC Input Program Normalize w.r.t NU Still Reducible? Simple No Need for costly/imprecise condition evaluation Improved through Ad-hoc completion If possible, Apply a rule from NC Y yes no NU – Unconditional rules NC – Conditional rules HALT

15. SCAM'06 Question: condition checking required? Conditional rules require checking of conditions Can be expensive, or impossible What is the practical penalty of incorrectly checking conditions? e.g., ignoring conditions completely?

16. SCAM'06 Case Study W32.Evol Virus can generate huge number of variants Tested the normalization schemes on 26 variants over 6 generations Manually Extracted rules used by W32.Evol 55 rules 84 overlaps TXL implementations: Ordinary and priority-based evaluation

17. Results

18. SCAM'06 Contributions Applications for assisting malware scanners Initial exploration of possibility of “perfect” normalization Indications of usefulness of heuristic alternatives (priority scheme and ignoring conditions)

19. SCAM'06 Future Work • Expanded scope and empirical study • Extensions for semantics-non-preserving metamorphic engines? • Localized normalization using term rewriting • M. Chouchane and A. Lakhotia “Using Engine Signature to Detect Metamorphic Malware”, Workshop on Rapid Malcode, Fairfax, VA, Nov. 2006 (to appear) • More at www.cacs.louisiana.edu/labs/SRL

20. SCAM'06 • Alumni • Nitin Jyoti, Avertlabs • Aditya Kapoor, McAfee • Erik Uday Kumar,Authentium • Rachit Mathur, McAfee • Moinuddin Mohammed,MicrosoftPrashant Pathak,Symantec • Prabhat Singh,SymantecFunded by: Louisiana Governor’s IT Initiative Software Research Lab Center for Advanced Computer Studies University of Louisiana at Lafayette Arun Lakhotia Director Andrew Walenstein Research Scientist Michael Venable Software Engineer and Alumnus Ph.D. StudentsMohamed R. ChouchaneMd Enamul Karim M.S. Students Christopher Thompson Matthew Hayes