CAP6135: Malware and Software Vulnerability Analysis Rootkits Cliff Zou Spring 2012

1. CAP6135: Malware and Software Vulnerability Analysis RootkitsCliff ZouSpring 2012

2. The Evolution of Malware • Malware, including spyware, adware and viruses want to be hard to detect and/or hard to remove • Rootkits are a fast evolving technology to achieve these goals • Cloaking technology applied to malware • Not malware by itself • Example rootkit-based viruses: W32.Maslan.A@mm, W32.Opasa@mm • Rootkit history • Appeared as stealth viruses • One of the first known PC viruses, Brain, was stealth • First “rootkit” appeared on SunOS in 1994 • Replacement of core system utilities (ls, ps, etc.) to hide malware processes

3. Cloaking • Modern rootkits can cloak: • Processes • Services • TCP/IP ports • Files • Registry keys • User accounts • Several major rootkit technologies • User-mode API filtering • Kernel-mode API filtering • Kernel-mode data structure manipulation • Process hijacking • Visit www.rootkit.com for tools and information

4. Explorer.exe,Winlogon.exe Explorer.exe, Malware.exe, Winlogon.exe User-Mode API Filtering • Attack user-mode system query APIs • Effect: listing system processes cannot see ‘malware.exe’ running process • Con: can be bypassed by going directly to kernel-mode APIs • Pro: can infect unprivileged user accounts • Examples: HackerDefender, Afx Taskmgr.exe Ntdll.dll Rootkit user mode kernel mode

5. Explorer.exe,Winlogon.exe Explorer.exe,Winlogon.exe Explorer.exe, Malware.exe,Winlogon.exe Kernel-Mode API Filtering • Attack kernel-mode system query APIs • Cons: • Requires admin privilege to install • Difficult to write • Pro: very thorough cloak • Example: NT Rootkit Taskmgr.exe Ntdll.dll user mode kernel mode Rootkit

6. Kernel-Mode Data Structure Manipulation • Also called Direct Kernel Object Manipulation (DKOM) • Attacks active process data structure • Query API doesn’t see the process • Kernel still schedules process’ threads • Cons: • Requires admin privilege to install • Can cause crashes • Detection already developed • Pro: more advanced variations possible • Example: FU Explorer.exe Malware.exe Winlogon.exe ActiveProcesses

7. Process Hijacking • Hide inside a legitimate process • Con: doesn’t survive reboot • Pro: extremely hard to detect • Example: Code Red Explorer.exe Malware

8. Detecting Rootkits • All cloaks have holes • Leave some APIs unfiltered • Have detectable side effects • Can’t cloak when OS is offline • Rootkit detection attacks holes • Cat-and-mouse game • Several examples • Microsoft Research Strider/Ghostbuster • RKDetect • Sysinternals RootkitRevealer • F-Secure BlackLight

9. Simple Rootkit Detection • Perform a directory listing online and compare with secure alternate OS boot (see http://research.microsoft.com/rootkit/ ) • Offline OS is Windows PE, ERD Commander, BartPE dir /s /ah * > dirscan.txt windiff dirscanon.txt dirscanoff.txt • This won’t detect non-persistent rootkits that save to disk during shutdown

10. Filtered Windows API omits malware files and keys Malware files and keys are visible in raw scan RootkitRevealer • RootkitRevealer (RKR) runs online • RKR tries to bypass rootkit to uncover cloaked objects • All detectors listed do the same • RKR scans HKLM\Software, HKLM\System and the file system • Performs Windows API scan and compares with raw data structure scan RootkitRevealer Rootkit Windows API Raw file system, Raw Registry hive

11. Demo • HackerDefender • HackerDefender before and after view of file system • Detecting HackerDefender with RootkitRevealer

12. Dealing with Rootkits • Unless you have specific uninstall instructions from an authoritative source: • Don’t rely on “rename” functionality offered by some rootkit detectors • It might not have detected all a rootkit’s components • The rename might not be effective Reformat the system and reinstall Windows!