1 / 41

“Rooting Out” Rootkits

“Rooting Out” Rootkits. David Taylor & John Lupton ISC Information Security Security-SIG, 15 December 2005. ISC/Information Security. rootkit: (n). A collection of software “tools” - utilities, scripts, data files, etc.

mirra
Download Presentation

“Rooting Out” Rootkits

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. “Rooting Out” Rootkits David Taylor & John Lupton ISC Information Security Security-SIG, 15 December 2005 ISC/Information Security

  2. rootkit: (n) • A collection of software “tools” - utilities, scripts, data files, etc. • Installed on a target computer following compromise (usually remote, but locally possible as well) • Used not only for operations on that machine, but also as a “stash” to retrieve when breaking into other machines ISC/Information Security security@isc.upenn.edu

  3. rootkit (n): (cont.) • Originally a (mostly) Unix/Linux threat • “rooted” (recompiled) versions of common utilities, e.g. ls, ps, netstat • Re-written to hide presence and activity of other rootkit files • Usually cleverly hidden in file system • Windows rootkits have (surprise!) become much more common in recent years • Structure and operation different than U/L rootkits, but still do essesntially the same sorts of things: ISC/Information Security security@isc.upenn.edu

  4. rootkit activity • Hide files, processes, network connections • Wipe logs (“cover your tracks”) • Install backdoors • Sniff networks • Replace binaries and executables • And??…Whatever else the attackers wants! ISC/Information Security security@isc.upenn.edu

  5. rootkit evolution • As operating system kernels have evolved, so have the ways rootkits are written to take advantage • Linux: LKM’s (Loadable Kernel Modules) • A compromised kernel means the machine is “0wn3d” to its very foundation • Windows rootkits often create and install a specialized system driver and configuration files that access API “hooks” allowing attacker to name the process and determine where and how to hide it. ISC/Information Security security@isc.upenn.edu

  6. Example: Hacker Defender Typical configuration file for Hacker Defender: [H<<<idden T>>a/"ble] h"xdef"* r|c<md\.ex<e:: /[/H/idd\en Ser:vi"ces]Ha>:ck"er//Def\ender *[Set/tin/:\gs] / P:assw\ord=hxdef-rulez Ba:ckd:"oor"Shell=hxdef$.exe Fil:eMappin\gN/ame=_.-=[Hacker Defender]=-._ Serv:iceName=HackerDefender100 Se|rvi:ceDisp<://la"yName=HXD Service 100 Ser>vic:eD||escr<ip:t"ion=powerful NT rootkit Dri<ve\rN:ame=HackerDefenderDrv100 D:riv>erFileNam/e=hxdefdrv.sys ISC/Information Security security@isc.upenn.edu

  7. rootkit vs. anti-virus • Sometimes, if the hacker is careless, the rootkit will be caught and quarantined by anti-virus software • For a knowledgeable hacker, this is only a temporary setback • Can usually turn A-V on and off at will • Don’t think deleting it out of quarantine will solve the problem - ”I’ll be back…” (The Terminator) ISC/Information Security security@isc.upenn.edu

  8. Rootkit Detectors • Like anti-virus software and firewalls…useful and effective up to a point • Can detect many well-known, widely distrubuted rootkits • Many rootkits are known only to one person - the one who wrote it ISC/Information Security security@isc.upenn.edu

  9. Rootkit Detectors: Dave T’s Picks Blacklight (Free Beta) http://www.f-secure.com/blacklight/ Free RootkitRevealer http://www.sysinternals.com/Utilities/RootkitRevealer.html Rkdetector http://www.rkdetector.com/ UnHackMe http://www.greatis.com/unhackme/ ISC/Information Security security@isc.upenn.edu

  10. “Be vewwy, vewwy quiet…I’m hunting wootkits” • Before you start: • Have on hand statically re-compiled versions of common operating system utilities • If you suspect the presence of a rootkit, you cannot trust any element of the file system on the machine • You might also want to check the MD5 hash of your “trusted” copies against a pre-written list and/or known good copies on other machines • Keep these trusted utilities on CD-ROM, stored in a secure place until needed • Decide whether this is a “live” patient or an “autopsy” ISC/Information Security security@isc.upenn.edu

  11. “Is it live, or is it Memorex?…” • On a live system, you can check: • Active processes • Open files • Changes in file sizes, attributes and access times • Active network connections • Sniff the network for traffic to and from the “patient” ISC/Information Security security@isc.upenn.edu

  12. “Autopsy” • If circumstances dictate the system be taken down and rebuilt immediately • Use dd or similar utility to make image file • rootkit presence can still be found by examining files, directories, attributes, “metadata”, etc • Can be done post facto, at leisure ISC/Information Security security@isc.upenn.edu

  13. Document what you do • Whether “live” or “autopsy” mode, keep a log of what you do, when you do it and what you find • May come in handy if situation arises again • You may find evidence of a crime • Might not even relate to rootkit, e.g. presence of child pornography ISC/Information Security security@isc.upenn.edu

  14. Check available logs • logins, ssh, sendmail, ftp, http, etc… • On target system, are likely to be wiped, but not always • Many systems configured to use remote logging utilities • “wiped” logs may exist elsewhere • Look for anomalies, e.g.: • user ‘davet’ shows up running ftp sessions, and you know he: • Doesn’t know what ftp is • Isn’t smart enough to use it if he did • Is dead, and you forgot to delete the account ISC/Information Security security@isc.upenn.edu

  15. Check cron jobs • cron runs processes, programs, scripts, etc. at predetermined times and intervals • Similar to Scheduled Tasks in Windows • Typical location is /var/spool/cron • Anything there that looks unfamiliar or suspicious? ISC/Information Security security@isc.upenn.edu

  16. Use ps -auxww or -elf to see what processes are running [root@dobro bin]# ps -elf F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 4 S root 1 0 0 76 0 - 1187 - Nov15 ? 00:00:00 init [5] 1 S root 2 1 0 94 19 - 0 ksofti Nov15 ? 00:00:01 [ksoftirqd/0] 1 S root 3 1 0 65 -10 - 0 worker Nov15 ? 00:00:00 [events/0] 1 S root 4 3 0 71 -10 - 0 worker Nov15 ? 00:00:00 [khelper] 1 S root 5 3 0 74 -10 - 0 worker Nov15 ? 00:00:00 [kacpid] 1 S root 34 3 0 65 -10 - 0 worker Nov15 ? 00:00:00 [kblockd/0] 1 S root 35 1 0 75 0 - 0 hub_th Nov15 ? 00:00:00 [khubd] 1 S root 46 3 0 75 0 - 0 pdflus Nov15 ? 00:00:00 [pdflush] 1 S root 47 3 0 75 0 - 0 pdflus Nov15 ? 00:00:02 [pdflush] 1 S root 49 3 0 67 -10 - 0 worker Nov15 ? 00:00:00 [aio/0] 1 S root 48 1 0 75 0 - 0 kswapd Nov15 ? 00:00:07 [kswapd0] 1 S root 122 1 0 84 0 - 0 serio_ Nov15 ? 00:00:00 [kseriod] 1 S root 192 3 0 65 -10 - 0 worker Nov15 ? 00:00:00 [ata/0] 1 S root 194 1 0 85 0 - 0 - Nov15 ? 00:00:00 [scsi_eh_0] 1 S root 195 1 0 85 0 - 0 - Nov15 ? 00:00:00 [scsi_eh_1] 1 S root 208 3 0 66 -10 - 0 worker Nov15 ? 00:00:00 [kmirrord] 1 S root 209 3 0 66 -10 - 0 worker Nov15 ? 00:00:00 [kmir_mon] 1 S root 217 1 0 75 0 - 0 kjourn Nov15 ? 00:00:10 [kjournald] 1 S root 1134 1 0 75 0 - 0 - Nov15 ? 00:00:00 [khpsbpkt] 0 S root 1141 1 0 69 -10 - 900 - Nov15 ? 00:00:00 udevd 1 S root 1569 1 0 76 0 - 0 - Nov15 ? 00:00:00 [knodemgrd_0] 1 S root 2009 3 0 66 -10 - 0 kaudit Nov15 ? 00:00:00 [kauditd] 1 S root 2060 1 0 75 0 - 0 kjourn Nov15 ? 00:00:00 [kjournald] 1 S root 2563 1 0 76 0 - 906 - Dec11 ? 00:00:03 syslogd -m 0 5 S root 2567 1 0 76 0 - 633 syslog Nov15 ? 00:00:00 klogd -x 5 S rpc 2586 1 0 75 0 - 1186 - Nov15 ? 00:00:00 portmap 5 S rpcuser 2606 1 0 81 0 - 1449 - Nov15 ? 00:00:00 rpc.statd 1 S root 2639 1 0 76 0 - 4963 - Nov15 ? 00:00:00 rpc.idmapd 1 S root 2710 1 0 79 0 - 634 - Nov15 ? 00:00:00 /usr/sbin/acpid 5 S ntp 2804 1 0 76 0 - 4638 - Nov15 ? 00:00:00 ntpd -u ntp:ntp -p ISC/Information Security security@isc.upenn.edu

  17. lsof can show which files a process has open… [root@dobro bin]# lsof -p 2563 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME syslogd 2563 root cwd DIR 253,0 4096 2 / syslogd 2563 root rtd DIR 253,0 4096 2 / syslogd 2563 root txt REG 253,0 37992 5144639 /sbin/syslogd syslogd 2563 root mem REG 253,0 105080 42975258 /lib64/ld-2.3.4.so syslogd 2563 root mem REG 253,0 1489097 42975260 /lib64/tls/libc-2.3.4.so syslogd 2563 root mem REG 253,0 56791 42975257 /lib64/libnss_files-2.3.4.so syslogd 2563 root 0u unix 0x000001012afb0e00 5893 /dev/log syslogd 2563 root 2w REG 253,0 430682 30002610 /var/log/messages syslogd 2563 root 3w REG 253,0 0 30002360 /var/log/secure syslogd 2563 root 4w REG 253,0 1248 30002361 /var/log/maillog syslogd 2563 root 5w REG 253,0 190062 30002364 /var/log/cron syslogd 2563 root 6w REG 253,0 0 30002362 /var/log/spooler syslogd 2563 root 7w REG 253,0 0 30002363 /var/log/boot.log ISC/Information Security security@isc.upenn.edu

  18. Check ifconfig In most installations, the running mode is normally MULTICAST… eth0 Link encap:Ethernet HWaddr 00:12:3F:64:7A:DA inet addr:192.168.0.2 Bcast:255.255.255.255 Mask:255.255.255.0 inet6 addr: fe80::212:3fff:fe64:7ada/64 Scope:Link UP BROADCAST RUNNING PROMISCUOUS MTU:1500 Metric:1 RX packets:247288 errors:0 dropped:0 overruns:0 frame:0 TX packets:382125 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:143347639 (136.7 MiB) TX bytes:59167774 (56.4 MiB) Base address:0xdcc0 Memory:dfee0000-dff00000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:8626 errors:0 dropped:0 overruns:0 frame:0 TX packets:8626 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:6840040 (6.5 MiB) TX bytes:6840040 (6.5 MiB) …there may be a sniffer running on eth0 ISC/Information Security security@isc.upenn.edu

  19. Use netstat to check network connections [root@dobro bin]# netstat -atup Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 *:32769 *:* LISTEN tcp 0 0 *:mysql *:* LISTEN tcp 0 0 *:netbios-ssn *:* LISTEN tcp 0 0 *:sunrpc *:* LISTEN tcp 0 0 *:auth *:* LISTEN tcp 0 0 localhost.localdomain:ipp *:* LISTEN tcp 0 0 192.168.0.2:ipp *:* LISTEN tcp 0 0 *:microsoft-ds *:* LISTEN tcp 0 1728 192.168.0.2:ssh 24.168.97.666:35424 ESTABLISHED udp 0 0 *:32768 *:* udp 0 0 172.16.213.1:netbios-ns *:* udp 0 0 172.16.245.1:netbios-ns *:* udp 0 0 192.168.0.2:netbios-ns *:* udp 0 0 *:netbios-ns *:* udp 0 0 172.16.213.1:netbios-dgm *:* udp 0 0 172.16.245.1:netbios-dgm *:* udp 0 0 192.168.0.2:netbios-dgm *:* udp 0 0 *:netbios-dgm *:* udp 0 0 *:662 *:* udp 0 0 *:bootpc *:* udp 0 0 *:sunrpc *:* udp 0 0 *:ipp *:* udp 0 0 192.168.0.2:ntp *:* udp 0 0 localhost.localdomain:ntp *:* udp 0 0 *:ntp *:* raw 0 0 *:icmp *:* ISC/Information Security security@isc.upenn.edu

  20. Inodes • Key part of file system “metadata” structure • Sequentially numbered “container” that contains file name, permissions, and location(s) in file system (i.e., disk) • Term “inode” most commonly applied to Unix/Linux, but same principle used in Windows/NTFS ISC/Information Security security@isc.upenn.edu

  21. Inode numbering [root@dobro bin]# ls -li l* 16433200 -rwxr-xr-x 1 root root 20088 Jun 20 07:45 link 16433242 -rwxr-xr-x 1 root root 31880 Jun 20 07:45 ln 16433221 -rwxr-xr-x 1 root root 100952 Jun 15 2004 loadkeys 16433184 -rwxr-xr-x 1 root root 28024 Sep 14 04:48 login 16433231 -rwxr-xr-x 1 root root 87608 Jun 20 07:45 ls [root@dobro bin]# ls -li /home/lupton/*.* 30409514 -rw-rw-r-- 1 lupton lupton 987 Dec 14 12:16 /home/lupton/ifconfig.txt 30409513 -rw-rw-r-- 1 lupton lupton 581 Dec 14 12:16 /home/lupton/ifconfig.txt~ 30409511 -rw-rw-r-- 1 lupton lupton 1160 Dec 14 12:15 /home/lupton/lsof-p2563.txt 30409274 -rw-rw-r-- 1 lupton lupton 0 Oct 31 11:52 /home/lupton/mandolin.iso 30409510 -rw-rw-r-- 1 lupton lupton 1746 Dec 14 12:10 /home/lupton/netstat-a--inet.txt 30409327 -rw-rw-r-- 1 lupton lupton 4535 Dec 14 11:46 /home/lupton/pself.txt 30408835 -rw-rw-r-- 1 lupton lupton 1656 Nov 21 10:49 /home/lupton/upd.txt ISC/Information Security security@isc.upenn.edu

  22. Inode behavior • Inode number remains same when file is edited • If file is deleted, and new file with same name is written to disk, will usually retain Inode number • Inode number changes when file is replaced or overwritten by new file ISC/Information Security security@isc.upenn.edu

  23. MAC Times • “MAC”: “Modified/Accessed/Changed” • “M-time”: date/time file contents last modified • “A-time”: date/time file was last accessed • “C-time”: date/time inode information last changed (chmod, new blocks written, defragmentation, etc.) ISC/Information Security security@isc.upenn.edu

  24. MAC Implications • Hackers like to use rootkits to hide things AND plant “rooted” versions of standard binaries • If the M- or C-times of standard utilities (e.g. ls, ps, netstat) have been altered, it may indicate a bogus version • Similarly, if the inode number appears to have changed, it may be a “rooted” version ISC/Information Security security@isc.upenn.edu

  25. Follow the changing inode… [root@dobro bin]# ls -li /home/lupton/inode_test.txt 30409516 -rw-rw-r-- 1 lupton lupton 77 Dec 14 13:24 /home/lupton/inode_test.txt [root@dobro bin]# cat /home/lupton/inode_test.txt This is the first file, before deletion. To be saved as "inode_test.txt"... [root@dobro bin]# rm /home/lupton/inode_test.txt rm: remove regular file `/home/lupton/inode_test.txt'? Y Now, I write and save a new file with the same name… [root@dobro bin]# ls -li /home/lupton/inode_test.txt 30409516 -rw-rw-r-- 1 lupton lupton 56 Dec 14 13:27 /home/lupton/inode_test.txt [root@dobro bin]# cat /home/lupton/inode_test.txt This is the second version, after deleting the first... Next, we overwrite the second version with another file… [root@dobro bin]# mv /home/lupton/inode_bogus.txt /home/lupton/inode_test.txt mv: overwrite `/home/lupton/inode_test.txt'? y [root@dobro bin]# ls -li /home/lupton/inode_test.txt 30409285 -rw-rw-r-- 1 lupton lupton 28 Dec 14 13:32 /home/lupton/inode_test.txt [root@dobro bin]# cat /home/lupton/inode_test.txt This is the "bogus" version [root@dobro bin]# ISC/Information Security security@isc.upenn.edu

  26. MAC Timeline • Almost always a major part of a full forensic examination • Correlates filenames and the dates/times their M, A and/or C were altered • Usually lengthy and time consuming to look through, but can often reveal exactly when and how a rootkit was installed ISC/Information Security security@isc.upenn.edu

  27. Basic MAC evaluation using ls • List inode # with M-time: ls -li • List inode # with A-time: ls -luti • Long listing with C-time: ls -lci ISC/Information Security security@isc.upenn.edu

  28. ls -li, -luti, -lci [root@dobro bin]# ls -li l* 16433200 -rwxr-xr-x 1 root root 20088 Jun 20 07:45 link 16433242 -rwxr-xr-x 1 root root 31880 Jun 20 07:45 ln 16433221 -rwxr-xr-x 1 root root 100952 Jun 15 2004 loadkeys 16433184 -rwxr-xr-x 1 root root 28024 Sep 14 04:48 login 20767391 -rwxr-xr-x 1 root root 90654 Aug 4 2005 ls [root@dobro bin]# ls -luti l* 16433231 -rwxr-xr-x 1 root root 87608 Dec 14 13:20 ls 16433200 -rwxr-xr-x 1 root root 20088 Dec 9 04:02 link 16433242 -rwxr-xr-x 1 root root 31880 Dec 9 04:02 ln 16433221 -rwxr-xr-x 1 root root 100952 Dec 9 04:02 loadkeys 16433184 -rwxr-xr-x 1 root root 28024 Dec 9 04:02 login [root@dobro bin]# ls -lci l* 16433200 -rwxr-xr-x 1 root root 20088 Nov 4 12:27 link 16433242 -rwxr-xr-x 1 root root 31880 Nov 4 12:25 ln 16433221 -rwxr-xr-x 1 root root 100952 Nov 4 12:24 loadkeys 16433184 -rwxr-xr-x 1 root root 28024 Nov 4 12:25 login 16433231 -rwxr-xr-x 1 root root 87608 Nov 4 12:22 ls ISC/Information Security security@isc.upenn.edu

  29. What’s wrong with this picture? [root@dobro bin]# ls -a . dd igawk nisdomainname tar .. df ipcalc pgawk tcsh .. dmesg kbd_mode ping touch alsaunmute dnsdomainname kill ping6 tracepath arch doexec ksh ps tracepath6 ash domainname link pwd traceroute ash.static dumpkeys ln red traceroute6 aumix-minimal echo loadkeys rm true awk ed login rmdir umount ISC/Information Security security@isc.upenn.edu

  30. Take a closer look… [root@dobro]# ls -a . dd igawk nisdomainname tar .. df ipcalc pgawk tcsh .. dmesg kbd_mode ping touch How can there be two ‘..’ directories?… ISC/Information Security security@isc.upenn.edu

  31. How did this happen? [root@dobro bin]# mkdir ..\ This is actually: mkdir <space><dot><dot><backslash><space><enter> It creates a directory actually named “dot-dot-space” ISC/Information Security security@isc.upenn.edu

  32. What’s in this “mystery” directory? [root@dobro bin]# cd ..\ [root@dobro .. ]# ls -l total 24 -rw-r--r-- 1 root root 0 Dec 15 12:19 rootkit_file_01 -rw-r--r-- 1 root root 0 Dec 15 12:19 rootkit_file_02 -rw-r--r-- 1 root root 0 Dec 15 12:19 rootkit_file_03 -rw-r--r-- 1 root root 0 Dec 15 12:19 rootkit_file_04 -rw-r--r-- 1 root root 0 Dec 15 12:19 rootkit_file_05 -rw-r--r-- 1 root root 0 Dec 15 12:19 rootkit_file_06 ISC/Information Security security@isc.upenn.edu

  33. What happens when… [root@dobro bin]# mkdir ..\ \ \ \ \ (i.e., dot-dot with 5 spaces) ISC/Information Security security@isc.upenn.edu

  34. What does ls show? [root@forensic_laptop bin]# ls -a . date hostname nice sync .. dd igawk nisdomainname tar .. df ipcalc pgawk tcsh .. dmesg kbd_mode ping touch Yet another ‘..’ directory… ISC/Information Security security@isc.upenn.edu

  35. Anything in it? [root@dobro bin]# cd ..\ \ \ \ \ [root@dobro .. ]# ls -l total 24 -rw-r--r-- 1 root root 0 Dec 15 12:23 evilroot_01 -rw-r--r-- 1 root root 0 Dec 15 12:23 evilroot_02 -rw-r--r-- 1 root root 0 Dec 15 12:23 evilroot_03 -rw-r--r-- 1 root root 0 Dec 15 12:23 evilroot_04 -rw-r--r-- 1 root root 0 Dec 15 12:23 evilroot_05 -rw-r--r-- 1 root root 0 Dec 15 12:23 evilroot_06 ISC/Information Security security@isc.upenn.edu

  36. Where do you find rootkit files? • Sometimes, right under your nose • Watch out for “stupid tricks” with ‘..’ directories and filenames beginning with ‘.’ • Use both -l and -a flags when using ls • -l does not list ‘.’ and ‘..’ entries: [root@forensic_laptop bin]# ls -l total 6864 -rwxr-xr-x 1 root root 15528 Jul 19 07:34 alsaunmute -rwxr-xr-x 1 root root 2812 Sep 14 04:52 arch -rwxr-xr-x 1 root root 98356 Jun 15 2004 ash -rwxr-xr-x 1 root root 522116 Jun 15 2004 ash.static -rwxr-xr-x 1 root root 12964 Jun 15 2004 aumix-minimal lrwxrwxrwx 1 root root 4 Oct 31 13:10 awk -> gawk -rwxr-xr-x 1 root root 13068 Jun 20 07:52 basename ISC/Information Security security@isc.upenn.edu

  37. Look for things that are odd, out of place, or you don’t recognize • Hackers usually aren’t going to name the files “rootkit_01”, etc. • They have an entire file system to hide them in ISC/Information Security security@isc.upenn.edu

  38. OK, I found a rootkit - or I’m pretty sure there’s one in there somewhere… What do I do?? ISC/Information Security security@isc.upenn.edu

  39. OK, I found a rootkit - or I’m pretty sure there’s one in there somewhere… What do I do?? REBUILD!!! ISC/Information Security security@isc.upenn.edu

  40. You cannot trust a system that has a rootkit installed. Rebuilding is not just the best option… IT’S THE ONLY OPTION See: www.microsoft.com/technet/community/columns/secmgmt/sm0504.mspx By Jesper Johansson (Microsoft Security Manager) ISC/Information Security security@isc.upenn.edu

  41. Questions? Comments? ISC/Information Security security@isc.upenn.edu

More Related