P rocedures in Intrusion Investigation of a UNIX/Linux Host

1. Procedures in Intrusion Investigation of a UNIX/Linux Host CS-585F: Computer-Related Law and Computer Forensics Fall 2002 By Yu-Li Chen Rajesh Menon Joe Meslovich

2. Introduction • UNIX does not have a good reputation for reliability or security (Gollmann, 1999). • UNIX systems directly connected to the Internet are often subject to hacking attempts (Kruse & Heiser, 2002). • The skill and knowledge to investigate a compromised UNIX system becomes essential for both UNIX users and forensic investigators.

3. Approach • Looking for signs of a compromised UNIX system • Preserving and gathering evidence • Collecting information and seizing the data • Analyzing the file system • Examining signs of intrusion

4. Signs of a compromised UNIX system 1. Examine log files 2. Look for setuid and setgid files 3. Check the system binaries 4. Look for signs of a network sniffer 5. Examine files run by cron and at 6. Check for unauthorized services 7. Examine /etc/passwd file 8. Check system and network configuration 9. Look every where for unusual or hidden files 10. Examine all machines on the local network

5. Examine log files • Refer to /etc/syslog.conf to see how logging is configured for each system service. The default location for these log file is the /var/log directory. • messages log - look for anomalies • xferlog will contain log files for all of the ftp transfers • utmp contains binary information for every user currently logged in-use whoor w to access this file • wtmp is modified every time a user successfully logs in, logs out, or your machine reboots-use lastto get information • secure is used by some versions of UNIX (RedHat Linux for example) to log tcp wrapper messages -look for anomalies

6. [root@localhost amanda]# more /etc/syslog.conf # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* /var/log/maillog # Everybody gets emergency messages, plus log them on another # machine. *.emerg * # Save mail and news errors of level err and higher in a # special file.

7. [root@localhost amanda]# w /var/log/utmp 1:16pm up 4:04, 2 users, load average: 0.09, 0.04, 0.01 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT [root@localhost amanda]# last /var/log/wtmp wtmp begins Fri Dec 6 22:55:48 2002 [root@localhost amanda]# more /var/log/secure Dec 8 13:17:29 localhost login: ROOT LOGIN ON tty2 Dec 8 13:25:14 localhost login: LOGIN ON tty1 BY amanda Dec 10 10:20:52 localhost login: ROOT LOGIN ON tty2

8. Look for setuid and setgid files • Intruders often leave setuid copies of /bin/sh or /bin/time around to allow them root access at a later time • The UNIX find program can be used to hunt for setuid and/or setgid files • find / -user root –perm –4000 –print • find / -group kmem –perm –2000 –print

9. [root@localhost amanda]# find / -user root -perm -4000 -print find: /mnt/cdrom: Input/output error find: /proc/6/fd: Permission denied find: /proc/1376/fd/4: No such file or directory /bin/mount /bin/umount /bin/su /bin/ping /sbin/pwdb_chkpwd /usr/X11R6/bin/Xwrapper /usr/X11R6/bin/imwheel-solo /usr/X11R6/bin/xlock /usr/bin/chage /usr/bin/gpasswd /usr/bin/chfn /usr/bin/chsh /usr/bin/newgrp /usr/bin/crontab /usr/bin/passwd

10. Check the system binaries • Intruders may change program on UNIX systems such as login, su, telnet, netstat, ifconfig, ls, find du, df, libc, sync • Compare the versions on your systems with known good copies • Trojan horse programs may produce the same standard checksum and timestamp as the legitimate version • Standard UNIX sum command and the timestamps associate with the programs are not sufficient • The use of cmp, MD5, Tripwire, and other cryptographic checksum tools is sufficient

11. Look for signs of a network sniffer • Intruders may install a network monitoring program, commonly called a sniffer (or packet sniffer) • A sniffer could be installed on your system if any process currently has any of your network interfaces in promiscuous mode • Some legitimate network monitors and protocol analyzers will set a network interface in promiscuous mode

12. Examine files run by cron and at • The cron and at commands are used to schedule commands and processes that repeat at specified intervals • Intruders may leave back doors in files run from cron or submitted to at • These techniques can let an intruder back on the system • Verify that all files/programs referenced (directly or indirectly) by the cron and at jobs • at -l • crontab -l

13. Check for unauthorized services • Inspect /etc/inetd.conf or /etc/xinetd.conf for unauthorized additions or changes • Search for entries that execute a shell program(for example, /bin/sh or /bin/csh • Check for legitimate services that you have commented out in your /etc/inetd.conf • Determine if intruders have turned on a service or replaced the inetd program with a Trojan horse program.

14. Examine the /etc/passwd file • Check the /etc/passwd file on the system for modifications • Look for the unauthorized creation of new accounts, accounts with no passwords, or UID changes (especially UID 0) to existing accounts • Use stat /etc/passwd to look at access and modification information • Use cat /etc/passwdto show the contents of the file

15. [root@localhost amanda]# more /etc/passwd root:x:0:0:root:/root:/bin/bash bin:x:1:1:bin:/bin: daemon:x:2:2:daemon:/sbin: adm:x:3:4:adm:/var/adm: lp:x:4:7:lp:/var/spool/lpd: sync:x:5:0:sync:/sbin:/bin/sync shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown halt:x:7:0:halt:/sbin:/sbin/halt mail:x:8:12:mail:/var/spool/mail: news:x:9:13:news:/var/spool/news: uucp:x:10:14:uucp:/var/spool/uucp: operator:x:11:0:operator:/root: games:x:12:100:games:/usr/games:

16. Check system and network configuration • Find any unauthorized entries, especially ‘+’(plus sign) entries on the system • Find inappropriate non-local host names in /etc/hosts • Confirm that these files existed prior to any intrusion and were not created by the intruder.

17. Look for unusual or hidden files • Files that start with a period . and are normally not shown by ls can be used to hide tools and information • A common technique on UNIX systems is to put a hidden directory in a user’s account with an unusual name • The find program can be used to look for hidden files • For example: • find / -name “..’’ –print –xdev • find / -name “.’’ –print –xdev | cat –v

18. Examine all machines on the local network • If one host has been compromised, others on the network have been, too.

19. Before collecting evidence…. • Keep a regular old notebook handy and take careful notes of what you do during your investigation. • Start with the script command to ensure that everything you do is captured in a file. • script investigation.txt • date • uname –a • use your own clean tools CD (if you have one) to read all the files that you are going to investigate • mount -t iso9660 /dev/cdrom /mnt/cdrom • PATH = /mnt/cdrom • echo $PATH • /mnt/cdrom

20. Ways to preserve evidence • CPU storage • Video • System memory • Network connections • Running processes

21. CPU storage • a.Registers: Infeasible to capture and of minimal utility. • b.Caches: Infeasible to capture as a discrete entity but should be captured as part of system memory image • When UNIX systems are shutdown with the shutdown command, all services are clearly shutdown and cached file system buffers are flushed, data is written to disc and can be captured as part of system memory image.

22. Video • To save a copy of the screen: • 1. Press Ctrl-Alt-F2 ( on a RedHat Linux system) to enter another virtual terminals on the console and log on as root(if you know the password). • 2. Use xwd command to dump an individual window or use –root option to dump entire screen • xwd –display localhost:0 –root > screenshot.xwd • 3. Return to the X display by pressing Ctrl-Alt-F7 • 4. Examine it to make sure that you didn’t capture the screen saver with xwud command • xwud –in screenshot.xwd

23. System memory • Easy to capture, but the act of capturing it changes it • The physical memory on a UNIX system is /dev/mem; the virtual memory is /dev/kmem • use the dd command to copy the memory from the suspect machine • dd if =/dev/mem of =suspect.mem.image • dd if =/dev/kmem of = suspect.kmem.image • It is impossible to verify accuracy when copying such highly volatile

24. Network connections • Network state provides important information on both current network connections and listening processes • Know when attacker has left a network process running or when unauthorized connections are taking place either inbound or outbound • netstat –p show the processes associated with specific network connections • netstat –rn display the routing tables • arp –v capture the address resolution tables • (date; netstat –p; netstat –rn; arp –v) > suspect.netstatus.txt

25. Running processes • utilities : • ps Display list of all running processes • last, w, who Get listings of logged in users, prior logins, etc • uptime Show current processing load and 2 previous load values • top Password cracking tools will show up clearly with top. • lsof Provide a list of all currently open files • fuser Identifies processes using a specific file or network socket. • strace List all system calls being made by a running process. • truss, ktrace Earlier versions of system call trace. • ltrace Library routine trace.

26. [root@localhost amanda]# ps PID TTY TIME CMD 1339 pts/2 00:00:00 bash 1404 pts/2 00:00:00 ps [root@localhost amanda]# uptime 1:42pm up 4:30, 2 users, load average: 0.24, 0.13, 0.09

27. /proc • Pseudo-filesystem that provides a structured interface to /dev/kmem • No attacks that hide /proc entries have been reported • A more reliable guide to currently running processes than the commonly hacked ps utility • ls –d /proc/[0-9]* >suspect_proc.txt

28. Signs of hostile processes • Any discrepancy betweenps, top, and /proc • Unrecognized commands, especially ones that start with a . or ./ • Daemons running more than once that should be running only once, such as inetd • High uptimes (high system utilization) and especially a process using an unusual level of system resources • High network utilization

29. The scenario • The production server of the ABC company is suspected to be compromised • The precondition imposed on the forensics team --- the server cannot be brought down • Forensics is to be conducted on the live server • MD5 hash of the original disks was not taken

30. Steps of the investigation • 1) Collection of the evidence for the file-system analysis • 2) Investigation into the suspect file-system

31. Steps in the collection of evidence • Using the netcat command • Command on collection system # nc –l –p 10000 > /tmp/suspect.passwd_file • Command on the suspect system # cat /etc/passwd /etc/shadow | nc 192.168.0.2 10000 –w3

32. Steps in the collection of evidence (contd) 1) Log the investigation steps carried out on the suspect system • Command: # script investigation.txt 2) Maintain a journal to enumerate the results of the investigation

33. Steps in the collection of evidence (contd) 3) Verify the date and time of the suspect system and determine the DNS name and the OS version of the system • Commands: # date # uname -a

34. Steps in the collection of evidence (contd) 4) Obtain the screen shots of the suspect system’s desktop • Command: # xwd –display localhost:0 –root > screen.xwd 5) Save the copy of the suspect’s password and shadow files and sent them to collection system • Command: # cat /etc/passwd/etc/shadow | nc 192.168.0.2 10000 –w3

35. Steps in the collection of evidence (contd) 6) Send the copies of the regular and kernel memory to collection system • Commands: # dd bs=1024 < /dev/mem | nc 147.138.69.59 10002 –w3 # dd bs=1024 < /dev/kmem | nc 147.138.69.59 10002 –w3

36. Steps in the collection of evidence (contd) • 7) Send the date, details of processes with network connections, the kernel routing table and the address resolution protocol to collection system • Command: # (date; netstat –p; netstat –rn; arp –v) | nc 147.138.69.59 10002 –w3

37. Steps in the collection of evidence (contd) 8) Commands to verify the state of suspect machine 1. Command to find the time for which the system was running since its last reboot • Command: # uptime 2. Command to find the users remotely connected to the system • Command: # who

38. Steps in the collection of evidence (contd) 3) Check for the processes that take up system resources • Command: # top

39. Steps in the collection of evidence (contd) 9) Information about the running processes sent to the collection system • Command: # (ps –aux; ps –auxeww; lsof) | nc 147.138.69.59 10002 –w3

40. Steps in the collection of evidence (contd) 10) Find the list of running processes on the suspect system and make a tar file to be sent to the collection system • Commands: # ls –d /proc/[0-9]* | nc 147.138.69.59 10002 –w3 # tar –cvpf proc.tar /proc/[0-9]* # nc 147.138.69.59 10003 –w3 < proc.tar

41. Steps in the collection of evidence (contd) 11) Determine the physical devices and partitions on the suspect system • Commands: # mount # fdisk –l /dev/hda

42. Steps in the collection of evidence (contd) 12) Make a data dump of the partitions of the suspect system hard disk and send it to collection system • Commands: # dd if=/dev/hda1 bs = 1024 | nc 147.138.69.59 10002 –w3

43. Steps in the collection of evidence (contd) 13) Making the MD5 hash of the files received and the data dump of the partitions of the suspect hard disk • Command: # md5sum –b suspect.hda1.image

44. The Analysis

45. The Suspect System • Hardware: Dell Optiplex GX1 • Operating System: RedHat Linux 8.0 • Kernel Version: 2.4.18-14 • Primary Use: Web Server (Apache/2.0.40)

46. The Collection System • Hardware: Dell Optiplex GX110 • Operating System: RedHat Linux 8.0 • Kernel Version: 2.4.18-14

47. The first problem during collection • We were unable to collect the /dev/kmem file. • The dd command terminated with the an error. # dd bs=1024 < /dev/kmem | nc 147.138.69.69 10003 -w 3 dd: reading `standard input': Bad address 0+0 records in 0+0 records out

48. The second problem during collection • The images of the hard drive created using dd were corrupt. • Each dd displayed an error message after each dump completed. dd: reading `/dev/hda1': Input/output error

49. Image corruption • The image files created were corrupt in two ways: • Superblocks within the image were damaged • Files being written to by the OS were empty

50. Compensating for the Corruption • The images that were used had to be mounted once in read-write mode to correct the superblocks if they were to be used. • The live file-systems were investigated instead of the image. • Tar balls of the missing log files were created.