NetFlows & Network Forensics

NetFlows & Network Forensics E. Larry Lidz ellidz@pobox.com

NetFlows • Logs right from routers and switches • Supported by most network vendors • Takes some CPU time on the routers • Might be turned on for network performance reasons • Quick to search through • Multiple ways to save on a system; multitude of available commercial tools

General Intro to Flows • Logs flows, not connections! • This is good… and bad. • Flows are uni-directional • You’ll see two flows for each connection: one in each direction. • Tools can reconnect them using heuristics

Introduction to Network Auditing • Obligatory "audit" definition: • 1. trans. To make an official systematic examination of (accounts), so as to ascertain their accuracy. • So, what is "Network" auditing? • Making a systematic examination of network traffic to understand the traffic and verify its legitimacy.

Introduction to Network Forensics • Obligatory "forensic" definition: • A. adj. Pertaining to, connected with, or used in courts of law; suitable or analogous to pleadings in court. forensic medicine: medicine in its relations to law; medical jurisprudence. • So, what is "Network" forensics? • Looking at traffic on a computer network in such a way that it could be used for evidence in a court of law.

So, what am I talking about? • Basically, the ability to log information about your network traffic in order to help with an investigation. • Generally, a compromise. • Also useful for trying to explain unexplained network problems.

A Quick Story • Machine on network compromised • Launched an evil DoS attack • Took out an ISP for a weekend • ISP thought they knew who did it • We were contacted by Alaska State Police and Secret Service • Suspect is in Anchorage, AK • Suspect is suspected of Credit Card Fraud • Compromise is a big, big problem • Lots of damage caused to ISP • Credit Card fraud via our network

A Quick Story, continued • The network audit logs show that the compromise occurred from *.ak.us.ibm.net • (IBM was a large ISP at the time) • We turn logs over to Alaska State Police • Note: none of our sensitive data was turned over; much, much easier to convince the powers that be to turn over • Kid denies it, confesses when shown the logs; turns on friends • (You can tell this is an old story – these days it’d be a professional crook) • Parents offer to pay restitution • (Do you think that ever happened?) • Network logs save the day!

Some Terminology • Network Flow • A group of packets related by having attributes in common. • Distinct from a “network connection.” • AS • Autonomous System Number • Every ISP has one, used in routing. • VLAN • Virtual LAN • A single broadcast segment of a network (a subnet). • These definitions are slightly simplified as the complexities aren’t relevant for this conversation. I’m sure you’ll hear more details on these later on… if you haven’t already.

TCP/IP • When looking at netflows, the more you know about TCP/IP, the better. • 3 main protocols: TCP, UDP, ICMP • UDP/ICMP are conectionless. • No way to easily tell if traffic is the start of a "connection" or part way through. • ICMP: Internet Control Message Protocol. • Ping, traceroute, etc. • Type: destinationunreachable (3), echo req (8), echo rep (0). • Code: 3.0: network unreachable, 3.3: port unreachable. • TCP has connections. • Controlled by TCP Flags: FIN, SYN, RST, PSH, ACK, URG • FIN: Sender is finished sending data. • SYN: Synchronize sequence numbers to initiate connection. • RST: Reset the connection (closing it). • PSH: Push data to the application as soon as possible. • ACK: Acknowledge • URG: Urgent flag set.

A TCP Connection • Client sends a SYN packet to server. • If the server is listening, it responds with a SYN-ACK packet. • Client then sends back an ACK packet. • Communication takes place with ACK (and often PSH) set. • Push flag is generally set when the application is waiting on input from a user or otherwise expects to be idle. • Side closing the connection sends a FIN. • Other side sends an FIN-ACK. • Other side sends a FIN. • Closing side sends an FIN-ACK. • If the server isn't listening, it sends a RST.

IP Addresses (v4) • Netmasking • 128.135.0.0/255.255.0.0 • 128.135.0.255 is broadcast for 128.135.0 subnet. • 127.0.0.1/255.0.0.0 • local net. 127.0.0.1 is local host • 10.0.0.0/255.0.0.0, 192.168.0.0/255.255.0.0, 172.16.0.0/255.240.0.0 • private IP space

TCP/UDP Ports • Useful to know a variety of ports. • In particular the ones that are used a lot on your networks. • General • 21: ftp • 22: ssh • 25: smtp • 53: dns • 80: http • 135, 137, 139, 445: windows ports • Details beyond the scope of this conversation • 443: https • 6667: irc

NetFlows vs. other security • vs. Intrusion Detection System • “We have an IDS, what else do we need?“ • vs. System Forensics • "We can always look at the system to figure out what happened.“ • vs. Sniffers • "We can always install a sniffer when we expect something is wrong.“ • vs. Firewalls • "We have firewall logs, who needs anything more?" • "When I've already got X, why do I need Network Audit Logs?"

NetFlow vs. … • IDS • Generally do not log traffic unless it is “suspicious” • System Forensics • With systems, you’re always looking at a system that has been tampered with • Sniffers • Disk consumption; privacy concerns • Firewalls • Often only at the network border; not optimized for searching logs

Defense in Depth • Use it all. • (Based on available budget, resources, etc.)

Network Design Considerations • Decide what you want to log. • Which machines? Which networks? • Choose places that handle interesting traffic. • Some examples: • Gateways • Peering points • DMZs • Important machines • Keep logging server close to router/switch • Theoretically possible to inject fake packets into stream • Difficult • Network traffic requirements for logs fairly small. • About a 1.5Mb/sec per 100 Mb/sec of traffic.

Prepare • Prepare ahead of time! • That way, when you you need it, you'll have logs to consult. • Too late if the incident has already happened. • Allocate hardware • Start small if necessary, build a case for the usefulness of the tools. • Install software • Test! See if you can avoid the logs. • If you can, it's probably misconfigured. • We noticed that we were missing logs from one segment to external networks. • Turns out we were missing an interface on our switch.

What can you see in a NetFlow? • Think of it like phone call records: who called whom, when. • Without the conversation. • Source/Destination IP; • Input/Output Router Interface • Protcol • Type of Service • NextHop • Packet Count • Octet Count • Start/End Time • TCP Flags • Source/Destination AS • Source/Dest Network Mask • Input Output Interface encapsulation Size • IP Address of next hop within the peer • Router IP of cache shortcut in supervisor

Reading NetFlows • Depends a bit on the tool… but: • 04/5/2011 11:59:46 -> 04/5/2011 11:59:46 6 01 65.219.38.62 3844 <-> 11 128.135.12.7 http 8 480 00 FS-PA- • 04/5/2011 11:59:46 -> 04/5/2011 11:59:46 6 11 128.135.12.7 http <-> 01 65.219.38.62 3844 11 11636 00 FS-PA- • Flow start time, flow stop time • Flows sorted by stop time • Protocol number • Source interface, source hostname, source port • <-> • Source and destination are the source and destination of the flow, not of the connection. • Should be two flows for each connection (assuming traffic is sent in both directions). • Destination interface, destination hostname, destination port • Packets in flow, Bytes in flow, Type of Service • TCP Flags: All TCP flags set in that flow.

An Incident • Received a report of a compromised system from a sysadmin • Well… of ten systems. • Systems had a file which listed the other machines in the same BotNet – 35 on our network • Each system we looked at had a different list of systems in that file. • The Challenge: find all compromised machines.

Incident, continued • We chose a system that wasn’t doing much (network-wise). • Why? • Pulled all the NetFlows for that system. • By reading the Flows, we were able to find the signs of a compromise (attack against an RPC service). • Also saw another host (38.31.107.236) connecting to a backdoor on a port (1524) • Then connections out from the machine to download a rootkit (to 152.3.127.99). • Then the BotNet traffic starts up. • Note: I’m skipping some of the analysis of how we figure out that it was downloading a rootkit, etc.

Incident, continued • Looked for all connections from our network to 152.3.127.99 • Sorted the output and got the list of our IP addresses in it. • We were able to easily pull together the list of compromised machines by looking at the network trail of the one machine. • In practice, we hand verified a lot of this. • It’s always good to double check. • Compared the machines to those listed in the file on the infected machines. • Did other analysis of the systems to ensure that they were compromised. (Forensics in depth?)

NetForensics: “when?” • When was the first traffic to a back door? • When did traffic first seem to successfully connect to ports that previous had no traffic? • When did BotNet traffic start up? • Did the machine start transferring lots more traffic at a certain point in time?

NetForensics: “how?” • What type of machine is it? • When you look at it, what vulnerable services, if any, do you see? • Did unexpected traffic start up shortly after a connection to a service running on the machine? • Connections to a port's service that isn't used. • ftp/http out to a third site • Particularly if it is grabbing a rootkit.

NetForensics: “who?” • Who is connecting to it that you don't expect to connect? • Which IP caused the compromise? • If you know when/how it was compromised. • Who connected to the backdoors? • Look for weird connections. • May be a backdoor or a covert channel.

NetForensics: “what?” • How much data was transferred? • Useful if you fear that information was stolen. • If they only transferred 500k of data, they didn't take the 20 Gig master plans for the new virtual-hyper-cyber-widget of doom. • What was on the machine in the first place?

NetForensics: “why?” • In my experience, NetFlows aren’t so useful for determining motive. • Sorry.

When in Doubt: • Read through the traffic logs for the compromised machine. • Extract logs for the specific host. • Very Time Consuming • You won't miss anything... • Unless there's too much data to look at. • Start ignoring stuff that you suspect to be legitimate (ntp, dns, etc.) • Don't worry about stuff before the compromise. • Build a timeline. • Keep notes as to the connections that you see. • Who connected out to where at what time? • Which are legitimate? • Interview system administrators and users.

Things to look for: • Intruders often script attacks • Machines will have similar traffic patterns • Same source IPs. • Same backdoor ports. • Connect to same external site. • Known bad traffic • IPs/hosts list on “bad traffic” reports • Known malware ports • Example: finding SQL Slammer • flow-extract -nd ft-v06.2003-01-25.030000 -e 'proto = 17 && port = 1434 && pkts = 1 && octets = 404 && ((srcnet = 10.0.0.0/255.255.0.0 && dstport = 1434 ) || (dstnet = 10.0.0.0/255.255.0.0 && srcport = 1434)) { print }

Argus • Logs off of a spanning port • No need to have router support • From QoSient. • Free for most purposes. Read the license agreement. • Can log bytes of application data, too. packet contents, smtp exchange, ftp username/password • A bit slower than NetFlows • I use it at home

Summary • NetFlows are a powerful tool. • Extremely useful for Security, but also for network management. • If I can find machines compromised by a malefactor trying to remain anonymous, I can easily find the bandwidth hogs. • Heavily adopted behind the scenes in Security and Network tools. • Very useful at making network traffic tangible as you’re learning more about it. • They’re a little bit higher level than a Sniffer would show you, and therefore easier to consume.

Questions? • Questions? • Comments? • Thoughts? • Anything else you’d like to learn about in the time remaining?

NetFlows & Network Forensics