Statistical Flow analysis

1. Statistical Flow analysis Section 4.1 Network Forensics TRACKING HACKERS THROUGH CYBERSPACE

2. purpose • Identify compromised hosts • Send out more traffic • Use usual ports • Communicate with known malicious systems • Confirm / Disprove data leakage • Volume of exported data • Individual profiling • Reveal • Normal working hours • Periods of inactivity • Sources of entertainment • Correlate activity exchanges

3. Process overview • Defined • “Flow record—A subset of information about a flow. Typically, a flow record includes the source and destination IP address, source and destination port (where applicable), protocol, date, time, and the amount of data transmitted in each flow.” (Davidoff & Ham, 2012)

4. Flow record processing system • Flow record processing systems include the following components: • Sensor—The device that is used to monitor the flows of traffic on any given segment and extract important bits of information to a flow record. • Collector—A server (or multiple servers) configured to listen on the network for flow record data and store it to a hard drive. • Aggregator—When multiple collectors are used, the data is typically aggregated on a central server for analysis. • Analysis—Once the flow record data has been exported and stored, it can be analyzed using a wide variety of commercial, open-source, and homegrown tools.1 1. Pg 161

5. sensors • Sensor types • Network Equipment • Many switches support flow record creation and export • Cisco - NetFlow format • Sonicwall – IPFIX and NetFlow • Be cautious of “sampling” which is not comprehensive data • Standalone appliances • Used if existing network software does not support flow data • Software • Argus – Audit Record Generation and Utilization System • Softflowd • Yaf – Yet Another Flowmeter

6. Sensor software • Argus • Two packages • Argus Server • Argus Client • Libpcap- based • Supports BPF filtering • Documentation specifically mentions forensic investigation • Argus’ compressed format over UDP • Softflowd • Passively monitor traffic • Exports record data in NetFlow format • Linux and OpenBSD • Libpcap- based • Yaf • Libpcap and live packet transfer • IPFIX format over SCTP, TCP or UDP • Supports BPF filters

7. Sensor placement • Investigators often do not have much control over placement • Infrastructures should be set up with flow monitoring in mind but usually are not • Factors to consider • Duplication is inefficient and must be minimized • Time synchronization is crucial • Most flow records are collected on external devices such as firewalls but this ignores internal network traffic which can be valuable • Resources are important when planning, prioritize • Do not over load your network capacity

8. Modifying the environment • Leverage existing equipment • Switches, routers, firewalls, NIDS / NIPS • Upgrade network equipment • If existing equipment will not work deploy replacements • Deploy additional sensors • Use port mirroring to send packets to standalone sensor • Network tap another option

9. Flow record export protocols • Proprietary – Cisco’s NetFlow • Open source – IPFIX • Relatively new and not yet matured – better tools on the horizon

10. netflow • Maintains a cache that tracks the state of all active flows observed • Completed flows marked as “expired” and exported as a “NetFlow Export” packet to a collector • Newer versions (NetFlow v9) are transport-layer independent: UDP, TCP and SCTP • Older versions only support UDP and IPv4

11. ipfix • Extends NetFlow v9 • Handles bidirectional flow reporting • Reduces redundancy • Better interoperability • Extensible flow record data using data templates • Template defines data to be exported • Sensor uses template to construct flow data export packets

12. sflow • Supported by many devices – not Cisco • Conduct statistical packet sampling • Does not support recording and processing every packet • Scales very well • Generally not very good for forensic analysis

13. Collection and aggregation • Placement factors to consider • Congestion • Flow records generate network traffic and can intensify congestion • Choose location where this will cause low network impact • Security • Export flow records on separate VLAN if possible • Isolate physical cables • Encrypt using IPSec or TLS • Reliability • Consider using TCP or SCTP over UDP • Capacity • One sensor or many? • Analysis strategy • Can affect all of the above, plan accordingly

14. Collection systems • Commercial options • Cisco NetFlow Collector • Manage Engine’s NetFlow Analyzer • WatchPointNetFlow Collector

15. Collection systems continued • Open source options • SiLK – System for Internet Level Knowledge • Command-line • Most powerful – biggest learning curve • Collector specific tools – flowcap and rwflowpack • Flow-tools • Modular and easily extensible • Only accepts UDP input • Nfdump / NfSen • Collector daemon – nfcapd • UDP network socket or pcap files • Argus • Supports Argus format and NetFlow v 1-8 • NetFlow v9 and IPFIX not yet supported

16. Analysis • Defined • “Statistics—“The science which has to do with the collection, classification, and analysis of facts of a numerical nature regarding any topic.” (The Collaborative International Dictionary of English v.0.48).” (Davidoff & Ham, 2012) • Purpose • Store a summary of information about the traffic flowing across the network • Forensic data carving does not apply • Still very useful

17. Flow record techniques • Goals and resources • This should shape your analysis • Access available time, staff, equipment and tools • Starting indicators – triggering event • Example evidence: • IP address of compromised or malicious system • Time frame of suspect activity • Known ports of suspect activity • Specific flows which indicate abnormal or unexplained activity

18. Flow record techniques continued • Analysis techniques • Filtering • Baselining • “Dirty Values” • Activity pattern matching

19. Filtering • Important to narrow down a large pool of evidence • Remove extraneous data • Start by isolating activity relating to specific IP address/es • Filter for known patterns of behavior • Use small percentages of data for detailed analysis

20. baselining • Advantage of flow record data vs full traffic capture • Dramatically smaller allowing for longer retention • Build a profile of “normal” network activity • Network baseline • General trends over a period of time • Host baseline • Historical baseline can identify anomalous behavior • Most flow patterns will change dramatically if host is compromised or under attack

21. “Dirty Values” • Suspicious keywords • IP addresses • Ports • Protocols

22. Activity pattern matching • Elements • IP address • Internal network or Internet-exposed network • Country of origin • Who are they registered too? • Ports • Assigned / well-known ports link to specific applications • Is system scanning or being scanned? • Protocols and Flags • Layer 3 and 4 are often tracked in flow record data • Connection attempts • Successful port scans • Data transfers • Directionality • Data coming in (something downloaded) or going out (something uploaded) • Volume of data transferred • Lots of small packets can indicate port scanning • Large amounts of data usually cause for concern

23. Simple patterns • Many-to-one IP addresses • DOS attack • Syslog server • “Drop box” data repository on destination IP • Email server (at destination) • One-to-many IP addresses • Web server • Email server (at source) • SPAM bot • Warez server • Network port scanning • Many-to-many IP addresses • Peer-to-peer file sharing • Widespread port scanning • One-to-one IP addresses • Targeted attack • Routine Server communication

24. Complex patterns • Fingerprinting • Matching complex flow record patterns to specific activities • Example: • TCP SYN port scan • One source IP address • One or more destination IP addresses • Destination port numbers increase incrementally • Volume of packets surpass a specified value within a given period of time • TCP protocol • Outbound protocol flags set to “SYN”

25. Flow record analysis tools • flowtools • SiLK • Argus • FlowTraq • Nfdump / NfSen

26. Silk • Rwfilter • Extracts flows of interest • Filters by time and category • Partitions them by protocol attributes • Generally as functional as BPF • Rwstats, rwcounts, rwcut, rwuniq • Basic manipulation utilities • Rwidsquery • Can be fed a Snort rule or alert file and it will figure out which flow matches it and writes an rwfilter to match it • Rwpmatch • Libpcap-based program that reads in SiLK-format flow metadata and an input source and save only the packets that match the metadata • Advanced SiLK • Includes a Python interpreter “PySiLK”

27. Flow-tools • Variety • Flow export data collection • Storage • Processing • Sending tools • “flow-report” • ASCII text report based on stored flow data • “flow-nfilter” • Filter based on primitives specific to flow-tools • “flow-dscan” • Identifies suspicious traffic based on flow export data

28. Argus client tools • Ra • Reads • Filters • Prints • Supports BPF filtering • Racluster • Exports based on user-specified criteria • Rasort • Sorts based on user-specified criteria • Ragrep • Regular expression and pattern matching • Rahisto • Generated frequency distribution table for user-selected metrics: flow duration, src and dst port numbers, byte transfer, packet counts, average duration, IP address, ports, etc

29. Flow traq • Commercial tool by ProQueSys • Supports many formats and sniffs traffic directly • Users can • Filter • Search • Sort • Produce reports • Designed for forensics and incident response

30. Nfdump • Part of the nfdump suite • Includes • Aggregate flow record fields by specific fields • Limit by time range • Generate statistics • IP addresses • Interfaces • Ports • Anonymize IP addresses • Customize output format • BPF-style filters Nfsen • Graphical, web-based interface for nfdump

31. etherape • Libpcap-based graphical tool • Visually displays activity in real time • Colors designate traffic protocol • HTTP • SMB • ICMP • IMAPS • Does not take flow records as input

32. Works Cited Davidoff, S., & Ham, J. (2012). Network Forensics Tracking Hackers Through Cyberspace. Boston: Prentice Hall.