1 / 13

Mining Anomalies in Network-Wide Flow Data

A system to detect and classify traffic anomalies at carrier networks using network-wide flow data.

cuomo
Download Presentation

Mining Anomalies in Network-Wide Flow Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Mining Anomalies in Network-Wide Flow Data Anukool Lakhina with Mark Crovella and Christophe Diot NANOG35, Oct 23-25, 2005

  2. My Talk in One Slide • Goal: A general system to detect & classify traffic anomalies at carrier networks • Network-wide flow data (eg, via NetFlow) exposes a wide range of anomalies • Both operational & malicious events • I am here to seek yourfeedback 

  3. Network-Wide Traffic Analysis • Simultaneously analyze traffic flows across the network; e.g., using the traffic matrix • Network-Wide data we use: Traffic matrix views for Abilene and Géant at 10 min bins

  4. NYC LA ATLA HSTN Power of Network-Wide Analysis Peak rate: 300Mbps; Attack rate ~ 19Mbps/flow IPLS Distributed Attacks easier to detect at the ingress

  5. But, This is Difficult! How do we extract anomalies and normal behaviorfrom noisy, high-dimensional data in a systematic manner?

  6. The Subspace Method[LCD:SIGCOMM ‘04] • An approach to separate normal & anomalous network-wide traffic • Designate temporal patterns most common to all the OD flows as the normal patterns • Remaining temporal patterns form the anomalous patterns • Detect anomalies by statistical thresholds on anomalous patterns

  7. One Src-Dst Pair Dominates: 32% of B, 20% of P traffic Cause:Bandwidth Measurement using iperf by SLAC An example user anomaly

  8. Multihomed customer CALREN reroutes around outage at LOSA An example operational anomaly

  9. Summary of Anomaly Types Found[LCD:IMC04] False Alarms Unknown Traffic ShiftOutageWormPoint-Multipoint Alpha FlashEvents DOS Scans

  10. Automatically Classifying Anomalies[LCD:SIGCOMM05] • Goal: Classify anomalies without restricting yourself to a predefined set of anomalies • Approach: Leverage 4-tuple header fields: SrcIP, SrcPort, DstIP, DstPort • In particular, measure dispersion in fields • Then, apply off-the-shelf clustering methods

  11. (SrcIP) Example of Anomaly Clusters Dispersed Legend Code Red Scanning Single source DOS attack Multi source DOS attack (DstIP) (SrcIP) Dispersed Concentrated Summary: Correctly classified 292 of 296 injected anomalies

  12. Summary • Network-Wide Detection: • Broad range of anomalies with low false alarms • In papers: Highly sensitive detection, even when anomaly is 1% of background traffic • Anomaly Classification: • Feature clusters automatically classify anomalies • In papers: clusters expose new anomalies • Network-wide data and header analysis are promising for general anomaly diagnosis

  13. More information • Ongoing Work: implementing algorithms in a prototype system • For more information, see papers & slides at: http://cs-people.bu.edu/anukool/pubs.html • Your feedback much needed & appreciated!

More Related