1 / 46

NetViewer: A Network Traffic Visualization and Analysis Tool

NetViewer: A Network Traffic Visualization and Analysis Tool. Seong Soo Kim L. Narasimha Reddy Electrical and Computer Engineering Texas A&M University. Contents. Introduction and Motivation Our Approach NetViewer’s Architecture NetViewer’s Functionality Evaluation of Netviewer

patsy
Download Presentation

NetViewer: A Network Traffic Visualization and Analysis Tool

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NetViewer: A Network Traffic Visualization and Analysis Tool Seong Soo Kim • L. Narasimha Reddy Electrical and Computer Engineering Texas A&M University

  2. Contents • Introduction and Motivation • Our Approach • NetViewer’s Architecture • NetViewer’s Functionality • Evaluation of Netviewer • Conclusion Texas A&M University

  3. Attack/ Anomaly • Single attacker (DoS) • Multiple Attackers (DDoS) • Multiple Victims (Worms, viruses) • Aggregate Packet header data as signals • Image based anomaly/attack detectors Texas A&M University

  4. Motivation (1) • Previous studies looked at individual flows behavior • These become ineffective with DDoS  Aggregate Analysis • Link speeds are increasing • currently at G b/s, soon to be at 10~100 G b/s • Need simple, effective mechanisms • Packet inspection can’t be expensive • Can we make them simple enough to implement them at line speeds? Texas A&M University

  5. Motivation (2) • Signature (rule)-based approaches are tailored to known attacks • Become ineffective when traffic patterns or attacks change • New threats are constantly emerging • Quick identification of network anomalies is necessary to contain threat • Can we design general mechanisms for attack detection that work in real-time? Texas A&M University

  6. Our Approach (1) • Look at aggregate information of traffic • Collect data over a large duration (order of seconds) • Can be higher if necessary • Use sampling to reduce the cost of processing • Process aggregate data to detect anomalies • Individual flows may look normal  look at the aggregate picture Texas A&M University

  7. Our Approach (2) - Environment Texas A&M University

  8. Packet Parser Statistical Analysis & Anomaly Detection Visualization & Alerting Network Traffic Detection Report The block diagram of NetViewer NetViewer’s Architecture • Packet Parser : Collects and filters raw packets and traffic data from packet header traces or NetFlow records. • Signal Computing Engine : Analyzes the statistical properties of aggregate traffic distributions. • Detection Engine : Thresholds setting through statistical measures of traffic signal. • Visualization Engine : Employing image processing , and displaying traffic signals and images • Alerting Engine : Attacks and anomalies are detected/identified in real-time Texas A&M University

  9. Packet Parser (1) • Packet headers carry a rich set of information • Data : Packet counts, byte counts, the number of flows • Domain : Source/destination address, source/destination Port numbers, protocol numbers • Processing traffic header poses challenges. • Discrete spaces • Large Domains • 232 IPv4 addresses • 216 Port numbers • Need Mechanisms to reduce the domain size • Need Mechanisms to generate useful signals Texas A&M University

  10. Packet Parser (2) –Data structure for reducing domain size • 2 dimensional arrays count[i][j] • To record the packet count for the address j in ith field of the IP address • Normalized packet counts • Effects • Constant, small memory regardless of the packets, 232 (4G)  4*256 (1K) • Running time O(n) to O(lgn) • Somewhat reversible hash function Texas A&M University

  11. Packet Parser (3) –Data structure for reducing domain size • Simple example • IP of Flow1 = 165. 91. 212. 255, Packet1 = 3 IP of Flow2 = 64. 58. 179. 230, Packet2 = 2 IP of Flow3 = 216. 239. 51. 100, Packet3 = 1 IP of Flow4 = 211. 40. 179. 102, Packet4 = 10 IP of Flow5 = 203. 255. 98. 2, Packet5 = 2 0 64 128 192 255 3 3 3 3 Texas A&M University

  12. 0 64 128 192 255 2 3 2 10 1 10 2 3 1 2 1 2 12 3 2 1 10 2 3 Packet Parser (3) –Data structure for reducing domain size • Simple example • IP of Flow1 = 165. 91. 212. 255, Packet1 = 3 IP of Flow2 = 64. 58. 179. 230, Packet2 = 2 IP of Flow3 = 216. 239. 51. 100, Packet3 = 1 IP of Flow4 = 211. 40. 179. 102, Packet4 = 10 IP of Flow5 = 203. 255. 98. 2, Packet5 = 2 Texas A&M University

  13. Signal Computing Engine • Correlation • To measure the strength of the linear relationship between adjacent sampling instants • Delta • The difference of traffic intensity • It is remarkable at the instant of beginning and ending of attacks • Scene change Analysis • Variance of pixel intensities in the image Texas A&M University

  14. Detecting Engine – Threshold setting • From generated distribution signals (Ss), derive statistical thresholds • High threshold TH : Traffic distribution less correlated than usual • Low threshold TL : Traffic distribution more uniform than usual Texas A&M University

  15. Visualization Engine • Treat the traffic data as images • Apply image processing based analysis Texas A&M University

  16. Image Generation Texas A&M University

  17. Texas A&M University

  18. Generated various traffic Images • Image reveals the characteristics of traffic • Normal behavior mode • A single target (DoS) • Semi-random target : a subnet is fixed and other portion of address is change (Prefix-based attacks) • Random target : horizontal (Worm) and vertical scan (DDoS) Texas A&M University

  19. Alerting Engine • Scrutinize the statistical quantities – correlation and delta • Identify the IP addresses of suspicious attackers and victims • Lead to some form of a detection signal • Generate the detection report Texas A&M University

  20. NetViewer’s Functionality • Traffic Profiling • General information of current network traffic • Monitoring • Monitor traffic distribution signal (Ss) over the latest time-window • Anomaly Reporting • Image-based traffic in the source/destination IP address domain and the 2-dimensional domain • Auxiliary Function • Multidimensional Image • Attack Tracking • Automatic Spoofed Address Masking Texas A&M University

  21. Traffic Profiling Function (1) Texas A&M University

  22. Traffic Profiling Function (2) • Understanding the general nature of the traffic ay the monitoring point • Bandwidth in Kbps and Kpps (packet per sec.) • Protocol : the proportion occupied by each traffic protocol in percent • Top 5 flows : the topmost 5 flows in packet count or byte count or flow number • Based on LRU (least Recently Used) policy cache Texas A&M University

  23. Monitoring Function (1) Texas A&M University

  24. Monitoring Function (2) • Traffic distribution signal (Ss) over the latest time-window • 3 kinds of selected signals – Ss of packet count, Ss of byte count, Ss of flow count • Source IP : packet count distribution signal in the source IP address domain • Source FLOW : the number of flow distribution signal in the source IP address domain • Source PORT : packet count distribution signal in the source IP port domain • MULTIDIMENSIONAL : multiple components of the above signals in source domain • Pr : the anomalous probability of current traffic under Gaussian distribution • Signal : the distribution signal computed by • illustrated with dotted vertical lines of 3s level • m and s : mean value and standard deviation of distribution signal using EWMA Texas A&M University

  25. Anomaly Reporting Function (1) Texas A&M University

  26. Anomaly Reporting Function (2)– normal network traffic • Use variance of pixel intensities • Distribution of traffic over the observed domain • During anomalies, the traffic distributions different from normal traffic • Higher correlation (DOS) • Lower correlation (worms) Texas A&M University

  27. Anomaly Reporting Function (3)– semi-random targeted attacks Texas A&M University

  28. Anomaly Reporting Function (4)– random targeted attacks • Worm propagation type attack • DDoS propagation type attack Texas A&M University

  29. Anomaly Reporting Function (5)– complicated attacks • Complicated and mixed attack pattern • The horizontal (dotted or solid) line => specific source scanning destination addresses. • The vertical line => random sources assail specific destination Texas A&M University

  30. Anomaly Reporting Function (6)– Summary of Visual representation of traffic • Worm attacks – horizontal line in 2D image • DDoS attacks – vertical line in 2D image • Line detection algorithm • Visual images look different in different traffic modes • Motion prediction can lead to attack prediction Texas A&M University

  31. Anomaly Reporting Function (7) Texas A&M University

  32. **************************************************************************************************************************** [ Time : Tue 10-14-2003 05:12:00 ] -------------------------------------------------------------- Source IP[1] 134. correlation = 17.48% possession = 18.77% delta = 2.50% S Source IP[1] 141. correlation = 4.33% possession = 3.94% delta = 0.79% S Source IP[1] 155. correlation = 58.20% possession = 56.80% delta = 2.84% S Source IP[1] 210. correlation = 5.66% possession = 6.51% delta = 1.60% S Source IP[2] 75. correlation = 17.47% possession = 18.77% delta = 2.51% S Source IP[2] 110. correlation = 4.62% possession = 5.25% delta = 1.21% S Source IP[2] 223. correlation = 4.31% possession = 3.94% delta = 0.78% S Source IP[2] 230. correlation = 58.21% possession = 56.84% delta = 2.76% S Source IP[3] 7. correlation = 15.59% possession = 17.02% delta = 2.74% S Source IP[3] 14. correlation = 53.99% possession = 52.31% delta = 3.41% S Source IP[4] 41 correlation = 15.16% possession = 16.36% delta = 2.30% S Source IP[4] 50 correlation = 52.58% possession = 50.83% delta = 3.54% S -------------------------------------------------------------- Identified No. 1st = 4, 2nd = 4, 3rd = 2, 4th = 2 ============================================================== Destination IP[1] 18. correlation = 4.37% possession = 3.88% delta = 1.01% S Destination IP[1] 128. correlation = 6.08% possession = 7.01% delta = 1.75% S Destination IP[1] 131. correlation = 53.65% possession = 52.33% delta = 2.67% S Destination IP[2] 181. correlation = 56.03% possession = 54.00% delta = 4.15% S Destination IP[4] 26 correlation = 3.89% possession = 3.58% delta = 0.65% S -------------------------------------------------------------- Identified No. 1st = 3, 2nd = 1, 3rd = 0, 4th = 1 ============================================================== * Identified Suspicious Source IP address(es) 134. 75. 7. 41 correlation = 17.48% possession = 18.77% delta = 2.50% S 141.223.xxx.xxx correlation = 4.33% possession = 3.94% delta = 0.79% S 155.230. 14. 50 correlation = 58.20% possession = 56.80% delta = 2.84% S 210.xxx.xxx.xxx correlation = 5.66% possession = 6.51% delta = 1.60% S ------------------------- * Identified Suspicious Destination IP address(es) 18.xxx.xxx.xxx correlation = 4.37% possession = 3.88% delta = 1.01% 128.xxx.xxx.xxx correlation = 6.08% possession = 7.01% delta = 1.75% S 131.181.xxx.xxx correlation = 53.65% possession = 52.33% delta = 2.67% ************************************************************** The detection report of anomaly identification. Anomaly Reporting Function (7)- Identification • Identify IP using statistical measures • Black list Texas A&M University

  33. Flow-based Network Traffic • The number of flows based visual representation • The number of flows in address domain. • The black lines illustrate more concentrated traffic intensity. • An analysis is effective for revealing flood types of attacks. Texas A&M University

  34. Port-based Network Traffic • Port number based visual representation • Normalized packet counts in port-number domain. • An analysis is effective for revealing portscan types of attacks. • Normal network traffic • Attack traffic: SQL Slammer worm • 0d 1434 = 0x 059A = 0d 5 + 0d 154 Texas A&M University

  35. Multidimensional Visualization • Study multi-dimensional signals in IP address i) packet counts  R ii) number of flows  G iii) the correlation of packet counts  B • Comprehensive characteristics. • Diverse analysis. Texas A&M University

  36. Evaluation in Address-based signals 1. True Positive rate by 3s, the number of detection / the number of anomalies. 2. False Positive rate by 3s 3. Expected true positive rate by NP test 4. Expected false positive rate by NP test 5. Likelihood Ratio in measurement by 3s / LR in NP test 6. Negative Likelihood Ratio by 3s / NLR in NP test • NP Test shows a little high performance than 3s • 2 dimensional is better than 1 dimensional. Texas A&M University

  37. Port-based signals • Port-based signal could be a powerful signal • Particularly useful for probing/scanning attacks Texas A&M University

  38. Multidimensional signals • Combined with three distinct image-based signals : address-based, flow-based and port-based • Improve the detection rates considerably • It is possible to detect complicated attacks using various signals Texas A&M University

  39. Attack Tracking - Motion prediction Texas A&M University

  40. Automatic Spoofed address Masking • Unassigned by IANA – especially, 1st byte • Blue-colored polygons indicate the reserved IP addresses – there should be no pixels matching the unassigned space • Destination IP : normal traffic • Source IP : SQL slammer using (randomly) address spoofed traffic Texas A&M University

  41. Comparison with IDS • Intrusion detection system (IDS) is signature-based compared to our measurement-based. • Compares with predefined rules • Need to be updated with the latest rules. • Snort as representative IDS. • Both show similar detection on TAMU trace. • Snort is superior in identification • But missed heavy traffic sources and new patterns • Required more processing time. Texas A&M University

  42. Advantages • Not looking for specific known attacks • Generic mechanism • Works in real-time • Latencies of a few samples • Simple enough to be implemented inline • Window and Unix versions are released at http://dropzone.tamu.edu/~skim/netviewer.html • Comments to seongsoo1.kim@samsung.com or reddy@ece.tamu.edu Texas A&M University

  43. Conclusion • We studied the feasibility of analyzing packet header data as Images for detecting traffic anomalies. • We evaluated the effectiveness of our approach for real-time modes by employing network traffic. • Real-time traffic analysis and monitoring is feasible • Simple enough to be implemented inline • Can rely on many tools from image processing area • More robust offline analysis possible • Concise for logging and playback Texas A&M University

  44. Thank you !! Texas A&M University

  45. Identification (2): Entire IP address level • Step 1: Employ 4 independent hash functions as a Bloom filter, h1(am), h2(am), h3(am), h4(am). • Step 2: Concatenation of suspicious IP bytes using e-vicinity. Continue to the 4th byte. • Step 3: Membership query of generated 4-byte IP address • Automatic containment for identified attacks Texas A&M University

  46. Processing and memory complexity • Two samples of packet header data 2*P, P is the size of the sample data • Summary information (DCT coefficients etc.) over samples S • Total space requirement O(P+S) • P is 232 4*256 = 1024 (1D), 264  256K (2D) • S is 32*32  16 • Memory requires 258K • Processing O(P+S) • Update 4 counters per domain • Per-packet data-plane cost low. Texas A&M University

More Related