1 / 23

SMARTxAC: A Passive Monitoring and Analysis System for High-Speed Networks

SMARTxAC is a low-cost platform for continuous monitoring and analysis of high-speed network links, detecting anomalies and irregular usage. It provides real-time traffic monitoring and data reduction to improve performance and reduce storage requirements.

sdell
Download Presentation

SMARTxAC: A Passive Monitoring and Analysis System for High-Speed Networks

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SMARTxAC: A Passive Monitoring and Analysis System for High-Speed NetworksTERENA Networking Conference 2006 Pere Barlet-Ros Josep Solé-Pareta Javier Barrantes Eva Codina Jordi Domingo-Pascual {pbarlet, pareta, jbarranp, ecodina, jordid}@ac.upc.edu http://www.ccaba.upc.edu/smartxac Acknowledgment: This work has been partially supported by CESCA (SMARTxAC agreement) and the Spanish MEC (ref. TSI2005-07520-C03-02)

  2. SMARTxAC • SMARTxAC: Traffic Monitoring and Analysis System for the Anella Científica • Operative since July 2003 • Developed under a collaboration agreement CESCA-UPC • Tailor-made traffic monitoring system for the Anella Científica • Main objectives • Low-cost platform • Continuous monitoring of high-speed links without packet loss • Detection of network anomalies and irregular usage • Multi-user system: Network operators and Institutions • Measurement of two full-duplex GigE links • Connection between Anella Científica and RedIRIS • Current load: ≈ 1.5 Gbps / ≈ 270 Kpps

  3. Anella Científica Measurement point 2 x GigE full-duplex

  4. Daily Network Usage

  5. System Architecture • Monitoring high-speed links is challenging • Collection of Gbps and storage of Terabytes of data per day • Limitations of current technology • CPU power, memory access speeds, bus and disk bandwidth, storage capacity, etc. • Tailor-made system divided according to real-time constraints and running on different computers • Capture System (severe real-time constraints) • Traffic Analysis System (soft real-time constraints) • Result Visualization System (user driven) • Data reduction: Early discard unnecessary information • Improve performance • Reduce storage requirements

  6. Measurement Scenario ANELLA CIENTÍFICA GÉANT Global Internet Juniper M-20 (RedIRIS) ESPANIX 2 x 2Gbps REDIRIS Other Regional Nodes RedIRIS RedIRIS (Madrid) CISCO 6513 (Anella Científica) Private network 2 Gbps Management network dag0 2 Gbps Internet Connection dag1 Result Visualization System Traffic Analysis System (Linux) Capture System(DAG 4.3GE + GPS)

  7. Capture System • Capture hardware • Intel Xeon 2.4 GHz. + 1 GB. RAM • 2 x Endace DAG 4.3GE • 4 x Optical splitters • Precise timestamping using GPS (Trimble Acutime 2000) • Capture software • Multi-threaded implementation • Collection of packet-headers without loss (no sampling) • 5-tuple flow aggregation • Aggregated flows are sent to the Analysis System • Data Reduction • Header collection: ≈1:10 (90 GB/min 9 GB/min) • Flow aggregation: ≈1:200 (45 GB/5 min 200 MB/5 min) • Some data is kept to analyze anomalies (window of ≈ 20 GB.)

  8. Measurement Scenario ANELLA CIENTÍFICA GÉANT Global Internet Juniper M-20 (RedIRIS) ESPANIX 2 x 2Gbps REDIRIS Other Regional Nodes RedIRIS RedIRIS (Madrid) CISCO 6513 (Anella Científica) Private network 2 Gbps Management network dag0 2 Gbps Internet Connection dag1 Result Visualization System Traffic Analysis System Capture System(DAG 4.3GE + GPS)

  9. Traffic Analysis System • Analysis hardware • Pentium IV 2.6 GHz. + 1 GB. RAM • Analysis Software • Aggregation of 5-tuple flows into classified flows • <srcIP, dstIP, srcPort, dstPort, proto>  <origin, dest., app> • Origins: Institutions (also Network access points) • Destinations: External networks RedIRIS is connected to • Bidirectional aggregation • This classification can be useful for charging/cost-sharing • Data reduction • Classified flows: >1:1000 (≈ 60 GB/day  ≈ 50 MB/day) • Compared with header traces: > 1:250000 (≈ 13 TB/day)

  10. Measurement Scenario ANELLA CIENTÍFICA GÉANT Global Internet Juniper M-20 (RedIRIS) ESPANIX 2 x 2Gbps REDIRIS Other Regional Nodes RedIRIS RedIRIS (Madrid) CISCO 6513 (Anella Científica) Private network 2 Gbps Management network dag0 2 Gbps Internet Connection dag1 Result Visualization System Traffic Analysis System Capture System(DAG 4.3GE + GPS)

  11. Result Visualization System • Hardware • Pentium III 450 MHz. • Software • Web-based graphical interface • Institutions only have access to their own statistics • Graphs are generated on demand • Available graphs • More than 300 combinations of graphs per institution and day • Statistics are updated every 5 minutes • Also weekly, monthly and yearly reports

  12. Use case 1: Port Scanning • Traffic profile per application (bps)

  13. Use case 1: Port Scanning • Traffic profile per application (flows/s)

  14. Use case 1: Port Scanning • Destination port: MySQL (tcp/3306)

  15. Use case 2: Warez Server • Traffic profile per application (bps)

  16. Use case 2: Warez Server • Top-10 (bytes)

  17. Use case 3: Denial-of-Service • Traffic profile per application (bps)

  18. Anomaly Detection • Threshold-based anomaly detection • An upper and lower traffic threshold can be set per institution • Thresholds: bits/sec, packets/sec and flows/sec • Different intervals: day/night and workday/weekend • Once an anomaly is detected additional information is kept • Additional information can be reviewed later offline • Profile-based anomaly detection (work in progress) • Time-series prediction (adaptive linear filter) • It is not needed to know the “ordinary” traffic profile • Anomalies are detected when actual traffic differs from its predicted value • Thresholds mitigate limitations of adaptive prediction with long-term anomalies

  19. Identification of Network Applications • Traffic classification in SMARTxAC is based on port numbers • Port-based classification is no longer reliable • P2P, dynamic ports, tunnelling, web-based services, … • We are developing a classification method based on machine learning techniques • It learns features of traffic flows that identify a given application • Packet payloads are only needed in the training phase • Once the system is trained only packet headers are needed

  20. Preliminary Results (Accuracy)

  21. Port-based vs. Machine Learning Port-based Machine learning

  22. Conclusions • SMARTxAC is a tailor-made network monitoring system that • Operates at gigabit speeds without packet loss • It is relatively low-cost • Provides very detailed information about the network usage • Multi-user system: network operators and institutions • Since 2003, SMARTxAC is daily used by CESCA to detect anomalies, attacks, performance problems, network faults, etc. • Future work • Anomaly detection and application identification • Sampling, IPv6 support, … • Deployment of more measurement points in the Anella Científica • Release the source code under an open-source license • Collaboration with Intel’s CoMo: http://como.intel-research.net

  23. SMARTxAC: A Passive Monitoring and Analysis System for High-Speed NetworksTERENA Networking Conference 2006 Pere Barlet-Ros Josep Solé-Pareta Javier Barrantes Eva Codina Jordi Domingo-Pascual {pbarlet, pareta, jbarranp, ecodina, jordid}@ac.upc.edu http://www.ccaba.upc.edu/smartxac Acknowledgment: This work has been partially supported by CESCA (SMARTxAC agreement) and the Spanish MEC (ref. TSI2005-07520-C03-02)

More Related