Wise * TrafView Flow-based Measurement and Analysis System

1. Wise*TrafViewFlow-based Measurement and Analysis System Pipefilters BOF, TIP2004 Jan. 27, 2004 Hyungseok Chung ETRI

2. Topics • Backgrounds • Content-aware Application Recognition • Wise*TrafView Functionality • GUI Snapshots & Demonstration

3. Common Measurement & Analysis Tools • RTT, Packet loss • Tool : ping, echoping, fping, gnuplotping, sting, … • System : Ping-ER, AMP, IWR - use ping internally • One-way delay, Packet delay • Designated measurement BOX • Surveyor, RIPE-NCC project, Smartbits • Route discovery • traceroute, skitter, mtr, ping plot, visualroute, neotrace,… • Misc : remote traceroute execution server • Throughput • netperf, iperf, treno, tcpblast, tcpspray, ttcp, pchar • Packet Capture & Analysis • tcpdump, Coralreef, cflowd, snoop, ethereal, … • Per Interface Traffic Volume and Errors • MRTG

4. Why they are not enough? • Capturing Packets in Current Networks • High-speed networks (Mbps  Gbps  Tbps) • High-volume traffic • Streaming media (Windows Media, Real Media, Quicktime) • P2P traffic • Network Games • Network Security Attacks • Typical Flow-based Measurement • Non-flow based measurement is not enough for the above requirements • Typical Flow-based Measurement • Typically a flow is defined as a set of packets passing an observation point in the network during a certain time interval and having a set of common properties • 5-tuple packet header fields are used for this • New applications such as P2P, streaming and network games have characteristics of dynamic port allocation • More Detailed Analysis is needed • Typical Flow-based Measurement is not enough • Need more detailed analysis depending on applications • It may require content filtering

5. Splitter Router Switch Network Operators Traffic Capture Agent raw streams of packets analysis result flow records Analysis Server How does Wise*TrafView work? AS 100 AS 200

6. Application Recognition • Limitations of port-based recognition • The port database maintained by IANA doesn’t reflect real-world situation well • Most newer applications simply do not register their ports • Sometimes they even take advantage of well-known ports to pass thorough firewalls • Most bandwidth hogs, nowadays, dynamically allocate ports • They are not linked up with any fixed ports!

7. Real-world Situation PosTech Traffic Breakdown • PosTech Campus Network • (24h sum in May, 304GB total volume)

8. Enhanced Application Recognition • Wise*TrafView utilizes some enhanced proprietary recognition mechanisms in a comprehensive way • Internet Application Classification • signature matching • flow correlation • dynamic port recognition and utilization • some heuristics • Not only capable of discriminating applications, but also their sub-flows • e.g., HTTP  HTTP_REQ, HTTP_REP, HTTP_REQACK, etc.

9. Internet Application Classification • Type S: Simple Application Type • for an application which uses a well-known port number or which uses a registered port number but are popularly used • Type P: Payload Application Type • for an application which uses a registered or ephemeral port number but requires payload inspections for precise classification • Type R: Reverse Application Type • for an application which uses a registered or ephemeral port number but requires comparison with a correlated reverse flow for the precise classification • Type C: Co-related Application Type • for an application which uses a dynamic port number assignment • Type U: Unknown Application Type • for applications which do not use registered port numbers and do not belong to any of the four types mentioned above

10. Application Recognition Configuration Language (ARCL) application WWW { port_rep_name HTTP port 80 protocol TCP{ decision_group HTTP_REQ_REP_ACK { src_port >= 1024 dst_port == 80 } decision_group HTTP_REP_REQ_ACK { src_port == 80 dst_port >= 1024 }} port_rep_name HTTP_ALT port 8080 protocol TCP{ src_disc_pattern=="HTTP" in pkt 0-2 at byte 0 - 4 ( dst_disc_pattern=="GET" in pkt 0-3 at byte 0 - 10 || dst_disc_pattern=="POST" in pkt 0-3 at byte 0 - 10 ) decision_group HTTP_ALT_REQ_REP_ACK { src_port >= 1024 dst_port == 8080 } decision_group HTTP_ALT_REP_REQ_ACK { src_port == 8080 dst_port >= 1024 }} } application EDONKEY { port_rep_name EDONKEY_DOWN port 4662 protocol TCP{ dst_disc_pattern=="0xe33d000000" in pkt 2-3 at byte 0 - 4 decision_group EDONKEY_DOWN_REQ_REP_ACK { src_port >= 1024 dst_port == 4662 ~ 4666 || 4242 || 4224 || 4660 || 5555 } decision_group EDONKEY_DOWN_REP_REQ_ACK { src_port == 4662 ~ 4666 || 4242 || 4224 || 4660 || 5555 dst_port >= 1024 }} application FTP { port_rep_name FTP port 21 protocol TCP{ src_ref_pattern=="r/227 Entering Passive Mode \(\d{1,3},\d{1,3},\d{1,3},\d{1,3},(\d{1,4}),(\d{1,4})\)/$src_port = atoi($1)*1024 + atoi($2)" in pkt any at byte 0-35 induce FTP_DOWN_P decision_group FTP_REQ_REP_ACK { src_port >= 1024 dst_port == 21 } decision_group FTP_REP_REQ_ACK { src_port == 21 dst_port >= 1024 }} }

11. Application Recognition Example % ftp server % ls % passive % quit % get wmggw.mp3 server.21 (FTP_CTRL_REQ) client.1302 server.21 (FTP_CTRL_REP) client.1302 49152 server.20 (FTP_DATA_DOWN) client.1303 client.1303 server.20 (FTP_DATA_UP) client.1306 server.49152 (FTP_DATA_PSV_UP) server.49152 (FTP_DATA_PSV_DOWN) client.1306 0 2 4 6 8 10 12 Time (sec)

12. System Architecture Overview GUI Database ARCL Config-File Recognition and analysis Results (ODBC) Analysis Server Flow and packet Records (NFS) Capture Agent Capture Agent . . . IPCAP Card IPCAP Card NIC NIC . . . . . .

13. Capturing Internet Traffic • Passive traffic capture • No side-effect imposed on any network devices and links • An optical or electric splitter, a.k.a. tap, is utilized • Wise*TrafView’s approach • Splitters + Packet Capture Card + High Performance Capture Engine • Adaptability maintained by supporting software-based capture as well • PCAP (Packet Capture) library • Doesn’t necessarily require a dedicated capture card; commonUnix boxes can substitute the cards • But yet, software-based capture is not equivalent to the card-based capture in terms of performance and functionality

14. Specialized Packet Capture Devices • Link Signal Splitters • Electrical • Ethernet tap, DS-3 tap, etc. • Optical • ordinary optical splitter • independent of physical and data-link layer protocols • High Performance Packet Capture Cards • Model A: for lower speed links • Ethernet, FastEthernet, DS-3/(E3) • Model B: for middle speed links • ATM at OC-3, POS at OC-3, OC-12 (622Mbps), and GigaEthernet

15. a flow a distinctive signature of application “X” Now, these pkts can also be identified as “X” Flow Concept • A “flow” is • a sequence of packets whose <src and dst IP addresses, src and dst port numbers, and protocol> are all identical • Why flow? • The size of entire raw packet streams for a given unit time are prohibitively enormous to be analyzed in time • Each individual packets in a flow contain duplicate information • Packets in the same flow are correlated; we can identify more packets which were previously categorized as unknown application a packet

16. Agent Side: Generating Flow Records • Agents • carry on simple filtering and signature matching functions • generate flow records • This procedure aggregates and organizes the traffic information and reduces the amount of traffic volume transferred to the server

17. Agent Structure

18. Server Side:AS and Country Mapping • Identifying flow sources and destinations • Both source and destination IP address of a flow are mapped to ASes and finally to countries • This helps to locate the source and the sink of a flow • Discrimination among transit, inbound, and outbound traffic flows

19. Analysis Server Structure

20. Configurability and Adaptability • Why adaptability so important? • The highly frequenting nature of Internet applications’ appearance and disappearance • Swift mutation of applications • Localization of the use patterns of applications • Wise*TrafViewcopes with the problem by introducingARCL • By taking advantage of ARCL, Wise*TrafView • doesn’t need to be re-built or re-installed by any module for extension • can be easily reconfigured to handle a new application; modifying the configuration in ARCL and re-enforcing suffices

21. The Major Functionality of Wise*TrafView • Transparent Packet Capture • complete independence of the existing networking equipment • Flow-based Measurement and Analysis • reduced load • higher degree of recognition • Understanding Application Specific Contexts • by means of enhanced application recognition algorithms • Scalable • can scale up from tens of Mbps to Gbps • supports various physical and data-link layer technologies • Highly Extensible and Adaptable • easy configuration with ARCL

22. User Interface • Web-based Interface • simple • easy to use • intuitive • portable • A web site for each measurement site can be easily established • Authentication and authorization supported

23. Visualization:Traffic Breakdown Report

24. Visualization:Traffic Matrices

25. Platforms • Hardware • For lower speed links (<= 622Mbps) • capture agent • high performance PC: 2 * P-III 1GHz+ CPU, 2GB+ RAM, 30GB+ HDD • analysis server • high performance PC: 2 * P-IV 1GHz+ CPU, 1GB+ RAM, 100GB+ HDD • For Higher speed links ( > 1 Gbps) • Dedicated Standalone Capture System • Hardwised logics for supporting wire-speed processing • Software • capture agent • Linux • analysis server • Linux, MySQL

26. GPS Satellite ISP 1 Traffic Generator CDMA Basestation Traffic Center Traffic Generator ISP 2 Server ISP 3 Server Traffic Generator Server Measurement Agent Analysis Server Traffic Center IX Router A Possible Deployment Scenario

27. Thank you ! Q & A Contact: Hyungseok Chung, Taesang Choi, Taesoo Jeong {chunghs, choits, tsjeong}@etri.re.kr