Wise * TrafView ETRI’s Content-aware Internet Application Traffic Measurement and Analysis System

1. Wise*TrafViewETRI’s Content-aware Internet Application Traffic Measurement and Analysis System APAN Network Technology WS January 29, 2004 IP Networking Technology Team, ETRI {jungsp, chunghs, choits, tsjeong}@etri.re.kr

2. Contents • Current Internet Application Traffic Characteristics • Wise*TrafView : Our Approach • Wise*TrafView : Implementation and Deployment Experiences • Summary

3. Measurement Application Areas • Network Problem Determination and Analysis • Traffic Report Generation • Intrusion & Hacking Attack (e.g., DoS, DDoS) Detection • Service Level Monitoring (SLM) • Network Planning • Usage-based Billing/Accounting (both between SPs and SP-and-Customer) • Customer Relationship Management (CRM) • Marketing

4. 2. CURRENT INTERNET APPLICATION TRAFFIC CHARACTERISTICS

5. Current Internet Traffic Characteristics • High-speed networks (Mbps  Gbps  Tbps) • High-volume traffic • Variety of Applications • Streaming media (Windows Media, Real Media, Quicktime) • P2P traffic • Network Games • Network Security Attacks • Etc.

6. Application Recognition(1) • Limitations of port-based recognition • The port database maintained by IANA doesn’t reflect the real-world situation • Most newer applications simply do not register their ports • Sometimes they even invade well-known port area to pass thorough firewalls • Most bandwidth hogs, nowadays, dynamically allocate ports • They are not linked up with any fixed ports!

7. Application Recognition(2) :Trend in Internet Application Traffic Characteristics PosTech Traffic Breakdown • PosTech Campus Network • (24h sum in May, 304GB total volume)

8. Application Recognition(3) • Many applications require to be identified by payload inspection • Why is payload inspection necessary? • Several applications can use the same port number • Identification error can be occurred by ephemeral port number • Some applications can use a dynamic port number • Etc.

9. Application Recognition(4) • Application example : Passive FTP % ftp server % ls % passive % quit % get wmggw.mp3 server.21 (FTP_CTRL_REQ) client.1302 server.21 (FTP_CTRL_REP) client.1302 49152 server.20 (FTP_DATA_DOWN) client.1303 client.1303 server.20 (FTP_DATA_UP) client.1306 server.49152 (FTP_DATA_PSV_UP) server.49152 (FTP_DATA_PSV_DOWN) client.1306

10. Why Port-based Approach is not enough? • Non-flow based measurement • Not enough for the above requirements • Typical Flow-based Measurement (like NetFlowTM, cflowd, LFAP) • Typically a flow is defined as a set of packets passing an observation point in the network during a certain time interval and having a set of common properties • 5-tuple packet header fields are used for this • New applications such as P2P, streaming and network games have characteristics of dynamic port allocation • More Detailed Analysis is needed!! • Typical Flow-based Measurement is not enough • Need more detailed analysis depending on applications • It may require content filtering

11. 3. Wise*TrafView : OUR APPROACH

12. Motivation • Develop precise Internet application traffic measurement and analysis system • Precise application analysis • Passive flow-based measurement • Sub-transaction(flow) level detailed application analysis • Pseudo-realtime analysis • No loss capture and analysis • No sampling but capturing all • For various Internet measurement purposes

13. a flow generated by application “X” a distinctive signature of application “X” Now, these pkts can also be identified as “X” Flow Concept • A “flow” is • a sequence of packets whose <src and dst IP addresses, src and dst port numbers, and protocol id> are all identical • Why flow? • The size of entire raw packet streams for a given unit time are prohibitively enormous to be analyzed in time • Each individual packets in a flow contain duplicate information • Packets in the same flow are correlated; we can identify more packets which were previously categorized as unknown application a packet

14. Internet Application Classification • Type S: Simple Application Type • for an application which uses a well-known port number or which uses a registered port number but is popularly used • Applications : WWW, FTP, SMTP, BGP, etc. • Type P: Payload Application Type • for an application which uses a registered port number but requires payload inspections for precise classification • Applications : HTTP_ALT(8080,8081,9000), MSNMessenger(6891-6900), KAZZA(1214), … • Type R: Reverse Application Type • for an application which uses a registered but requires comparison with a correlated reverse flow for the precise classification • Applications : eDonkey down, WINMX down, GuruGuru BBS(9999)… • Type C: Co-related Application Type • for an application which uses a dynamic port number assignment • Applications : Passive FTP, RTSP, Windows Streaming, …

15. System Architecture Overview GUI Database ARCL Config-File recognition and analysis results (ODBC) Analysis Server flow and packet records (NFS) ... Capture Agent Capture Agent ... ... NIC IPCAP Card NIC IPCAP Card . . . . . . splitter

16. Agent : Generating Flow & Packet Records • Carries on simple filtering and signature matching functions • Generates flow records & packet records • Flow record • For flow information • Fields : IP addr, port, protocol, flow duration, packets, bytes, … • Packet record • for individual packet • Fields : timestamp, TOS, TTL, TCP flags, payload, … • Important for analysis server’s precise application identification • This procedure aggregates and organizes the traffic information and reduces the amount of traffic volume transferred to the server

17. Analysis Server :Enhanced Application Recognition • Wise*TrafView utilizes some enhanced proprietary recognition mechanisms in a comprehensive way • Application specific signature matching, • temporal and spatial flow correlation, • dynamic port recognition and utilization, and • some heuristics • Not only capable of discriminating applications, but also their sub-flows • e.g., HTTP  HTTP_REQ, HTTP_REP, HTTP_REQACK, etc.

18. Analysis Server : AS and Country Mapping • Identifying flow sources and destinations • Both source and destination IP address of a flow are mapped to ASes and finally to country codes • This helps to locate the source and the sink of a flow • enables discrimination among transit, inbound, and outbound traffic flows

19. Application Recognition Configuration Language (ARCL) • Configurability and Adaptability • Why adaptability so important? • The highly frequent nature of Internet applications’ appearance and disappearance • Swift mutation of applications • Localization of the use patterns of applications • Wise*TrafViewcopes with the problem by introducing ARCL (Application Recognition Configuration Language) • By taking advantage of ARCL, Wise*TrafView • doesn’t need to be re-built or re-installed by any module for extending or modifying recognition coverage; editing the configuration in ARCL and re-enforcing suffices

20. Config-file by ARCL application WWW { port_rep_name HTTP port 80 protocol TCP{ // S type decision_group HTTP_REQ_REP_ACK { src_port >= 1024 dst_port == 80 } decision_group HTTP_REP_REQ_ACK { src_port == 80 dst_port >= 1024 } } port_rep_name HTTP_ALT port 8080 protocol TCP{ // P type src_disc_pattern=="HTTP" in pkt 0-2 at byte 0 - 4 ( dst_disc_pattern=="GET" in pkt 0-3 at byte 0 - 10 || dst_disc_pattern=="POST" in pkt 0-3 at byte 0 - 10 ) decision_group HTTP_ALT_REQ_REP_ACK { src_port >= 1024 dst_port == 8080 } decision_group HTTP_ALT_REP_REQ_ACK { src_port == 8080 dst_port >= 1024 } } } application EDONKEY { // R type port_rep_name EDONKEY_DOWN port 4662 protocol TCP{ dst_disc_pattern=="0xe33d000000" in pkt 2-3 at byte 0 - 4 decision_group EDONKEY_DOWN_REQ_REP_ACK { src_port >= 1024 dst_port == 4662 ~ 4666 || 4242 || 4224 || 4660 || 5555 } decision_group EDONKEY_DOWN_REP_REQ_ACK { src_port == 4662 ~ 4666 || 4242 || 4224 || 4660 || 5555 dst_port >= 1024 } } …… } application FTP { // C type port_rep_name FTP port 21 protocol TCP{ src_ref_pattern=="r/227 Entering Passive Mode \(\d{1,3},\d{1,3},\d{1,3},\d{1,3},(\d{1,4}),(\d{1,4})\)/$src_port = atoi($1)*1024 + atoi($2)" in pkt any at byte 0-35 induce FTP_DOWN_P decision_group FTP_REQ_REP_ACK { src_port >= 1024 dst_port == 21 } decision_group FTP_REP_REQ_ACK { src_port == 21 dst_port >= 1024 } } }

21. 4. Wise*TrafView : IMPLEMENTATION & DEPLOYMENT EXPERIENCES

22. Deployment Experiences • ETRINet • Link speed : 100Mbps FastEthernet, using libpcap • Traffic volume : 70Mbps • Period : May 2003 – Current • Analysis result : S(52.83%), P(9.99%), R(2.38%), C(4.92%), Unknown(28.88%) • Postech • Link speed : 1Gbps Ethernet, using libpcap • Traffic Volume : 60 – 70Mbps • Period : May 2003(1week) • Univ. of Andong • Link speed : FastEthernet, using capturing card developed by ETRI • Traffic volume : 60-70Mbps • Period : Oct. 2003 - Current • Other experiences • Deployment on the International link of one of Korean Internet Exchange point using OC-3 POS card developed by ETRI

23. PosTech Traffic Analysis Result • PosTech Campus Network • (24h sum in May, 304GB total volume)

24. System Spec.(1) • Hardware • For lower speed links (<= 622Mbps) • Capture agent • high performance PC: Zeon 2.4GHz * 2 + CPU, 2GB+ RAM • Analysis server • high performance PC: Zeon 2.8GHz * 2 + CPU, 1GB+ RAM, 100GB+ HDD • For Higher speed links ( > 1 Gbps, under developing) • Clustered capture system • Hardwired logic for supporting wire-speed processing • Software • Capture agent • Linux • Analysis server • Linux, MySQL

25. System Spec.(2) • Link Signal Splitters • Electrical • Ethernet tap, DS-3 tap, etc. • Optical • ordinary optical splitter • independent of physical and data-link layer protocols • High Performance Packet Capture Cards • Model A: for lower speed links • Ethernet, FastEthernet, DS-3/(E3) • Model B: for middle speed links • ATM at OC-3, and POS at OC-3, OC-12 (622Mbps)

26. User Interface • Web-based Interface • simple • easy to use • intuitive • portable • A web site for each measurement site can be easily established • Autonomous authentication and authorization can be supported

27. GUI (Traffic Report)

29. GUI (Traffic Matrix)

30. 5. SUMMARY

31. The Merits of Wise*TrafView • Transparent Packet Capture • Complete independence of the existing networking equipment • Flow-based Measurement and Analysis • Reduced load • Higher degree of recognition • Understanding Application Specific Contexts • By means of enhanced application recognition algorithms, sub-flows can be detected • Scalable • Can scale up from tens of Mbps to Gbps • Supports various physical and data-link layer technologies • Highly Extensible and Adaptable • Easy configuration with ARCL

32. Thank you! Q&AContact: jungsp@etri.re.kr,chunghs@etri.re.kr, choits@etri.re.kr, tsjeong@etri.re.kr