1 / 27

Network Monitoring using MonALISA and NetFlow

Network Monitoring using MonALISA and NetFlow. Practical Applications born out of Collaborative Research NSF Award Number SCI – 0231844. Ernesto Miguel Rubi Research Network Engineer CIARA / CHEPREO / AMPATH Florida International University eMail: ernesto@cs.fiu.edu.

rodrick
Download Presentation

Network Monitoring using MonALISA and NetFlow

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Network Monitoring using MonALISA and NetFlow Practical Applications born out of Collaborative Research NSF Award Number SCI – 0231844 Ernesto Miguel RubiResearch Network EngineerCIARA / CHEPREO / AMPATHFlorida International UniversityeMail: ernesto@cs.fiu.edu PASI Conference - Mendoza, Argentina - May 16th to 21st, 2005

  2. Some Background & Acknowledgments • Born out of Summer 2003 NSF REU Award • Studied at-length the network utilization of an AmPATH link leading to U.S. FedNets ( non-educational US research networks ) • Used NetFlow v5, deployed throughout the AmPATH core to explore WHO, WHEN, HOW and HOW MUCH of a particular network segment. • Why is this important? • It’s like knowing which genes in your body serve what purpose... • Can adjust various network policies according to findings ( QoS, routing ) • Can justify to funding agencies the need for increased funding for a particular link and present factual evidence with specific networks and agency usage numbers • Well, it’s really cool! No really, if you’re a network engineer you want / can verify your routing policies are working and can have a DETAILED view of your network traffic. • I haven’t done the Acknowledgments yet...

  3. Would like to thank... • AmPATH / CIARA • Julio Ibarra / Heidi Alvarez • CalTech • Xun Su - Network Engineer • FIU • Jose Fernandez • Eric Johnson • CERN / SLAC - SDSC • MonALISA Team • Joeerg Michel, entire Monitoring Team ( NLANR / DAST ) • Coffee break is soon, I promise it’ll be interesting...let’s check it out!

  4. Where do you want to go? • Really far.... • Realtime In-Depth network monitoring, scalable with a built-in historical component; quick, cheap platform independent solution built on something reliable and already proven • Issues which we knew we’d find • Large data sets to analyze ( typical AmPATH circuits 10GigE, GigE, OC-12, OC-3 ) • What is a good enough time to keep historical data ( week, month, year...can we keep that much? ) • What defines real-time network monitoring? Up to the second, minute, hour? • What network blindspots does NetFlow create with its sampling characteristics • We don’t want to re-invent the wheel, who can accommodate the data we have into a convenient already proven framework. • Can we collaborate with our REU colleagues across multiple timezones and across language barriers and busy schedules? • Let’s start at the beginning, NetFlow is the starting point since we already have plenty of experience with it from everyday use and previous REU work.

  5. NetFlow and Flow-Tools • NetFlow • Was originally developed by Darren Kerr and Barry Bruins at Cisco Systems in 1996 as a switching path. Today NetFlow is primarily used for network accounting. AmPATH uses NetFlow v5 in its core routers; different versions available. • Different vendors use NetFlow and call it different things but everyone supports it. • NetFlow is a process that: • Runs in the background on the router, • Samples data at a pre-configured rate and • Exports it to a pre-determined host or hosts. • So...what does a NetFlow configuration on a router looks like? Well..just a line or two.. Cisco GSR 12012 – NetFlow Configuration ip flow-export source Loopback0 ip flow-export version 5 ip flow-export destination 131.94.191.101 2058 ip flow-sampling-mode packet-interval 100

  6. Let’s first define a flow • A flow is IP data which has the following seven identical characteristics: • Source IP address • Destination IP address • Source port • Destination port • Layer 3 protocol type • TOS byte ( RFC 791 - Type of Service ) • Input logical interface on a router

  7. So what does the output look like? • It’s trivial!!!

  8. So Sometimes you Need an interpreter.... • Flow-Tools ( http://www.splintered.net/sw/flow-tools ) • A collection of programs and libraries used to collect and process NetFlow data. These tools allow users to process stored flow data from a series of command line interfaces. ( WARNING: Only works on Cisco and Juniper routers ). Other interpreters out there.... • For our analysis we used Flow-Tools version 0.66: • The example below shows traffic destined outside of the router via interface 61 using reporting format 5 (destination port) and sorting by octets. $flow-cat ft-v05.2005-01-19.135207-0500 | flow-filter -I 61 | flow-stat -f 5 -S 2 # --- ---- ---- Report Information --- --- --- # # Fields: Total # Symbols: Disabled # Sorting: Descending Field 2 # Name: UDP/TCP destination port # # Args: flow-stat -f 5 -S 2 # # # port flows octets packets # 80 7036 2693420 14157 44191 233 1602211 2456 25 793 1041730 2394 46878 2 937500 625

  9. The AmPATH network comes under a watchful eye...

  10. MonALISA ( from Iosif Legrand ) • Reliable Registration and Discovery for Services and Applications. • Monitoring all aspects of complex systems: • System information for computer nodes and clusters • Network information : WAN and LAN • Monitoring the performance of Applications or services • The End User Systems • Can interact with any other services to provide in near real-time customized / filtered information based on monitoring data • Secure, remote administration for services and applications • Agents to supervise applications, to restart or reconfigure them, and to notify other services when certain conditions are detected. • The MonALISA framework can be used to develop higher level decision services, implemented as a distributed network of communicating agents, to perform global optimization tasks. • Powerful Graphical User Interfaces

  11. An Aside - MonALISA and Grids

  12. MonALISA and ApMon • ApMon is an Application Programming Interface (API) which interacts with the MonALISA service. • ApMon allows any application to send parameterized monitoring information to MonALISA. The data can be sent as UDP datagrams to multiple hosts running the MonALISA service. • ApMon has been implemented in C, C++, Java, Perl, Python. • NetFlow allows for the monitoring of a large number of parameters. We decided to limit the parameters monitor to...

  13. MonALISA and ApMon • UDP/TCP port destination/source traffic • IP destination/source traffic • IP protocol traffic • IP Next Hop traffic • AS destination/source traffic • Prefix destination/source traffic • For most of these parameters we will be monitoring that total traffic in octets over a period of time. The initial period of time was 5 minutes.

  14. Our Development... The application written will parse the flow-stat report and generate a name/value pair. The name of the parameter will always be the first column of the flow-stat report. This name will be one of the following: • A Port number (or name when applicatble) • An IP address • A protocol name or number • An AS number • A Prefix (in the form of network/mask bit) The value of the parameter will always be the total number of reported octets for that parameter. This will generate a large number of parameters which will be difficult to sort visually.

  15. Two Sides to the Coin... • A configuration file • Drawback: Large and clumsy to edit. Not easy to manage but this can be easily solved with a couple of more weeks of development time. • Code to parse configuration file and submit to MonALISA the results of the parsing algorithm. cluster_name:GSR-StarLight VLAN node_name:Ingress Destination Ports flow_file_directory:/home/netflow/flows/gsr/ flow_stat_options:-f 5 -S 2 -n flow_filter_options:-i 60 value_param:2 max_params:15 next

  16. Processing... Our application would parse this information and generate the following report: $flow-cat ft-v05.2005-01-19.224516-0500 | flow-filter -i 60 | flow-stat -f 5 -S 2 -n # --- ---- ---- Report Information --- --- --- # # Fields: Total # Symbols: Enabled # Sorting: Descending Field 2 # Name: UDP/TCP destination port # # Args: flow-stat -f 5 -S 2 -n # # # port flows octets packets # 57893 1 913876 1106 32808 1 730500 487 6881 52 210301 283 32920 1 151132 101 32918 1 85132 57 1144 3 84854 61 http 39 68171 106 1279 1 67500 45 eDonkey-20 67 64742 107 3113 2 63044 49 33498 3 58552 40 The application will parse the generated report and retrieve the parameters name and values as specified in the configuration. For this example the parameter names will be the port numbers and the values will be the octets as specified by the configuration file (value_param:2).

  17. MonALISA Customizations • I knew I was forgetting something...we wrote this in Perl. • ml.properties file heavily modified due to large number of data. • lia.Monitor.use_epgsqldb=true • lia.Monitor.jdbcDriverString=com.mckoi.JDBCDriver • These parameters tell MonALISA to use the embedded PGSQL database as opposed to the MYSQL database. • After working with the MonALISA development team they suggested to use the PGSQL database as it was faster and more stable for our configuration.

  18. MonALISA customizations lia.Monitor.Store.TransparentStoreFast.web_writes = 3 Specifies that 3 tables will be created for historical data. lia.Monitor.Store.TransparentStoreFast.writer_0.total_time=43200 lia.Monitor.Store.TransparentStoreFast.writer_0.table_name=monitor_s_12hour lia.Monitor.Store.TransparentStoreFast.writer_0.writemode=1 Defines the first table to maintain data for 12 hours. The data stored in this table written in writemode 1 which will write every value received by MonALISA. lia.Monitor.Store.TransparentStoreFast.writer_1.total_time=10800 lia.Monitor.Store.TransparentStoreFast.writer_1.table_name=monitor_s_e_3hour lia.Monitor.Store.TransparentStoreFast.writer_1.writemode=2 Defines the second table which will maintain data for 3 hours. This table will use writemode 2 which will store all data included objects which can not be displayed with the MonALISA client. lia.Monitor.Store.TransparentStoreFast.writer_2.total_time=16070400 lia.Monitor.Store.TransparentStoreFast.writer_2.samples=4464 lia.Monitor.Store.TransparentStoreFast.writer_2.table_name=monitor_s_6months lia.Monitor.Store.TransparentStoreFast.writer_2.writemode=0 This last table will store sampled stat for 186 days approximately 6 months. Writemode 0 specifies that the data will be sampled.

  19. Use Cases • Let’s do a live demo...risks are fun...

  20. In case that didn’t work...

  21. Tool Detail

  22. Tool Detail ( cont. )

  23. Tool Detail ( cont. )

  24. You never talked about realtime... • Right, so NetFlow is not realtime, data collected in 5 minute snaps, so when we analyze the data it’s already 5 minutes old! • Also, it’s not a 1:1 view of packets in the network; remember it’s sampled and 1:100 is the best ration we can get without crashing our router. • Wouldn’t be nice to capture 1:1 samples on a router with 25 - 10GigE interfaces...besides routers are made to route, their primary function is not monitoring. • Right around.....here...we ran out of time but...here’s a teaser..we’ve been awarded the Cisco University Research Program Grant to continue monitoring research and integrate the NLANR Passive Monitoring Agent into our work • Can help answer the question: How good is NetFlow, how many flows are we missing when we use NetFlow... • It helps to answer them by:

  25. PMA Overview • Visit: http://pma.nlanr.net/ • Tap into physical network media ( fiber ) in a non-obstrusive way and generate traces of packet headers. • Limited to short time samples because of data generated • IP address of hosts on flows masked due to some security concerns ( can cause issues for research ) • Not much of an overview but let’s talk afterwards if you want more info. • Which leads to us to....

  26. Next Steps • Integrate PMA results into current tool • Correlate data obtained in both environments to come up with comprehensive answers to problems posed. • Begin dissimenating the REU results and collaborating others who are interested in helping / deploying. • Look at emerging issues such as user privacy and address them to be able to release a public version of software. • Sleep.

  27. See, that wasn’t that bad.... • Questions... • How about a more extensive demo later? • Please say yes....!!

More Related