1 / 15

DataGrid Wide Area Network Monitoring Infrastructure (DWMI) aka IEPM-BW

DataGrid Wide Area Network Monitoring Infrastructure (DWMI) aka IEPM-BW. Connie Logg February 13 and 17, 2005. History. Originally done for SC2001 demo and called IEPM-BW After SC2001, development continued FNAL picked up IEPM-BW and adapted it to their site

alvaro
Download Presentation

DataGrid Wide Area Network Monitoring Infrastructure (DWMI) aka IEPM-BW

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DataGrid Wide Area Network Monitoring Infrastructure(DWMI) aka IEPM-BW Connie Logg February 13 and 17, 2005

  2. History • Originally done for SC2001 demo and called IEPM-BW • After SC2001, development continued • FNAL picked up IEPM-BW and adapted it to their site • In Spring 2004 – redesigned for TeraPaths monitoring project • Currently still called IEPM-BW, and deployed at 4 sites

  3. Architecture - I • Use MySQL database • All configuration is in the database so the code can self configure • Allows flexibility for adding new types of data • Written in perl • Low impact probes (currently abwed, traced, and pingd) have daemons that run independently • High impact probes have a daemon (bw-synchd) which insures that high impact probes do not run simultaneously and that there is a break between each test.

  4. Architecture - II • Results from all probes written to a data directory and are loaded by load-datad daemon which assures that the data base is not bombarded by hundreds of writes simultaneously. • Analysis scripts run every hour or two depending upon how long they take • Plot data, traceroute reports, master web page generation

  5. MySQL Database Tables - I • NODES – Each node has an entry and its specs (latitude, longitude, contact, paths, et al.) • MONHOST – Active monitoring host(s) information (web/cgi paths, data analysis specs, et al.) • TOOLSPECS – Probe specifications (probe, probe options, frequency, testtype, et al.)

  6. MySQL Database Tables - II • Many types of tests possible • background – low impact tests which can run concurrently (traceroute, ping, abwe) • background-syn – Tests which must be run one at a time (iperf) • On demand – to be implemented

  7. MySQL Database Tables - III • SCHEDULE • scheduler inserts probe requests into the SCHEDULE • Daemons read SCHEDULE table for the probes they are responsible for within the “current” timeframe, and run the probes. All results are written to a data directory and loaded by the data loading daemon

  8. APIs and other utilities • Fetch-ping-data • Fetch-abwe-data • Fetch-trace-data • Fetch-bw-data (e.g iperf) • Fetch-trace-data • Etc.. • All take a nodename and timespan and return a filename where the data is stored

  9. Data Analysis • Time series plots – group and individual • Diurnal analysis & fitting • Traceroute analysis • Bandwidth Change Analysis – will be augmented by other methods currently be researched and developed

  10. CGI Utilities – in development • Add and update NODES • Add and update TOOLSPECS • Add and update MONHOST • Interactive data analysis

  11. Informational Web Pages • Table of defined NODES • Table of defined MONHOST • Table of TOOLSPECS – probe specifications • Description of data base tables • Report on data logging for past few weeks • PLM – needs updating • Others to come – every time I have to look at something for validation, I create a web page

  12. Futures • Make data available via web services • Interactive data analysis CGIs • Add additional probe types • Develop complete distribution kit – complicated by differing locations and versions of perl, gnuplot, mysql, graphics libs, ploticus, iperf, etc. • Add additional anomaly detection techniques

  13. Summary • The objective is to provide for regular and reliable network probe testing and data collection from several locations around the world • Make the data available to the community • Provide a framework for the incorporation of a variety of analysis tools

  14. Acknowledgements • Many people have contributed content to this system over the years Maxim Grigoriev (FNAL), I-Heng Mei (SLAC RA), Manish Bhargava (SLAC RA), Ruchi Gupta (SLAC RA) Mahesh Chhaparia (SLAC RA) Parakram Khandpur (SLAC RA) And of course: Les Cottrell

  15. Questions & Considerations • BWCTL – • not installed everywhere and it is one more thing I would need to install as part of the distribution kit and maintain • Does not do multiple iperf streams • May want other heavyweight tests that bwctl does not provide for • OWAMP – special NTP configuration

More Related