1 / 33

End-to-end Monitoring of High Performance Network Paths

End-to-end Monitoring of High Performance Network Paths. Les Cottrell , Connie Logg, Jerrod Williams SLAC, for the ESCC meeting, Columbus Ohio, July 2004 www.slac.stanford.edu/grp/scs/net/talk03/escc-jul04.ppt.

vevay
Download Presentation

End-to-end Monitoring of High Performance Network Paths

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. End-to-end Monitoring of High Performance Network Paths Les Cottrell, Connie Logg, Jerrod Williams SLAC, for the ESCC meeting, Columbus Ohio, July 2004 www.slac.stanford.edu/grp/scs/net/talk03/escc-jul04.ppt Partially funded by DOE/MICS Field Work Proposal on Internet End-to-end Performance Monitoring (IEPM), also supported by IUPAP

  2. Need • Data intensive science (e.g. HENP) needs to share data at high speeds • Needs high-performance, reliable e2e paths and the ability to use them • End users need long and short term estimates of network and application performance for: Planning, setting expectations & trouble shooting • You can’t manage what you can’t measure

  3. IEPM-BW • Toolkit: • Enables regular, E2E measurements with user selectable: • Tools: iperf (single & multi-stream), bbftp, bbcp, GridFTP, ping (RTT), traceroute • Periods (with randomization) • Remote hosts to monitor • Hierarchical to match the tiered approach of BaBar & LHC computation / collaboration infrastructures • Includes: • Auto-clean up of hung processes at both ends • Management tools to look for failures (unreachable hosts, failing tools etc.) • Web navigation of results • Visualization of data as time-series, histograms, scatter plots, tables • Access to data in machine readable form • Documentation on host etc. requirements, program logic manuals, methods

  4. Requirements • Requires: • Monitoring toolkit installed on Linux monitoring host • Host provided & administered by monitoring site personnel • No need for root privileges • Appropriate iperf, bbftp etc. ports to be opened • SLAC can do initial install & configuration for monitoring host • 50 line configuration file for each remote host, tells where directories, applications are located, options for various tools etc (mainly defaults) • Small toolkit installed at remote (monitored hosts) • Ssh access to an account at remote hosts • This is the biggest problem with deployment

  5. Achievable throughput & file transfer • IEPM-BW • High impact (iperf, bbftp, GridFTP …) measurements 90+-15 min intervals Fwd route change Iperf abing bbftp iperf1 Min RTT Rev route change Avg RTT Select focal area

  6. Visualization: traceroutes • Compact table to see correlations between many routes • Identify significant changes in routes • Differences in > 1 hop, NOT same first 3 octets, NOT same AS • Report all traceroute pathologies: • ! Annotations, ICMP checksum errs, non-responding interfaces, unreachable end host, stutters, multi-homed end host • Note, we observe: • most route changes (>98%) do not result in significant performance changes • Many performance changes (~50+-20%) are NOT due to route changes • Applications, host congestion, level 2 changes etc.

  7. Route table Example • Compact so can see many routes at once History navigation Route # at start of day, gives idea of root stability Multiple route changes (due to GEANT), later restored to original route Mouseover for hops & RTT Available bandwidth Raw traceroute logs for debugging Textual summary of traceroutes for email to ISP Description of route numbers with date last seen User readable (web table) routes for this host for this day

  8. Another example Get AS information for routes Level change Host not pingable TCP probe type Intermediate router does not respond ICMP checksum error

  9. Topology • Choose times and hosts and submit request Hour of day Alternate route SLAC ESnet Alternate rt GEANT JAnet Nodes colored by ISP Mouseover shows node names Click on node to see subroutes Click on end node to see its path back Also can get raw traceroutes with AS’ IN2P3 CESnet CLRC DL CLRC

  10. IEPM-BW HENP Deployment June 2004 • Measurements from SLAC & FNAL • BaBar, CMS, D0, CDF + • 60-70 remote hosts in 12 countries • Toolkits needed in monitor & remote hosts Range of bandwidths:500Kbps to 1 Gbps

  11. Working on: • Provide more options for security for remote hosts • Web services API access to data • Provide & integrate low network utilization tool: • ~ 25% of Abilene traffic is net measurement • Automate detection of anomalous step changes in performance • Evaluate using QOS or HSTCP-LP to reduce impact of iperf traffic • Evidence that causes packet loss (ESnet/FNAL/SLAC)

  12. Simplify remote security • Currently use ssh to start, kill servers, check things etc. • Instead run servers all time at remote host • Check & restart with cron job • Also kill hung processes with cron jobs • More work for remote admin • More difficult to check why things not working • NASA very hard to get account (requires training etc.), so this will be a work-around

  13. Data Access • Interactive web accessible • Most data can be downloaded in space or comma separated etc. (accessible via link or to program (e.g. using lynx to access URL)) • However non standard • Web services (GGF NMWG definitions) • Working (with Warren Matthews/GATech/I2) on defining / providing access to traceroutes for AMP & IEPM-LITE • MonALISA is accessing data via Web services

  14. Low impact bandwidth measurement • Goals: • Make a measurement in < second rather than tens of seconds • Injects little network traffic • Provide reasonable agreement with more intense methods (e.g. iperf) • Enables: • Measurements of low performance links (e.g. to developing countries) • Helps avoid need for scheduling • More frequent measurements (minutes vs. hours) • Lower impact more friendly

  15. Low impact Bandwidth • Use 20 packet pairs to roughly estimate dynamic bw Capacity & Xtraffic, then Available = Capacity – Xtraffic • Capacity µ min pair separation; Xtrafficµ packet pair dispersion Dynamic bandwidth capacity (DBC) Iperf Available bandwidth = DBC – X-traffic Cross-traffic ABwE SLAC to Caltech Mar 19, 2004

  16. Anomalous Event Detection • Too many graphs to scan by hand, need to automate • SLAC Caltech link performance dropped by factor 5 for ~ month before noticed, fixed within 4 hours of reporting • Looking for long-term step down changes in bandwidth • Use modified “plateau” algorithm from NLANR • Divide data into history & trigger buffer • If y < mh – b * sh then trigger, else history (b = 2) • When trigger buffer fills: if mt < d * mh, then have an event

  17. Anomalous Event Detection • Length of trigger buffer (t) determines how long a step down must last before being interesting, we use 1 to 3 hours • E.g. 20 mins saw 9 events, 40mins saw 3, 60mins none • Works well unless strong (>40%) diurnal changes • Next step incorporate diurnal checks Events caused by application on Caltech host (not network related)

  18. Putting it together SLAC ESnet P P P CENIC Abilene P P Supernet SOX

  19. Future plans • Looking for funding… • Integrate it all • Improve distribution and management tools • Add monitoring sites e.g. HENP tier 0 & 1 sites such as CERN, BNL, IN2P3, DESY …; ESnet, StarLight, Caltech … • Add extra functionality: • Improved event detection • include diurnals, multivariate • Filter alerts • Upon detecting anomaly gather relevant information (network, host etc.) including on-demand measurements (e.g. NDT) and prepare web page & email • Improved web services access

  20. Thanks: Development • Jiri Navratil(Prague) – bandwidth estimation (ABwE) • Paola Grosso(SLAC) & Warren Matthews(GATech) - web services • Maxim Grigoriev(FNAL) – event detection, IEPM visualization, major monitoring site • Ruchi Gupta(Stanford) – event visualization • Prof Arshad Ali & Fahad Khalid(NIIT, Pakistan) – data collection after event • Rich Carlson(I2), NDT

  21. Thanks: on-going • Foreign: • Andrew Daviel (TRIUMF), Simon Leinen (SWITCH), Olivier Martin (CERN), Sven Ubik (CESnet), Kars Ohrenberg (DESY), Bruno Hoeft (FZK), Dominique (IN2P3), Fabrizio Coccetti (INFN), Cristina Bulfon (INFN), Yukio Karita (KEK), Takashi Ichihara (RIKEN), Yoshinori Kitasuji (APAN), Antony Antony (NIKHEF), Arshad Ali (NIIT), Serge Belov (BINP), Robin Tasker (DL & RAL), Yee Ting Lee (UCL), Richard Hughes-Jones (Manchester) • US • Shawn McKee (Michigan), Tom Hacker (Michigan), Eric Boyd (I2), Stanislav Shalunov (SOX), George Uhl (GSFC), Brian Tierney (LBNL), John Hicks (Indiana), John Estabrook (UIUC), Maxim Grigoriev (FNAL), Joe Izen (UT Dallas), Chris Griffin (U Florida), Tom Dunigan (ORNL), Dantong Yu (BNL), Suresh Singh (Caltech), Chip Watsom (JLab), Robert Lukens (JLab), Shane Canon (NERSC), Kevin Walsh (SDSC), David Lapsley (MIT/Haystack/ISI-E)

  22. More information • IEPM-BW home page • http://www-iepm.slac.stanford.edu/bw/ • Comparison of Internet E2E Measurement infrastructures; • http://www-iepm.slac.stanford.edu/grp/scs/net/proposals/infra-mon.html • ABwE lightweight bandwidth estimation • http://www-iepm.slac.stanford.edu/abing/ • Anomalous Event Detection • www.slac.stanford.edu/grp/scs/net/papers/sigcomm2004/nts26-logg.pdf • IEPM Web Services • http://www-iepm.slac.stanford.edu/tools/web_services/

  23. Extra Slides

  24. Web Services • See http://www-iepm.slac.stanford.edu/tools/web_services/ • Working for: RTT, loss, capacity, available bandwidth, achievable throughput • No schema defined for traceroute (hop-list) • PingER • Definition WSDL • http://www-iepm.slac.stanford.edu/tools/soap/wsdl/PINGER_profile.wsdl • path.delay.roundTrip ms (min/avg/max + RTTs), • path.loss.roundTrip • IPDV(ms), • <definitions name="PINGER" targetNamespace="http://www-iepm.slac.stanford.edu/tools/soap/wsdl/PINGER_profile.wsdl"> • <message name="GetPathDelayRoundTripInput"> • <part name="startTime" type="xsd:string"/> • <part name="endTime" type="xsd:string"/> • <part name="destination" type="xsd:string"/> • </message> • Also dups, out of order, IPDV, TCP thru estimate • Require to provide packet size, units, timestamp, sce, dst • path.bandwidth.available, path.bandwidth.utilized, path.bandwidth.capacity • Mainly for recent data, need to make real time data accessible • Used by MonALISA so need coordination to change definitions

  25. Perl access to PingER

  26. PingER WSDL

  27. Output from script

  28. Perl AMP traceroute

  29. AMP traceroute output

  30. Intermediate term access • Provide access to analyzed data in tables via .tsv format download from web pages.

  31. Bulk Data • For long term detailed data, we tar and zip the data on demand. Mainly for PingER data.

  32. New CENIC path 1000 Mbits/s Forward Routing changes AbWE Iperf back to new CENIC path Bbftp Dropto 100 Mbits/s by Routing (BGP) errors Iperf 1 stream RTT Drop to 622 Mbits/s path Reverse Routing changes 28 days bandwidth history. During this time we can see several different situations caused by different routing from SLAC to CALTECH Scatter plot graphs of Iperf versus ABw on different paths (range 20–800 Mbits/s) showing agreement of two methods (28 days history)

  33. Changes in network topology (BGP) can result in dramatic changes in performance Hour Samples of traceroute trees generated from the table Los-Nettos (100Mbps) Remote host Snapshot of traceroute summary table Notes: 1. Caltech misrouted via Los-Nettos 100Mbps commercial net 14:00-17:00 2. ESnet/GEANT working on routes from 2:00 to 14:00 3. A previous occurrence went un-noticed for 2 months 4. Next step is to auto detect and notify Drop in performance (From original path: SLAC-CENIC-Caltech to SLAC-Esnet-LosNettos (100Mbps) -Caltech ) Back to original path Dynamic BW capacity (DBC) Changes detected by IEPM-Iperfand AbWE Mbits/s Available BW = (DBC-XT) Cross-traffic (XT) Esnet-LosNettos segment in the path (100 Mbits/s) ABwE measurement one/minute for 24 hours Thurs Oct 9 9:00am to Fri Oct 10 9:01am

More Related