1 / 80

PingER: Navigating the web site and mining the data

PingER: Navigating the web site and mining the data. DRAFT. Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013. Agenda. Navigating the PingER web site Accessing and viewing analyzed data Downloading and mining with Excel etc. Managing the system.

merrill
Download Presentation

PingER: Navigating the web site and mining the data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. PingER: Navigating the web site and mining the data DRAFT Les CottrellSLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013

  2. Agenda • Navigating the PingER web site • Accessing and viewing analyzed data • Downloading and mining with Excel etc. • Managing the system

  3. PingER home site • Set bookmark at http://www-wanmon.slac.stanford.edu/cgi-wrap/pingtable.p • Prefer Chrome or Firefox, mostly works with MSIE • Public face ofPingER

  4. Home site menu bar: Introduction • Provides: • Upper level information • An IEEE publication • Technical details • Etc.

  5. Home page: Visualizations • Click on African Internet Weather • Congestion wave • Monitoring sites/ participants • Hosts per country by region

  6. Access to PingER site map • Provides site map & • Geographical deployment maps PingER site map

  7. PingER site map

  8. Home page Icons access • Pingtable: workhorse of PingER analysis • Summary table of country to country connections • Google map of PingER hosts • Google Explorer • Motion Metrics Bubble chart

  9. Form Table Pingtable Help • This is the workhorse • www.slac.stanford.edu/cgi-wrap/pingtable.pl UNIMAS Workshop, Sarawak, Dec 2012

  10. Pingtable: Help What to click on Coloring by thresholds Help on table content etc., Technical assistance on metrics meanings etc, UNIMAS Workshop, Sarawak, Dec 2012

  11. Host labeling • Host labeled <country>.<site>.. (most important 1st). • Assist in sorting etc. UNIMAS Workshop, Sarawak, Dec 2012

  12. Pingtable: Scroll right for host names etc. • Get Internet host name, TLD and region for each host pair UNIMAS Workshop, Sarawak, Dec 2012

  13. Pingtable: Form selection eGY Africa 2012 Workshop, Nairobi Oct 2012

  14. Pingtable: Metric selection • Pull down to select from Avg RTT, jitter (ipdv), loss, throughput, MOS etc. UNIMAS Workshop, Sarawak, Dec 2012

  15. Pingtable: Select tick (time window) • Pull down to select: hourly, daily, monthly, yearly etc. UNIMAS Workshop, Sarawak, Dec 2012

  16. Pingtable: Select From (Monitor(s)) • Pull down to select monitoring node(s): single monitoring host or all monitors in a country or region etc. UNIMAS Workshop, Sarawak, Dec 2012

  17. Pingtable: Select To (monitored host or site) • Pull down to select To node(s): single monitored host/site or all hosts/sites in a country or region etc. UNIMAS Workshop, Sarawak, Dec 2012

  18. Aggregate all hosts in a site • Pull down to select whether to aggregate all hosts in a site into one measurement UNIMAS Workshop, Sarawak, Dec 2012

  19. Select packet size UNIMAS Workshop, Sarawak, Dec 2012

  20. Pingtable: Select host type • Pull down to select type of hosts to display: All, Monitoring, Beacons, Remote UNIMAS Workshop, Sarawak, Dec 2012

  21. Pingtable: other choices • Change dataset: no longer any other choices • Data points: removes pairs with little data • Filter: Removes known anomalous data UNIMAS Workshop, Sarawak, Dec 2012

  22. Pingtable: Load selected data • Click on Load into table and wait while data loads and display is rendered UNIMAS Workshop, Sarawak, Dec 2012

  23. Pingtable: Active links UNIMAS Workshop, Sarawak, Dec 2012

  24. Pingtable: Sorting • Click on column heading: • Sorts by host name, host type (? Monitor|Beacon|remote), metric value, TLD, Region UNIMAS Workshop, Sarawak, Dec 2012

  25. Pingtable ? • Click ? for more info • Details on hosts, • Latest raw data, • Format (see http://www-iepm.slac.stanford.edu/pinger/tools/retrievedata.html#rawdataformat) • Summarized data UNIMAS Workshop, Sarawak, Dec 2012

  26. Pingtable (?): More information on pair • Select ‘From’ as EDU.SLAC.STANFORD.N3 & Load into Table • Click on ? • Provides info from database on hosts, e.g.: • Lat/lon, location, web site, comments etc. UNIMAS Workshop, Sarawak, Dec 2012

  27. Pingtable: Rawdata • Useful for seeing exactly when a host disappears or is reachable again by PingER , e.g. in case of events such as loss of Internet connectivity due to earthquakes, cable cuts, uprisings UNIMAS Workshop, Sarawak, Dec 2012

  28. Pingtable(?): Summarized data Latest summarized data from (pinger.slac.stanford.edu): click here • Provides summary statistics (min, avg, median max) for all chosen remote hosts seen from monitor, for 100 &1000 Byte pings • Plus the slope of Bytes vs. RTT UNIMAS Workshop, Sarawak, Dec 2012

  29. Graphs of usage (via ?) • Time series plot of RTT & Loss last 7 days including today so far from MY.UNIMAS.N4(pinger.unimas.my) to MY.UMT.EDU.N1(www.umt.edu.my):  click here, add csv file. 100% loss =unreachable UNIMAS Workshop, Sarawak, Dec 2012

  30. Graphs of RTT • Valuable if do not have Smokeping graphs, e.g. for non SLAC monitors • One day I will fix the tick marks. • Also add csv file and cut and paste into Excel for more elegant plots UNIMAS Workshop, Sarawak, Dec 2012

  31. Pingtable: (?) Dynamic on demand traceoute • Uses traceroute server available at most PingER monitors • Click on Current traceroute using  http://www-wanmon.slac.stanford.edu/cgi-bin/nph-traceroute.pl?target=www.uda.ad UNIMAS Workshop, Sarawak, Dec 2012

  32. Traceroutes • Measured once per day UNIMAS Workshop, Sarawak, Dec 2012

  33. Traceroutes: Form • Can select 3 days to view concurrently to look for important changes • Also can select the remote host • Currently only available for SLAC monitoring host UNIMAS Workshop, Sarawak, Dec 2012

  34. Traceroute: format Traceroute for First selected date www-wanmon.slac.stanford.edu_to_www.uda.ad_2012_11_30 1 rtr-servcore1-serv01-webserv.slac.stanford.edu (134.79.197.130) 0.452 ms2 rtr-core2-p2p-servcore1.slac.stanford.edu (134.79.252.162) 0.336 ms3 rtr-border1-p2p-core2.slac.stanford.edu (134.79.252.137) 0.388 ms4 slac-mr2-p2p-rtr-border1.slac.stanford.edu (192.68.191.245) 0.284 ms5 sunnsdn2-ip-slacmr2.es.net (134.55.217.2) 0.622 ms6 sunncr1-sunnsdn2.es.net (134.55.209.98) 0.714 ms7 eqxsjrt1-te-sunncr1.es.net (134.55.38.146) 1.138 ms8 *9 xe-4-1-0.par72.ip4.tinet.net(89.149.181.146) 151.131 ms10 interoute-gw.ip4.tinet.net (77.67.75.238) 209.653 ms11 ae1-0.mad-001-score-1-re1.interoute.net (89.202.161.21) 178.515 ms12 ae0-0.mad-001-score-2-re0.interoute.net (89.202.161.18) 178.086 ms13 ae1-0.bcn-002-score-1-re1.interoute.net (217.118.118.237) 178.303 ms14 (195.81.193.146) 176.546 ms15 *16 * UNIMAS Workshop, Sarawak, Dec 2012

  35. More Traceroutes • Background information on traceroute servers • http://www.slac.stanford.edu/comp/net/wan-mon/traceroute-srv.html • List of servers by country: • http://www.traceroute.org/ UNIMAS Workshop, Sarawak, Dec 2012

  36. PingER traceroute servers API • Most PingER installations have traceroute servers. • Perl API and web accessible xml file for database can be found by accessing file http://www-iepm.slac.stanford.edu/pinger/pingerworld/nodes.cf, see comment at start • This can be used to find out about PingER hosts & in particular PingER traceroute servers

  37. Pingtable: Smokeping • Click on the SLAC monitoring host in a line • Not available for other monitors UNIMAS Workshop, Sarawak, Dec 2012

  38. Pingtable: Smokeping graph • Background is colored for loss (see legend) • Black not reachable • Red 50%-90% loss • Magenta 20%-40% loss • Orange 5%-10% loss • Light yellow 0%-5% loss • Cyan 0% loss UNIMAS Workshop, Sarawak, Dec 2012

  39. Pingtable: Smokeping Blue line average RTT, Smoke=jitter X axis in GMT Y axis in seconds (m = msec) Start and end times given in legend Summary statistics given in legend UNIMAS Workshop, Sarawak, Dec 2012

  40. Pingtable: Smokeping graph Form • Allows time window selection (also icons) • Size of graph, manual scaling • Colors for losses UNIMAS Workshop, Sarawak, Dec 2012

  41. Pingtable:Summary table • At the bottom provides: • Summary stats (min, median, 95%, # pairs, etc.) for each time interval for chosen metric • Can click on columns heading to download statistic UNIMAS Workshop, Sarawak, Dec 2012

  42. Pingtable.pl extract summary • Summary table at end (e.g. access via wget) - Buried in the output from pingtable.pl is a comment line containing the summary data, forma (… = more of same)t: • <!--?+Oct2007+...+Nov2005,+0.141+...+737.923,,--> • <!--?+1998+...+2010,.+17.1429+...+35.7714+32.9469,--> • Each line of data is separated by a comma (,) • within a line each token is separated by a +. • The lines in turn are those from the 'Summary for Sites' table at the bottom of pingtable.pl output. • The first line gives the date, the second the min, the 3rd the 25%, the 4th the average, the 5th (i.e. \$lines[4]) line the median etc. • If there is no data for a token then there is an adjacent ++. UNIMAS Workshop, Sarawak, Dec 2012

  43. Download Pingtable data • First choose some simple, interesting data: • Select From = Malaysia • Select To = Malaysia • Metric = Average RTT • Tick-type = last 60 days UNIMAS Workshop, Sarawak, Dec 2012

  44. Pingtable: Downloading data & further analysis • Click here This report is also provided in downloadable tab-separated-value (.tsv) format for use with Excel. Chrome: Appears at bottom left of browser window. Drag it to desktop Firefox: box pops up, choose save. It should be in Downloads. Can try Start and Search for it MSIE: box pops up, choose ‘save as’ and save on the desktop UNIMAS Workshop, Sarawak, Dec 2012

  45. Open downloaded file with Excel • Right click the pingtable.pl icon (probably on desktop). • Choose “Open with” => “Choose Microsoft Excel”. • Open the file Dots = no data UNIMAS Workshop, Sarawak, Dec 2012

  46. Preamble to Excel analysis • Insert the URL where in a newline at the start • Delete columns A & C • Select all (e.g. click on top left hand corner of table) • Click on Replace in Home menu • Replace all cells with only a period with a null • Fix the date: take the Left most add in Excel format, the take the cell in next column to the right set =left one -1 and replicate right UNIMAS Workshop, Sarawak, Dec 2012

  47. Draw chart • Select relevant data (e.g. a2:bf13) • Click on Scatter chart under Insert Tab • Now it’s a matter of improving the format to better understand the data UNIMAS Workshop, Sarawak, Dec 2012

  48. Icons access • Table of country to country connections • Maps of metric performance by Country • Maps of number of PingER hosts by country • Google map of PingER hosts • Motion Metrics Bubble chart UNIMAS Workshop, Sarawak, Dec 2012

  49. PingER Summary Table • URL= www.slac.stanford.edu/cgi-wrap/table.pl • Patience takes a long time Metric Help Choose area aggregation Month UNIMAS Workshop, Sarawak, Dec 2012

  50. Drilling down • Colors represent quality • Click on headings & values to drill down to pingtable UNIMAS Workshop, Sarawak, Dec 2012

More Related