Forecasting Network Performance

1. Forecasting Network Performance Les Cottrell, Grid Performance Workshop, Edinburgh, June 22-34, 2005 http://www.slac.stanford.edu/grp/scs/net/talk05/predict-edinburgh05.ppt Partially funded by DOE/MICS for Internet End-to-end Performance Monitoring (IEPM)

2. Outline • Why do we want forecasting & anomaly detection? • What are we using for the input data • And what are the problems • How do we make forecasts, detect anomaly? • First approaches • The real world • Results • Conclusions & Futures • Possible uses

3. Uses of Techniques • Automated problem identification: • Alerts for network administrators, e.g. • Bandwidth changes in time-series, iperf, SNMP • Alerts for systems people • OS/Host metrics • Anomalies for security • Forecasts (are a fallout of the techniques) for Grid Middleware, e.g. replica manager, data placement

4. Data

5. Using Active IEPM-BW measurements • Focus on high performance for a few hosts needing to send data to a small number of collaborator sites, e.g. HEP tiered model • Makes regular measurements • Ping (RTT, connectivity), traceroute • pathchirp, ABwE (packet pair dispersion) • iperf (single & multi-stream), thrulay, • Bbftp (file transfer application) • Looking at GridFTP but complex requiring renewing certificates • Lots of analysis and visualization • Running at CERN, SLAC, FNAL, BNL, Caltech to about 40 remote sites • http://www.slac.stanford.edu/comp/net/iepm-bw.slac.stanford.edu/slac_wan_bw_tests.html

6. ABwE/abing • Uses packet pair dispersion of 20 packets to provide: • Capacity, X-traffic, available bandwidth • At 3 minute intervals • Very noisy time series data Moving averaged over 1 hour Capacity

7. Pathchirp/Rice/INCITE • From PAM paper, pathchirp more accurate but • Ten times as long (10s vs 1s) • More network traffic (~factor of 10) • Pathload factor of 10 again more • IEPM-BW now supports both

8. BUT… • Packet pair dispersion relies on accurate timing of inter packet separation • At > 1Gbps this is getting beyond resolution of Unix clocks • AND 10GE NICs are offloading function • Coalescing interrupts, Large Send & Receive Offload, TOE • Need to work with TOE vendors • Turn off offload • Do timing in NICs

9. Iperf vs thrulay Thrulay Maximum RTT • Iperf has multi streams • Thrulay more manageable & gives RTT • They agree well • Throughput ~ 1/avg(RTT) Average RTT RTT ms Minimum RTT Achievable throughput Mbits/s

10. BUT… • At 10Gbits/s on transatlantic path Slow start takes over 6 seconds • To get 90% of measurement in congestion avoidance need to measure for 1 minute (5.25 GBytes at 7Gbits/s (today’s typical performance)

11. Passive • Use Netflow records at border • Per flow provide start/stop time, bytes/packets etc. • Collect records for several weeks • Divide by remote site, add parallel streams • Fold data onto one week, see bands at known capacities

12. Netflow 2/2 • Use existing traffic, no extra traffic • Works on fast networks

13. Forecasting and Anomaly detection

14. Anomaly Detection • Anomaly is when the actual value significantly differs from the expected value • So need forecasts to find anomalies • Focus has been on ABwE time-series measurements: • Packet pair dispersion on 20 packets • Send 20 packet pairs back to back and measure one-way packet separation at remote end • Minimum gives an indication of bottleneck capacity of link • Measurement each 3 minutes • Low network impact BUT very noisy so hard test case

15. Plateau, most intuitive • Each observation: • If outside history buffer mean mh±b*sh then add to trigger buffer • Else add to history, and remove oldest from trigger buffer • When trigger buffer > t points then trigger issued • Check if (mh -mt) / mh > D& 90% trigger in last T mins then have trigger • Move trigger buffer to history buffer Observations Event * • = history length = 1 day, t = trigger length = 3 hours • = standard deviations = 2 Trigger % full History mean History mean – 2 * stdev

16. K-S • For each observation: for the previous 100 observations with next 100 observations • Compare the vertical difference in CDFs • How does it differ from random CDFs • Expressed as % difference Compare K-S with Plateau

17. Compare • Results between K-S & plateau very similar, using K-S coefficient threshold = 70% • Current plateau only finds negative changes • Useful to see when condition returns to normal • K-S implemented in C and executes faster than Plateau (in Perl), depends on parameters • K-S more formalized • Plateau and K-S work well for non seasonal observations (e.g. small changes day/night)

18. Seasons & false alerts • Congestion on Monday following a quiet weekend causes a high forecast, gives an alert • Also a history buffer of not a day causes History mean to be out of sync with observations

19. Diurnal Variation People arriving at work between 19:00 & 22:00 PDT (7:00 & 10:00 PK time) cause sudden drop in dynamic capacity

20. Effect on events • Change in bandwidth (drops) between 19:00 & 22:00 Pacific Time (7:00-10:00am PK time) • Causes more anomalous events around this time

21. Seasonal Changes • Use Holt-Winters (H-W) technique: • Uses triple exponential weighted moving average • EWMA(i) = Obs(i) * a + (1-a) * EWMA(i-1) • Three terms each with its own parameter (a, b, ) that take into account local smoothing, long term seasonal smoothing, and trends

22. H-W Implementation • Need regularly spaced data (else going back one season is difficult, and gets out of sync): • Interpolate data: select bin size • Average points in bin • If no points in first week bin then get data from future weeks • For following weeks, missing data bins filled from previous week • Initial values for smoothing from NIST “Engineering Statistics Handbook” • Choose parms by minimizing (1/N)Σ(Ft-yt)2 • Ft=forecast for time t as function of parameters, yt= observation at time t

23. H-W Implementation • Three implementations evaluated (two new) • FNAL (Maxim Grigoriev) • Inspiration for evaluating this method • Part of RRD (Brutlag) • Limited control over what it produces and how it works • SLAC • Implemented NIST formulation, different formulation/parameter values from Brutlag/FNAL, also added minimize sums of squares to get parms

24. Results

25. Example • Local smoothing 99% weight for last 24 hours • Linear trend 50% last 24 hours • Seasonal mainly from last week, but includes several weeks • Within an 80 minute window, 80% points outside deviation envelope ≡ event 1 hr avg Observations Deviations Forecast Weekend Weekdays

26. Evaluation • Created a library of time series for 100 days from June through Sep 2004 for 40 hosts • Analyzed using Plateau and saved all events where trigger buffer filled (no filters on size of step) • 23 hosts had 120 candidate events • Event types: steps; diurnal changes; congestion from cron jobs, bandwidth tests, flash crowds • Classify ~120 events as to whether interesting • Large, sharp drop in bandwidth, persist for >> 3hrs

27. Results • K-S shows similar results to Plateau • As adjust parameters to reduce false positives then increase missed events • E.g. for plateau with trigger buffer = 3 hrs filled to 90% in < 220 minutes, history buffer=1 day, effect of threshold D=(mh-mt)/mh Plateau (b=2) K-S with ± 100 observations

28. Conclusions • A few paths (10%) have strong seasonal effects • Plateau & K-S work well if only weak seasonal effects • K-S detects both step downs & up, also gives accurate time estimate of event (good for correlations) • H-W promising for seasonal effects, but • Is more complex, and requires more parameters which may not be easy to estimate • Requires regular data (interpolation step) • CPU time can depend critically on parameters chosen, e.g. increasing K-S range from ±100 to say ±400 increases CPU time by factor 14 • H-W works, still need to quantify its effectiveness • Looking at PCA to evaluate multiple metrics simultaneously (e.g. fwd & bwd traffic, RTT, multiple paths) AND multiple paths

29. Future Work • Future Development in PCA • Enable looking at multiple measurements simultaneously • E.g. RTT, loss, capacity …; multiple routes • Neural networks to interpolate heavyweight/infrequent measurements based on light weight more frequent • Continue Netflow passive exploration

30. Some Uses: • Detect anomalies reliably (few false positives, few misses): • Make extra measurements related to anomaly, e.g. ping, traceroute, performance history etc. • Notify people (e.g. via email) • Forecast into future taking account diurnal changes: • Make long-term (hours – days) integrated estimates of performance with probabilities • Use for data location selection

31. Apply forecasts to Router utilizations to find bottlenecks • Get measurements from Internet2/ESnet/Geant SONAR project via NMWG web services • Save as time series, forecast for each interface • For given path and duration forecast most probable bottlenecks • Use MPLS to apply QoS at bottlenecks (rather than for the entire path) for selected applications

32. More information • SLAC Plateau implementation • www.acm.org/sigs/sigcomm/sigcomm2004/workshop_papers/nts26-logg1.pdf • SLAC H-W implementation • www-iepm.slac.stanford.edu/monitoring/forecast/hw.html • Eng. Statistics Handbook • http://www.itl.nist.gov/div898/handbook/pmc/section4/pmc435.htm • IEPM-BW Measurement Infrastructure • http://www-iepm.slac.stanford.edu/

33. Events • Can look at residuals (Ft – yt), or Χ2 • Could use K-S or plateau on: residuals, or on the local smoothing (i.e. after removing long term seasonal effects)

34. Mark Burgess Method • A two dimensional time-series approach in order to classify a periodic, adaptive threshold for service level anomaly detection • An iterative algorithm is applied to history analysis on this periodic time to provide a smooth roll-off in the significance of the data with time. • This method was originally designed to detect anomalous behavior on a single host.

35. Compare with KS Iperf from SLAC to Caltech – Feb & Mar 05 KS-Result KS Technique works Very well for the long Term anomalous Variations in internet End-to-end traffic. Mark Burgess technique detects the anomalies for each and every Unwanted huge spikes/variation (Real Time) Mark Burgess Tech-Result

36. PCA • PCA is a coordinate transformation method that maps a given set of data points onto new axes. These axes are called the principal axes or principal components. • For network anomaly detection PCA divides the data into normal & abnormal subspace • Procedure • Arrangement of data into matrix form • Zero meaning the matrix data • Calculating the covariance matrix • Calculate principal components • Application of the formulae (I-PPT)(data-matrix) yields the result. P is the matrix of Principal Components.

37. PCA Results PCA Results on SLAC-BINP (June-Sep, 2004) Due to 10% rise in dbcap Anomalous Good Events 10% rise in RTT • Caught all the events that were detected by HW, Plateau and KS • Can work on multiple parameters • Tested PCA on six routes so far SLAC-FZK, SLAC-DESY, SLAC-CALTECH, SLAC-NIIT, SLAC-BINP, SLAC-UMICH