Download
1 / 68
680 likes | 699 Views
Learn about the self-similar nature of Ethernet traffic and its implications. Explore the importance, problems with current models, and consequences of self-similarity in network traffic behavior. Discover definitions, properties, and methods to measure self-similarity.
E N D
On the Self-Similar Nature of Ethernet TrafficWill E. Leland, Walter Willinger and Daniel V. Wilson BELLCOREMurad S. Taqqu BU Analysis and Prediction of the Dynamic Behavior of Applications, Hosts, and Networks
Overview • What is Self Similarity? • Ethernet Traffic is Self-Similar • Source of Self Similarity • Implications of Self Similarity
Intuition of Self-Similarity • Something “feels the same” regardless of scale (also called fractals)
What is Self-Similarity? In case of stochastic objects like time-series, self-similarity is used in the distributional sense
The Famous Data • Leland and Wilson collected hundreds of millions of Ethernet packets without loss and with recorded time-stamps accurate to within 100µs. • Data collected from several Ethernet LAN’s at the Bellcore Morristown Research and Engineering Center at different times over the course of approximately 4 years.
Why is Self-Similarity Important? • Recently, network packet traffic has been identified as being self-similar. • Current network traffic modeling using Poisson distributing (etc.) does not take into account the self-similar nature of traffic. • This leads to inaccurate modeling which, when applied to a huge network like the Internet, can lead to huge financial losses.
Problems with Current Models • A Poisson process • When observed on a fine time scale will appear bursty • When aggregated on a coarse time scale will flatten (smooth) to white noise • A Self-Similar (fractal) process • When aggregated over wide range of time scales will maintain its bursty characteristic
Consequences of Self-Similarity • Traffic has similar statistical properties at a range of timescales: ms, secs, mins, hrs, days • Merging of traffic (as in a statistical multiplexer) does not result in smoothing of traffic Aggregation Bursty Data Streams Bursty Aggregate Streams
Definitions and Properties • Long-range Dependence • autocorrelation decays slowly • Hurst Parameter • Developed by Harold Hurst (1965) • H is a measure of “burstiness” • also considered a measure of self-similarity • 0 < H < 1 • H increases as traffic increases
Definitions and Properties Cont.’d • low, medium, and high traffic hours • as traffic increases, the Hurst parameter increases • i.e., traffic becomes more self-similar
Properties of Self Similarity • X = (Xt : t = 0, 1, 2, ….) is covariance stationary random process (i.e. Cov(Xt,Xt+k) does not depend on t for all k) • Let X(m)={Xk(m)} denote the new process obtained by averaging the original series X in non-overlapping sub-blocks of size m. • Mean , variance 2 • Suppose that Autocorrelation Functionr(k) k-β, 0<β<1 E.g. X(1)= 4,12,34,2,-6,18,21,35Then X(2)=8,18,6,28X(4)=13,17
Auto-correlation Definition • X is exactly second-orderself-similar if • The aggregated processes have the same autocorrelation structure as X. i.e. • r (m) (k) = r(k), k0 for all m =1,2, … • X is [asymptotically] second-orderself-similar ifthe above holds when [ r (m) (k) r(k), m ] • Most striking feature of self-similarity: Correlation structures of the aggregated process do not degenerate as m
Traditional Models • This is in contrast to traditional models • Correlation structures of their aggregated processes degenerate as m i.e. r (m) (k) 0 as m , for k = 1,2,3,... • Example: • Poisson Distribution • Self-Similar Distribution
Long Range Dependence • Processes with Long Range Dependence are characterized by an autocorrelation function that decays hyperbolically as k increases • Important Property: This is also called non-summability of correlation
Intuition • Short-range processes: • Exponential Decay of autocorrelations , i.e.: • r(k) ~ pk , as k , 0 < p < 1 • Summation is finite • The intuition behind long-range dependence: • While high-lag correlations are all individually small, their cumulative affect is important • Gives rise to features drastically different from conventional short-range dependent processes
The Measure of Self-Similarity • Hurst Parameter H , 0.5 < H < 1 • Three approaches to estimate H (Based on properties of self-similar processes) • Variance Analysis of aggregated processes • Analysis of Rescaled Range (R/S) statistic for different block sizes • A Whittle Estimator
Variance Analysis • Variance of aggregated processes decays as: • Var(X(m)) = am-b as m inf, • For short range dependent processes (e.g. Poisson Process), • Var(X(m)) = am-1 as m inf, • Plot Var(X(m)) against m on a log-log plot • Slope > -1 indicative of self-similarity
The R/S statistic For a given set of observations, Rescaled Adjusted Range or R/S statistic is given by where
Example • Xk = 14,1,3,5,10,3 • Mean = 36/6 = 6 • W1 =14-(1.6 )=8 • W2 =15-(2.6 )=3 • W3 =18-(3.6 )=0 • W4 =23-(4.6 )=-1 • W5 =33-(5.6 )=3 • W6 =36-(6.6 )=0 R/S = 1/S*[8-(-1)] = 9/S
The Hurst Effect • For self-similar data, rescaled range or R/S statistic grows according to cnH • H = Hurst Paramater, > 0.5 • For short-range processes , • R/S statistic ~ dn0.5 • History: The Nile river • In the 1940-50’s, Harold Edwin Hurst studies the 800-year record of flooding along the Nile river. • (yearly minimum water level) • Finds long-range dependence.
Whittle Estimator • Provides a confidence interval • Property: Any long range dependent process approaches FGN, when aggregated to a certain level • Test the aggregated observations to ensure that it has converged to the normal distribution
Recap • Self-similarity manifests itself in several equivalent fashions: • Non-degenerate autocorrelations • Slowly decaying variance • Long range dependence • Hurst effect
Plots Showing Self-Similarity (Ⅰ) H=1 H=0.5 H=0.5 Estimate H 0.8
Plots Showing Self-Similarity (Ⅱ) High Traffic 5.0%-30.7% Mid Traffic 3.4%-18.4% Low Traffic 1.3%-10.4% Higher Traffic, Higher H
H : A Function of Network Utilization • Observation shows “contrary to Poisson” • Network Utilization H • As we shall see shortly, H measures traffic burstiness As number of Ethernet users increases, the resulting aggregate traffic becomes burstier instead of smoother
Difference in low traffic H values • Pre-1990: host-to-host workgroup traffic • Post-1990: Router-to-router traffic • Low period router-to-router traffic consists mostly of machine-generated packets • Tend to form a smoother arrival stream, than low period host-to-host traffic
H : Measuring “Burstiness” • Intuitive explanation using M/G/ Model • As α 1, service time is more variable, easier to generate burst • Increasing H !
Summary • Ethernet LAN traffic is statistically self-similar • H : the degree of self-similarity • H : a function of utilization • H : a measure of “burstiness” • Models like Poisson are not able to capture self-similarity
Discussions • How to explain self-similarity ? • Heavy tailed file sizes • How this would impact existing performance? • Limited effectiveness of buffering • Effectiveness of FEC • How to adapt to self-similarity? • Prediction • Adaptive FEC
Introduction • The superposition of many ON/OFF sources whose ON-periods and OFF-periods exhibit the Noah Effect produces aggregate network traffic that features the Joseph Effect. • Noah Effect: high variability or infinite variance • Joseph Effect: self-similar or long-range dependent Also known as packet train models
The Noah Effect • Noah Effect is the essential point of departure from traditional to self-similar traffic modeling • Results in highly variable ON-OFF periods : Train length and inter-train distances can be very large with non-negligible probabilities • Infinite Variance Syndrome : Many naturally occurring phenomenon can be well described with infinite variance distributions • Heavy-tail distributions, parameter
Existing Models • Traditional traffic models: finite variance ON/OFF source models • Superposition of such sourcesbehaves like white noise, with only short range correlations
Idealized ON/OFF Model • Lengths of ON- and OFF periods are iid positive random variables, Uk • Suppose that U has a hyperbolic tail distribution, • Property (1) is the infinite variance syndrome or the Noah Effect. • 2 implies E(U2) = • > 1 ensures that E(U) < , and that S0 is not infinite
