230 likes | 443 Views
CS 221. Analysis of Algorithms Empirical Analysis Instructor: Don McLaughlin. Empirical Analysis of Algorithms. Experimentation Carry out experiments to see how the algorithm behaves/grows remember: we are interested in the algorithm, not its implementation on specific machine
E N D
CS 221 Analysis of Algorithms Empirical Analysis Instructor: Don McLaughlin
Empirical Analysis of Algorithms • Experimentation • Carry out experiments to see how the algorithm behaves/grows • remember: we are interested in the algorithm, not its implementation on specific machine • Analysis should be platform/device independent
Empirical Analysis of Algorithms • Experimentation Plan • What is the experiment’s purpose? • What kind of efficiency are you interested in? • What kind of metric will you use to measure it? • What are the characters of the input? • Code algorithm for execution and experimentation • Generate inputs • Run code expressing algorithm with sample data • Collect data at each level of input • Analyze the results
Empirical Analysis of Algorithms • Experimentation Plan • What kind of efficiency are you interested in? • Run-time (we have talked a lot about this) • memory usage • Event counts
Empirical Analysis of Algorithms • Experimentation Plan • Metrics • How will you measure it? • What is wrong with Walltime? • What is wrong with CPU time? • Memory references – a data intensive application – counting memory references might be a good measure of algorthm’s efficiency. • Comparisons – in some algorithms (like sorting or searching) the comparison of two values is the dominant operation – counting them will give a measure that may be correlated with the algorithm’s efficiency • Counting arithmetic operations might give a serviceable measure of algorithm’s performance
Empirical Analysis of Algorithms • Experimentation Plan • Must have enough input samples to generate statistically valid results. • Sample inputs should range over the range of real data for the problem domain • Sample data should be representative of real data • Sample distribution should be consistent with the distribution of data in the real problem domain
Empirical Analysis of Algorithms • Experimentation Plan • Coding • Good code takes skill and experience • Code should be a concise implementation of the algorithm • Same is true of alternative algorithms • Big deal, especially when comparing algorithms • is code true to algorithm? • is code expressed as efficiently as algorithm?
Empirical Analysis of Algorithms • Data Analysis • Regression Analysis • T as dependent variable • n as independent variable • in other words • Testing the degree of fit • T = f(n) • Linear Regression analysis • that T(n) is O(n) • non-linear regression techniques • that T(n) is O(n2), O(n3),…
Empirical Analysis of Algorithms • Data Analysis • Regression Analysis • To test a derived theoretical T(n) • Determine sample of inputs for n • calculated T(n) • Run Regression Analysis of T(n) on n • If correlation not very high • T(n) not good
Empirical Analysis of Algorithms • Data Analysis • Regression Analysis • To test a derived theoretical T(n) • Determine sample of inputs for n • calculated T(n) • Run Regression Analysis of T(n) on n • If correlation not very high • T(n) not good
Empirical Analysis of Algorithms • Data Analysis • Ratio Test • Derive F(n) = nc for main term of relation • Test if F(n) is Θ(nc) • Calculate F(n) across a range of selected sample inputs • Run test algorithm code across same range of sample inputs – call this T(n) • Calculate the ratio R(n) = T(n)/F(n) • Plot R(n) vs n
Empirical Analysis of Algorithms • Data Analysis • Ratio Test • If R(n) grows as n grows – • F(n) under estimates observed run-times T(n) • If R(n) goes to zero as n grows • F(n) overestimates observed run-times T(n) • If R(n) converges on a constant (other than zero) • then F(n) is a good estimate of T(n) (actual run-times) • … and the constant is an approximate estimate of the constant factor in F(n)
Empirical Analysis of Algorithms • Data Analysis • Power Test • Support you don’t already have theoretically derived estimate of T(n) • No worries mon • so,…
Empirical Analysis of Algorithms • Data Analysis • Power Test • Collect experimental data across range of n values • lets say n values are x… • …and observed run-times for each x is y • tranform x to x’ and y to y’ • x’ = log x • y’ = log y • Plot x’ vs y’ • and…
Empirical Analysis of Algorithms • Data Analysis • Power Test • if plotted (x’, y’) pairs form a tight fit to a line • this implies the run-time function F(n) is • bnc • and… • c = slope of line • b = y-intercept of line
Empirical Analysis of Algorithms • Data Analysis • Power Test • if plotted (x’, y’) pairs grows • then T(n) is super-polynomial • if plotted (x’, y’) pairs converge to a constant • then T(n) is sublinear
Empirical Analysis of Algorithms • Data Analysis • Power Test • Can only get approximate estimates for c and b • but still considered better than regression techniques… • regression techniques are over sensitive to noise
Empirical Analysis of Algorithms • CPU Performance Counters • http://ieeexplore.ieee.org/iel5/7558/20587/00952282.pdf?arnumber=952282
Empirical Analysis of Algorithms PAPI • What does PAPI mean? • designated hitter for the Boston Red Sox • Precision Approach Path Indicator • Paper and Pencil Interviewing • Performance Applications Programming Interface
Empirical Analysis of Algorithms PAPI • Many modern CPUs have special registers • for tracking hardware events • Hardware Performance Counters • Different for different CPU architectures • AMD-Athlon 4 • IA-64 - 4 • Pentium4 - 18
Empirical Analysis of Algorithms PAPI • Many modern CPUs have special registers • for tracking hardware events • Hardware Performance Counters • Number of cache misses • Number of instructions issued • Number of floating point instructions issued • Number of vector instructions issued
Empirical Analysis of Algorithms PAPI • Hardware Performance Counter Basics • http://perfsuite.ncsa.uiuc.edu/publications/LJ135/x27.html • RabbitA Performance Counters Libraryfor Intel/AMD Processors and Linux • http://www.scl.ameslab.gov/Projects/Rabbit/