CS 221

1. CS 221 Analysis of Algorithms Empirical Analysis Instructor: Don McLaughlin

2. Empirical Analysis of Algorithms • Experimentation • Carry out experiments to see how the algorithm behaves/grows • remember: we are interested in the algorithm, not its implementation on specific machine • Analysis should be platform/device independent

3. Empirical Analysis of Algorithms • Experimentation Plan • What is the experiment’s purpose? • What kind of efficiency are you interested in? • What kind of metric will you use to measure it? • What are the characters of the input? • Code algorithm for execution and experimentation • Generate inputs • Run code expressing algorithm with sample data • Collect data at each level of input • Analyze the results

4. Empirical Analysis of Algorithms • Experimentation Plan • What kind of efficiency are you interested in? • Run-time (we have talked a lot about this) • memory usage • Event counts

5. Empirical Analysis of Algorithms • Experimentation Plan • Metrics • How will you measure it? • What is wrong with Walltime? • What is wrong with CPU time? • Memory references – a data intensive application – counting memory references might be a good measure of algorthm’s efficiency. • Comparisons – in some algorithms (like sorting or searching) the comparison of two values is the dominant operation – counting them will give a measure that may be correlated with the algorithm’s efficiency • Counting arithmetic operations might give a serviceable measure of algorithm’s performance

6. Empirical Analysis of Algorithms • Experimentation Plan • Must have enough input samples to generate statistically valid results. • Sample inputs should range over the range of real data for the problem domain • Sample data should be representative of real data • Sample distribution should be consistent with the distribution of data in the real problem domain

7. Empirical Analysis of Algorithms • Experimentation Plan • Coding • Good code takes skill and experience • Code should be a concise implementation of the algorithm • Same is true of alternative algorithms • Big deal, especially when comparing algorithms • is code true to algorithm? • is code expressed as efficiently as algorithm?

8. Empirical Analysis of Algorithms • Data Analysis • Regression Analysis • T as dependent variable • n as independent variable • in other words • Testing the degree of fit • T = f(n) • Linear Regression analysis • that T(n) is O(n) • non-linear regression techniques • that T(n) is O(n2), O(n3),…

9. Empirical Analysis of Algorithms • Data Analysis • Regression Analysis • To test a derived theoretical T(n) • Determine sample of inputs for n • calculated T(n) • Run Regression Analysis of T(n) on n • If correlation not very high • T(n) not good

10. Empirical Analysis of Algorithms • Data Analysis • Regression Analysis • To test a derived theoretical T(n) • Determine sample of inputs for n • calculated T(n) • Run Regression Analysis of T(n) on n • If correlation not very high • T(n) not good

11. Empirical Analysis of Algorithms • Data Analysis • Ratio Test • Derive F(n) = nc for main term of relation • Test if F(n) is Θ(nc) • Calculate F(n) across a range of selected sample inputs • Run test algorithm code across same range of sample inputs – call this T(n) • Calculate the ratio R(n) = T(n)/F(n) • Plot R(n) vs n

12. Empirical Analysis of Algorithms • Data Analysis • Ratio Test • If R(n) grows as n grows – • F(n) under estimates observed run-times T(n) • If R(n) goes to zero as n grows • F(n) overestimates observed run-times T(n) • If R(n) converges on a constant (other than zero) • then F(n) is a good estimate of T(n) (actual run-times) • … and the constant is an approximate estimate of the constant factor in F(n)

13. Empirical Analysis of Algorithms • Data Analysis • Power Test • Support you don’t already have theoretically derived estimate of T(n) • No worries mon • so,…

14. Empirical Analysis of Algorithms • Data Analysis • Power Test • Collect experimental data across range of n values • lets say n values are x… • …and observed run-times for each x is y • tranform x to x’ and y to y’ • x’ = log x • y’ = log y • Plot x’ vs y’ • and…

15. Empirical Analysis of Algorithms • Data Analysis • Power Test • if plotted (x’, y’) pairs form a tight fit to a line • this implies the run-time function F(n) is • bnc • and… • c = slope of line • b = y-intercept of line

16. Empirical Analysis of Algorithms • Data Analysis • Power Test • if plotted (x’, y’) pairs grows • then T(n) is super-polynomial • if plotted (x’, y’) pairs converge to a constant • then T(n) is sublinear

17. Empirical Analysis of Algorithms • Data Analysis • Power Test • Can only get approximate estimates for c and b • but still considered better than regression techniques… • regression techniques are over sensitive to noise

18. Empirical Analysis of Algorithms • CPU Performance Counters • http://ieeexplore.ieee.org/iel5/7558/20587/00952282.pdf?arnumber=952282

19. Empirical Analysis of Algorithms PAPI • What does PAPI mean? • designated hitter for the Boston Red Sox • Precision Approach Path Indicator • Paper and Pencil Interviewing • Performance Applications Programming Interface

20. Empirical Analysis of Algorithms PAPI • Many modern CPUs have special registers • for tracking hardware events • Hardware Performance Counters • Different for different CPU architectures • AMD-Athlon 4 • IA-64 - 4 • Pentium4 - 18

21. Empirical Analysis of Algorithms PAPI • Many modern CPUs have special registers • for tracking hardware events • Hardware Performance Counters • Number of cache misses • Number of instructions issued • Number of floating point instructions issued • Number of vector instructions issued

22. Empirical Analysis of Algorithms PAPI • Hardware Performance Counter Basics • http://perfsuite.ncsa.uiuc.edu/publications/LJ135/x27.html • RabbitA Performance Counters Libraryfor Intel/AMD Processors and Linux • http://www.scl.ameslab.gov/Projects/Rabbit/