Algorithm Analysis

1. Algorithm Analysis

2. Why do we care? Every 18-24 months, manufacturers are introducing faster machines with larger memories. Why do we need to write efficient code?

3. Well, just suppose… • Imagine we are defining a Java class “Huge” to represent long integers. • We want a method for our class that • adds two large integers – two instances of our class • multiplies two large integers. Suppose we have successfully implemented the add() method and are moving on to multiply().

4. Well, just suppose… Suppose we have successfully implemented the add() method and are moving on to multiply(). Object oriented coding is all about using what you already have, right? So, since multiplication is equivalent to repeated addition, to compute the product 7562 * 423 we could initialize a variable to 0 and then add 7562 to it 463 times – why not, add() works, right?

5. Efficiency Example public class BigOEx { public static void main (String [] args) { long firstOp = 7562; long secondOp = 463 ; long product = 0; for (long i=secondOp; i>0; i--) product += firstOp; System.out.println("Product of "+firstOp+" and "+secondOp+" = "+product); } }

6. Efficiency Example • If we run the previous code, we should get a result in a reasonable amount of time. So, let’s run it but replace the 463 with 100000000 (8 0’s). Will we still get a result in a reasonable time? How about with 1000000000. (9 0’s) Is something wrong?

7. Efficiency Example Why a code delay? Can we do better?

8. Efficiency Example • How might we rethink our code to produce a more efficient result?

9. Efficiency Example Consider the product of the original numbers (463 * 7562). The secondOp has 3 digits – a 100’s digit, a 10’s digit, and a 1’s digit. (463 = 400 + 60 + 3) So: 7562 * 463 = 7562 * (400 + 60 + 3) = 7562 * 400 + 7562 * 60 + 7562 * 3 = 756200 * 4 + 75620 * 6 + 7562*3 = 756200+756200+756200+756200+75620= 75620+75620+75620+75620+75620+7562+7562+7562

10. Efficiency Example public class BetterBigOEx { public static void main (String [] args) { long firstOrig,secondOrig; long firstOp = firstOrig = 7562; long secondOp = secondOrig = 1000000000 ; int secOpLength=10; long product = 0; for (int digitPosition=0; digitPosition<secOpLength;digitPosition++) { int digit = (int)(secondOp - (secondOp/10)*10); for (int counter = digit; counter > 0; counter--) product = product+firstOp; secondOp = secondOp/10; // discard last digit firstOp = 10*firstOp; //tack a 0 to the right } System.out.println("Product of "+firstOrig+" and "+secondOrig+" = "+product); } }

11. Efficiency Example • Does efficiency matter? • How do we measure it? • Create 2 programs and measure the difference (not always possible) • Measure algorithm before implementation

12. Efficiency Calculation • Let’s say you need to get downtown. You can walk, drive, ask a friend to take you, or take a bus. What’s the best way?

13. Efficiency Measurement • Algorithms have measurable time and space requirements called complexity. • We are not considering how difficult it is to code, but rather the time it takes to execute and the memory it will need.

14. Analysis of Algorithms • Usually measure time complexity • Compute approximation, not actual time • Typically estimate the WORST (maximum) time the algorithm could take. Why? • Measurements also exist for best and average cases, but generally you look for the worse case analysis.

15. Best Case W To improve the best case, all we have to do it to be able to solve one instance of each size efficiently. We could modify our algorithm to first test whether the input is the special instance we know how to solve, and then output the canned answer. For sorting, we can check if the values are already ordered, and if so output them. For the traveling salesman, we can check if the points lie on a line, and if so output the points in that order. The supercomputer people pull this trick on the linpack benchmarks!

16. Because it is so easy to cheat with the best case running time, we usually don’t rely too much about it. Because it is usually very hard to compute the average running time, since we must somehow average over all the instances, we usually strive to analyze the worst case running time. The worst case is usually fairly easy to analyze and often close to the average or real running time.

17. Exact Analysis is Hard! • We have agreed that the best, worst, and average case complexity of an algorithm is a numerical function of the size of the instances.

18. Names of Bounding Functions Now that we have clearly defined the complexity functions we are talking about, we can talk about upper and lower bounds on it: • g(n) = O(f(n)) means C × f(n) is an upper bound on g(n). • g(n) = W(f(n)) means C×f(n) is a lower bound on g(n). • g(n) = Q (f(n)) means C1 × f(n) is an upper bound on g(n) and C2 × f(n) is a lower bound on g(n). Got it? C, C1, and C2 are all constants independent of n. All of these definitions imply a constant n0beyond which they are satisfied. We do not care about small values of n.

19. O, W, and Q • The value of n0 shown is the minimum possible value; any greater value would also work. • f(n) = Q(g(n)) if there exist positive constants n0, c1, and c2 such that to the right of n0, the value of f(n) always lies between c1 · g(n) and c2 · g(n) inclusive.

20. O, W, and Q • The value of n0 shown is the minimum possible value; any greater value would also work. • f(n) = Q(g(n)) … • f(n) = O(g(n)) if there are positive constants n0 and c such that to the right of n0, the value of f(n) always lies on or below c · g(n).

21. O, W, and Q • The value of n0 shown is the minimum possible value; any greater value would also work. • f(n) = Q(g(n)) … • f(n) = O(g(n)) … • f(n) = W(g(n)) if there are positive constants n0 and c such that to the right of n0, the value of f(n) always lies on or above c · g(n).

22. What does this mean?

23. Testing Dominance

25. Functions • While there are “tons” of different functions, only a few function classes tend to show up in when analyzing algorithms: • Constant functions f(n) = 1 • Logarithmic functions f(n) = log n • Linear functions f(n) = n • Superlinear functions f(n) = n lg n • Quadratic functions f(n) = n2 • Cubic functions f(n) = n3 • Exponential functions f(n) = cn for c>1 • Factorial functions f(n) = n!

26. Logarithms • An important concept in algorithm analysis • Simply an inverse exponential function • bx = y is equivalent to saying that x = logby • Exponential functions grow alarmingly quickly (when you borrow money from a credit card or mortgage, you are dealing with a function of interest that grows exponentially) • Logarithms grow refreshing slowly • Binary Search is an algorithm that grows logarithmically because we cut by half each time, the amount of data we have to deal with. (Phone Book)

27. Properties of Logarithms • Definition: • Asymptotically, the base of the log does not matter.

28. Properties of Logarithms • Asymptotically, any polynomial function of n does not matter.

29. Federal Sentencing Guidelines 2F1.1. Fraud and Deceit; Forgery; Offenses Involving Altered or Counterfeit Instruments other than Counterfeit Bearer Obligations of the United States. (a) Base offense Level: 6 (b) Specific offense Characteristics (1) If the loss exceeded $2,000, increase the offense level as follows:

31. Working with the Asymptotic Notation Suppose f(n) = O(n2) and g(n) = O(n2). • What do we know about g′(n) = f(n) + g(n)? • What do we know about g′′(n) = f(n) − g(n)? • What do we know about the lower bounds (W) on g′ + g′′ ?

32. Working with the Asymptotic Notation • Suppose f(n) = W(n2) and g(n) = W(n2). • What do we know about g′(n) = f(n) + g(n)?

33. The Complexity of Songs Suppose we want to sing a song which lasts for n units of time. Since n can be large, we want to memorize songs which require only a small amount of brain space, i.e. memory. Let S(n) be the space complexity of a song which lasts for n units of time. The amount of space we need to store a song can be measured in either the words or characters needed to memorize it. Note that the number of characters is (words) since every word in a song is at most 34 letters long – Supercalifragilisticexpialidocious! What bounds can we establish on S(n)? • S(n) = O(n), since in the worst case we must explicitly memorize every word we sing – “The Star-Spangled Banner” • S(n) = W(1), since we must know something about our song to sing it.

34. The Refrain Most popular songs have a refrain, which is a block of text which gets repeated after each stanza in the song: Bye, bye Miss American pie Drove my chevy to the levy but the levy was dry Them good old boys were drinking whiskey and rye Singing this will be the day that I die. Refrains made a song easier to remember, since you memorize it once yet sing it O(n) times. But do they reduce the space complexity? Not according to the big oh. If n = repetitions × (verse-size + refrain-size) Then the space complexity is still O(n) since it is only halved (if the verse-size = refrain-size): S(n) = repetitions × verse-size + refrain-size

35. The k Days of Christmas To reduce S(n), we must structure the song differently. Consider “The k Days of Christmas”. All one must memorize is: On the kth Day of Christmas, my true love gave to me, giftk ... On the First Day of Christmas, my true love gave to me, a partridge in a pear tree

36. 100 Bottles of Beer What do kids sing on really long car trips? n bottles of beer on the wall, n bottles of beer. You take one down and pass it around n − 1 bottles of beer on the ball. All you must remember in this song is this template of size (1), and the current value of n. The storage size for n depends on its value, but log2 n bits suffice this for this song, S(n) = O(lg n). Is there a song which eliminates even the need to count?

37. That’s the way, uh-huh, uh-huh I like it, uh-huh, huh Notes from the author’s website, with references to: Reference: D. Knuth, ‘The Complexity of Songs’, Comm. ACM, April 1984, pp.18-24