Analysis of Algorithm

1. Analysis of Algorithm

2. Why Analysis? • We need to know the “behavior” of algorithms • How much resource (time/space) does it use • So that we know when to use which algorithm • So that two algorithm can be compared whether which one is better, with respect to the situtation

3. Can we? • Sure! • If we have the algorithm • We can implement it • We can test it with input • But… • Is that what we really want? • If you wish to know whether falling from floor 20 of Eng. 4 building would kill you • Will you try?

4. Prediction • We wish to know the “Behavior” of the algorithm • Without actually trying it • Back to the suicidal example • Can you guess whether you survive jumping from 20th floor of Eng. 4 building? • What about 15th floor? • What about 10th floor? • What about 5th floor? • What about 2nd floor? • Why?

5. Modeling • If floor_number > 3 then • Die • Else • Survive (maybe?) Describe behavior using some kind of model, rule, etc.

6. Generalization • What about jumping from Central World’s 20th floor? • What about jumping from Empire State’s 20th floor? • What about jumping from BaiYok’s 20th floor? • Can our knowledge (our analysis of the situation) be applicable on the above questions?

7. Generalization • Knowledge from some particular instances might be applicable to another instance

8. Analysis • We need something that can tell us the behavior of the algorithm that is… • Useful (give us knowledge without actually doing it) • Applicable (give us knowledge for similar kind of situation) Modeling Generalization

9. Analysis (Measurement) • What we really care? RESOURCE Space (amount of RAM) Time (CPU power)

10. Model • How to describe performance of an algorithm? Usage of Resource how well does an algo use resource?

11. Model • Resource Function Time Function of algorithm A Time used Input ? Size of input Space Function of algorithm A Space used Input ?

12. Example • Inserting a value into a sorted array • Input: • a sorted array A[1..N] • A number X • Output • A sorted array A[1..N+1] which includes X

13. Algorithm • Element Insertion • Assume that X = 20 • What if A = [1,2,3]? How much time? • What if A = [101,102,103]? idx = N; while (idx >= 1 && A[idx] > X) { A[idx + 1] = A[idx]; idx--; } A[idx] = X; Usually, resource varies according to size of input

14. Using the Model Time average best worst Size of Input

15. Resource Function • Give us resource by “size of input” • Why? • Easy to compute • Applicable (usually give meaningful, fairly accurate result without much requirement)

16. Conclusion • Measurement for algorithm • By modeling and generalization • For prediction of behavior • Measurement is functions on the size of input • With some simplification • Best, avg, worst case

17. Asymptotic Notation

18. Comparing two algorithms • We have established that a “resource function” is a good choice of measurement • The next step, answering which function is “better”

19. What is “better” in our sense? • Takes less resource • Consider this which one is better? f(x) g(x)

20. Slice f(x) g(x)

21. What is “better” in our sense? • which one is better? • Performance is now a function, not a single value • Which slice to use? • Can we say “better” based on only one slice? • Use the slice where it’s really matter • i.e., when N is large • What is large N? • Infinity? • Implication?

22. Comparison by infinite N • There is some problem • Usually, • The larger the problem, the more resource used

23. Separation between Abstraction and Implementation • Rate of Growth • by changing the size of input, how does the TIME and SPACE requirement change • Compare by how f(x) grows when x increase, w.r.t. g(x)

24. Compare by RoG 0 : f(x) grows “slowzer” than g(x) ∞ : f(x) grows “faster” than g(x) else : f(x) grows “similar” to g(x)

25. 0.5n 1 log n log6 n n0.5 n3 2n n! Growth Rate Comparison Sometime it is simple Some time it is not

26. l’Hôpital’s Rule • Limit of ratio of two functions equal to limit of ratio of their derivative. • Under specific condition

27. l’Hôpital’s Rule • If • then

28. The problem of this approach • What if f(x) cannot be differentiated? • Too complex to find derivative

29. Compare by Classing • Coarse grain comparison • Another simplification • Work (mostly) well in practice • Classing

30. Classing • Simplification by classification • Grading Analogy

31. Compare by Classification algo

32. Compare by Classification algo Group B Group A Group F Group C Group D grouping

33. Compare by Classification algo Group B Group A Group F 70 <= x < 80 >= 80 Group C 60 <= x < 70 x < 50 Group D Describe “simplified” property 50 <= x < 60

34. Compare by Classification • Group by some similar property • Select a representative of the group • Use the representative for comparison • If we have the comparison of the representative • The rest is to do the classification

35. Complexity Class • We define a set of complexity class • using rate of growth • Here comes the so-called Asymptotic Notation • Q, O, W, o, w • Classify by asymptotic bound

36. Asymptote • Something that bounds curves Curve Asymptote

37. Remember hyperbola? Asymptote

38. O-notation cg(x) For function g(n), we define O(g(n)), big-O of n, as the set: f(x) O(g(n)) ={f(n) :  positive constants c and n0,such that n  n0, we have 0 f(n) cg(n) } Intuitively: Set of all functions whose rate of growthis the same as or lower than that of g(n). n0 f(x) O(g(x)) g(n) is an asymptotic upper boundfor f(n).

39.  -notation For function g(n), we define (g(n)), big-Omega of n, as the set: f(x) (g(n)) ={f(n) :  positive constants c and n0,such that n  n0, we have 0 cg(n) f(n)} cg(x) n0 Intuitively: Set of all functions whose rate of growthis the same as or higher than that of g(n). f(x) (g(x)) g(n) is an asymptotic lower boundfor f(n).

40. -notation For function g(n), we define (g(n)), big-Theta of n, as the set: c2g(x) (g(n)) ={f(n) :  positive constants c1, c2, and n0,such that n  n0, we have 0 c1g(n)  f(n) c2g(n) } f(x) c1g(x) n0 f(x) (g(x)) Intuitively: Set of all functions that have the same rate of growthas g(n). g(n) is an asymptotically tight boundfor f(n).

41. Example F(n) = 300n + 10 is a member of (30n) why? let c1 = 9 let c2 = 11 let n = 1

42. Another Example F(n) = 300n2 + 10n is a member of (10n2) why? let c1 = 29 let c2 = 31 let n = 11

43. How to Compute? Remove any constant F(n) = n3+2n2 + 4n + 10 is a member of (n3+n2 + n) Remove any lower degrees F(n) = n3+2n2 + 4n + 10 is a member of (n3)

44. Relations Between Q, W, O • I.e., (g(n)) = O(g(n)) ÇW(g(n)) • In practice, asymptotically tight bounds are obtained from asymptotic upper and lower bounds. For any two functions g(n) and f(n), f(n) = (g(n))if and only if f(n) =O(g(n)) and f(n) = (g(n)).

45. Practical Usage • We say that the program has a worst case running time of O(g(n)) • We say that the program has a best case running time of W(g(n)) • We say that the program has a tight-bound running time of Q(g(n))

46. Example • Insertion sort takes O(n2) in the worst case • Meaning: at worst, insertion sort, takes time that grows not more than quadratic of the size of the input • Insertion sort takes W(n) in the best case • Meaning: at best, insertion sort, takes time that grows not less than linear to the size of the input

47. o-notation For a given function g(n), the set little-oh: o(g(n)) = {f(n): c > 0, n0 > 0 such that n n0, we have0 f(n)<cg(n)}.

48. w-notation For a given function g(n), the set little-omega: ω(g(n)) = {f(n): c > 0, n0 > 0 such that n n0, we have0 cg(n) < f(n)}.

49. Remark on Notation • An asymptotic group is a set • Hence f(n) is a member of an asymptotic group • E.g., f(n)  O( n ) • Strictly speaking, f(n) = O( n ) is syntactically wrong • But we will see this a lot • It’s traditions

50. Comparison of Functions f (n) O(g(n))f (n)g(n) f (n)(g(n))f (n)g(n) f (n)(g(n))f (n) = g(n) f (n) o(g(n))f (n)<g(n) f (n)w (g(n))f (n)>g(n) Where < , > , = means grows slower, faster, equally