Analysis of Algorithms

1. Analysis of Algorithms

2. Analyzing Algorithms • We need methods and metrics to analyze algorithms for: • Correctness • Methods for proving correctness • Efficiency • Time complexity, Asymptotic analysis

3. Lecture Outline • Short Review on Asymptotic Analysis • Asymptotic notations • Running time estimation: summations, recurrences

4. Review - Asymptotic Analysis • Running time depends on the size of the input • T(n): the time taken on input with size n • it is the rate of growth, or order of growth, of the running time that really interests us • Look at growth of T(n) as n→∞. • Worst-case and average-case running times are difficult to compute precisely, so we calculate upper and lower bounds of the function.

5. Review - Asymptotic Notations • O: Big-Oh • Ω: Big-Omega • Θ: Theta [CLRS] – chap 3

6. O (g(n)) is the set of all functions with a smaller or same order of growth as g(n), within a constant multiple If we say f(n) is in O(g(n)), it means that g(n) is an asymptotic upper bound of f(n) Intuitively, it is like f(n) ≤ g(n) Big O [CLRS], Fig. 3.1

7. Ω (g(n)) is the set of all functions with a larger or same order of growth as g(n), within a constant multiple f(n)  Ω(g(n)) means g(n) is an asymptotic lower bound of f(n) Intuitively, it is like g(n) ≤ f(n) Big Ω [CLRS], Fig. 3.1

8. Informally, Θ (g(n)) is the set of all functions with the same order of growth as g(n), within a constant multiple f(n)  Θ(g(n)) means g(n) is an asymptotically tight bound of f(n) Intuitively, it is like f(n) = g(n) Theta (Θ) [CLRS], Fig. 3.1

9. Running Time Estimation • Find f(n), T(n) in O(f(n)) or T(n) in Θ(f(n)) • Simplifying assumption: • each statement takes the same unit amount of time. • usually we can assume that the statements within a loop will be executed as many times as the maximum permitted by the loop control. • More complex situations: • Summations (a loop is executed many times, each time with a different complexity) • Recurrences (recursive algorithms)

10. n O(n3) i<=n i<=n O(1) Summations - Example For i=1 to n do For j=1 to i do For k=1 to i do something_simple But what about Θ ? Function something_simple is executed exactly S(n) times: S(n)= Σi=1n i2 = n(n+1)(2n+1)/6 S(n) in Θ(n3) See also: [CLRS] – Appendix A - Summation formulas

11. Recurrences-Example1 Factorial (n) is if (n=1) return 1 else return n * Factorial (n-1) T(n) = c1, n=1 T(n-1) +c2, n>1 Substitution method: T(n) = T(n-1) + c2= T(n-2) + 2*c2 = T(n-3) + 3*c2 = … = T(1) + (n-1)*c2 Θ(n)

12. T(n) = Θ(1), n=1 2*T(n/2) + Θ(n), n>1 Recurrences-Example 2 MERGE-SORT(A[p..r]) if p < r q= (p+r)/2 MERGE-SORT(A[p..q]) MERGE-SORT(A[q+1..r]) MERGE(A[p..r],q) • Methods to solve: • Substitution method • Recursion tree method

13. Substitution meth: T(n)=2 *T(n/2)+n T(n) = 2*T(n/2)+n T(n/2)=2*T(n/22)+n/2 T(n) = 22*T(n/22)+2*n/2+n = 22*T(n/22)+2*n T(n/22) = 2*T(n/23)+n/22 T(n) = 23*T(n/23)+22*n/22+2*n = 23*T(n/23)+3*n .. T(n) = 2k*T(n/2k)+k*n T(n/2k) = 2*T(n/2k+1)+n/2k T(n)=2k+1*T(n/2k+1)+(k+1)*n … n/2x=1 x=log2 n T(n)=n * T(1) + n * log2 n Θ(n * log2 n)

14. Recursion tree: T(n)=2 *T(n/2)+n T(n) n n T(n/2) T(n/2) n/2 n/2 T(n/4) T(n/4) T(n/4) T(n/4)

15. n n log 2 n n n Recursion tree: T(n)=2 *T(n/2)+n n n/2 n/2 n/4 n/4 n/4 n/4 T(1) T(1) T(1) T(1) T(1) T(1) T(1) T(1) Θ(n * log2 n)

16. Solving Recurrences • General methods: • Substitution: guess the solution and prove it with math induction • Recursion-tree: use math techniques for bounding summations • The master theorem: provides bounds for recurrences of the general form T(n)=aT(n/b)+f(n), f(n)=c*nk • This form of recurrence appears frequently in divide-and-conquer algorithms • 3 cases, according to the values of a, b and k • Can be proved by substitution or recursion-tree • [CLRS] chap 4

17. T(n)=aT(n/b)+f(n) Recursion tree

18. T(n)=aT(n/b)+f(n) • Which is the height of the tree ? • How many nodes are there on each level ? • How many leaves are there ? • Which is the workload on each non-leaf level ? • Which is the workload on the leaves level ? Recursion tree

19. T(n)=aT(n/b)+f(n) [CLRS] Fig 4.4

20. T(n)=aT(n/b)+f(n) T(n) will result good if: b is big a is small f(n)=O(nk), k is small [CLRS] Fig 4.4

21. T(n)=aT(n/b)+f(n) From the recursion tree: Workload in leaves Workload per level i Sum for all levels

22. T(n)=aT(n/b)+f(n), f(n)=c*nk Geometric series of factor a/bk

23. Math review See also: [CLRS] – Appendix A - Summation formulas

24. T(n)=aT(n/b)+f(n), f(n)=c*nk Theorem: The solution of the recurrence relation is:

25. T(n) = Θ(1), n=1 2*T(n/2) + Θ(n), n>1 Merge-sort revisited • The recurrence relation of Merge-sort is a case of the master theorem, for a=2, b=2, k=1 • Case a=bk => Θ(nk * log n), k=1 =>Θ(n * log n) • Conclusion: In order to solve a recurrence, we can either: • Memorize the result of the Master Theorem and apply it directly • Do the reasoning (by substitution or by recursion-tree) on the particular case

26. Bibliography • Review Analysis of algorithms: • [CLRS] – chap 3 (Growth of functions), chap 4 (Recurrences, Master Theorem) or • [Manber] – chap 3