Introduction to Algorithms (2 nd edition)

1. Introduction to Algorithms(2nd edition) by Cormen, Leiserson, Rivest & Stein Chapter 3: Growth of Functions (slides enhanced by N. Adlai A. DePano)

2. Overview • Order of growth of functions provides a simple characterization of efficiency • Allows for comparison of relative performance between alternative algorithms • Concerned with asymptotic efficiency of algorithms • Best asymptotic efficiency usually is best choice except for smaller inputs • Several standard methods to simplify asymptotic analysis of algorithms

3. Asymptotic Notation • Applies to functions whose domains are the set of natural numbers:N = {0,1,2,…} • If time resource T(n) is being analyzed, the function’s range is usually the set of non-negative real numbers:T(n)R+ • If space resource S(n) is being analyzed, the function’s range is usually also the set of natural numbers:S(n)N

4. Asymptotic Notation • Depending on the textbook, asymptotic categories may be expressed in terms of -- • set membership (our textbook): functions belong to a family of functions that exhibit some property; or • function property (other textbooks): functions exhibit the property • Caveat: we will formally use (a) and informally use (b)

5. c2⋅g f c1⋅g n0 The Θ-Notation Θ(g(n))= { f(n) :∃c1, c2 > 0, n0 > 0 s.t. ∀n ≥ n0: c1 · g(n) ≤ f(n) ≤ c2⋅g(n) }

6. c⋅g f n0 The O-Notation O(g(n))= {f(n): ∃c > 0, n0 > 0 s.t. ∀n ≥ n0: f(n) ≤ c⋅g(n) }

7. f c⋅g n0 TheΩ-Notation Ω(g(n))= { f(n): ∃c > 0, n0 > 0 s.t. ∀n ≥ n0: f(n) ≥ c⋅g(n) }

8. c3⋅g c2⋅g c1⋅g f n1 n3 n2 The o-Notation o(g(n))= {f(n): ∀c > 0 ∃n0 > 0 s.t. ∀n ≥ n0: f(n) ≤ c⋅g(n) }

9. f c3⋅g c2⋅g c1⋅g n3 n2 n1 Theω-Notation ω(g(n))= { f(n): ∀c > 0 ∃n0 > 0 s.t. ∀n ≥ n0: f(n) ≥ c⋅g(n) }

10. Comparison of Functions • f(n) = O(g(n)) and g(n) = O(h(n)) ⇒ f(n) = O(h(n)) • f(n) = Ω(g(n)) and g(n) = Ω(h(n)) ⇒ f(n) = Ω(h(n)) • f(n) = Θ(g(n)) and g(n) = Θ(h(n)) ⇒ f(n) = Θ(h(n)) • f(n) = O(f(n)) f(n) = Ω(f(n)) f(n) = Θ(f(n)) Transitivity Reflexivity

11. Comparison of Functions • f(n) = Θ(g(n)) ⇐⇒ g(n) = Θ(f(n)) • f(n) = O(g(n)) ⇐⇒ g(n) = Ω(f(n)) • f(n) = o(g(n)) ⇐⇒g(n) = ω(f(n)) • f(n) = O(g(n)) and f(n) = Ω(g(n)) ⇒ f(n) = Θ(g(n)) Symmetry Transpose Symmetry Theorem 3.1

12. Asymptotic Analysis and Limits

13. Comparison of Functions • f1(n) = O(g1(n)) and f2(n) = O(g2(n)) ⇒ f1(n) + f2(n) = O(g1(n) + g2(n)) • f(n) = O(g(n)) ⇒ f(n) + g(n) = O(g(n))

14. Standard Notation and Common Functions • Monotonicity A function f(n) is monotonically increasing if m n implies f(m)f(n) . A function f(n) is monotonically decreasing if m n implies f(m)f(n) . A function f(n) is strictly increasingif m< n implies f(m)<f(n) . A function f(n) is strictly decreasingif m <n implies f(m)>f(n) .

15. Standard Notation and Common Functions • Floors and ceilings For any real number x, the greatest integer less than or equal to x is denoted by x. For any real number x, the least integer greater than or equal to x is denoted by x. For all real numbers x, x1 < x x x< x+1. Both functions are monotonically increasing.

16. Standard Notation and Common Functions • Exponentials For all n and a1, the function an is the exponential function with base a and is monotonically increasing. • Logarithms Textbook adopts the following conventionlg n = log2n (binary logarithm),ln n = logen (natural logarithm), lgkn = (lg n)k(exponentiation),lg lg n = lg(lg n)(composition),lg n + k = (lg n)+k (precedence of lg). ai

17. Standard Notation and Common Functions • Important relationships For all real constants a and b such that a>1,nb = o(an)that is, any exponential function with a base strictly greater than unity grows faster than any polynomial function. For all real constants a and b such that a>0,lgbn = o(na)that is, any positive polynomial function grows faster than any polylogarithmic function.

18. Standard Notation and Common Functions • Factorials For all n the function n! or “n factorial” is given byn! = n (n1) (n  2) (n 3) …  2  1 It can be established thatn! = o(nn) n! = (2n) lg(n!) = (nlgn)

19. Standard Notation and Common Functions • Functional iteration The notation f(i)(n) represents the function f(n) iteratively applied i times to an initial value of n, or, recursivelyf(i)(n) = nif n=0f(i)(n) = f(f(i1)(n)) if n>0 Example: If f(n) = 2n then f(2)(n) = f(2n) = 2(2n) = 22n then f(3)(n) = f(f(2)(n)) = 2(22n) = 23n then f(i)(n) = 2in

20. Standard Notation and Common Functions • Iterated logarithmic function The notation lg* n which reads “log star of n” is defined aslg* n = min {i0 : lg(i) n 1 Example:lg* 2 = 1 lg* 4 = 2 lg* 16 = 3 lg* 65536 = 4 lg* 265536 = 5

21. Asymptotic Running Time of Algorithms • We consider algorithm A better than algorithm B ifTA(n) = o(TB(n)) • Why is it acceptable to ignore the behavior of algorithms for small inputs? • Why is it acceptable to ignore the constants? • What do we gain by using asymptotic notation?

22. Things to Remember • Asymptotic analysis studies how the values of functions compare as their arguments grow without bounds. • Ignores constants and the behavior of the function for small arguments. • Acceptable because all algorithms are fast for small inputs and growth of running time is more important than constant factors.

23. Things to Remember • Ignoring the usually unimportant details, we obtain a representation that succinctly describes the growth of a function as its argument grows and thus allows us to make comparisons between algorithms in terms of their efficiency.