Order of growth

1. Order of growth Suppose you have analyzed two algorithms and expressed their run times in terms of the size of the input: • Algorithm A: takes 100 n + 1 steps to solve a problem with size n; • Algorithm B: takes n2 + n + 1 steps. The leading term is the term with the highest exponent.

2. The following table shows the run time of these algorithms for different problem sizes:

3. Notes • At n=10, Algorithm A looks bad • For Algorithm A, the leading term has a large coefficient, 100, which is why B does better than A for small n. • But for n=100 they are about the same, • for larger values of n, A is much better. • any function that contains an n2 term will grow faster than a function whose leading term is n. • Even if the run time of Algorithm A were n + 1000000, it would still be better than Algorithm B for sufficiently large n.

4. How to compare algorithms? • for large problems, we expect an algorithm with a smaller leading term to be a better algorithm • but for smaller problems, there may be a crossover point where another algorithm is better. • The location of the crossover point depends on the details of the algorithms, the inputs and the hardware,

5. How to compare algorithms? • If two algorithms have the same leading order term, it is hard to say which is better; the answer will depend on the details. • they are considered equivalent, even if they have different coefficients.

6. Order of growth An order of growth is a set of functions whose growth is considered equivalent. Examples: • 2n, 100n and n + 1 belong to the same order of growth, which is written O(n) in “Big-Oh notation” • All functions with the leading term n2 belong to O(n2); • What is the order of growth of n3 + n2? • What about 1000000 n3 + n2. What about n3 + 1000000 n2? • What is the order of growth of (n2 + n) * (n + 1)?

7. The following table shows some of the orders of growth that appear most commonly in algorithmic analysis, in increasing order of badness.

8. Asymptotic Analysis of Algorithms(Asymptotic  for large n) big oh expressions greatly simplify the analysis of the running time of algorithms • all that we get is an upper bound on the running time of the algorithm • the result does not depend upon the values of the constants • the result does not depend upon the characteristics of the computer and compiler actually used to execute the program! 8

9. Let f and g be nonnegative functions on the positive integers We write f(n)=O(g(n)) And say that f(n) is of order at most g(n) or, f(n) is big oh of g(n) or, g is an asymptotic upper bound for f if there exist constants C1>0 and N1 such that f(n)  C1g(n), for all n  N1 Big O notation 9

10. Is 2n+1=O(2n)? 2n+1<=c*2n? Yes, if c>=2 for all n Is 22n=O(2n)? 22n<=c*2n? 2n*2n<=c*2n? 2n<=c? No f(n)=5n3f(n)=O(n3) g(n)= 3n2g(n)=O(n3)  but f(n) not equal g(n) 10

11. Notation O() is set of functions. But common to abuse notation, writing T(n) = O(…) instead of T(n)  O(…) as well as T(n) = f(n) + O(…) 11

12. Conventions for Writing Big Oh Expressions • Ignore the multiplicative constants • Instead of writing O(3n2), we simply write O(n2) • If the function is constant (e.g. O(1024) we write O(1)) • Ignore the lower order terms • Instead of writing O(nlogn+n+n2), we simply write O(n2) 12

13. Examples T(n) = 32n2 + 17n + 32 T(n)=O(n2), O(n3), not O(n) NOTE: O(n2) tight bound, O(n3) not tight bound, • n, n+1, n+80, 40n, n+log n is O(n) • n2 + 10000000000n is O(n2) • 3n2 + 6n + log n + 24.5 is O(n2) 13

14. Properties of Big Oh If f1(n)=O(g1(n)) and f2(n)=O(g2(n)) , then f1(n)+f2(n)=O(max(g1(n),g2(n))) If f1(n)=O(g1(n)) and f2(n)=O(g2(n)) , then f1(n)X f2(n)=O(g1(n)X g2(n)) f(n) = O(g(n)) & g(n) = O(h(n))  f(n) = O(h(n)) 14

15. e.g.6 (revisited): c1 O(1) c2(n+1)O(n) c3n O(n) c4 O(1) T(n)=c1+c2(n+1)+c3n+c4 T(n)=a+bnfor some constants a, b T(n)=O(n) Horner(int a[], n, x){ result=a[n] for(i=n-1;i>=0,--i) result=result*x+a[i] return result } 15

16. e.g.8 (revisited): asymptotic analysis O(1) O(n) O(n) O(n2) O(n2) O(n) T(n) = O(n2) fun (int x, int n){ Sum=0; For(i=0n){ P=1 For(j=0; j < n ;++j) P*=x Sum+=p } } 16

17. e.g.10 (revisited): Sequential search The worst case time T(n) = an+b The best case time T(n) = constant Obtain asymptotic O bound on the solution? The worst case is O(n) The best case is O(1) 17

18. e.g.11: find a O-notation in terms of n for the number of times the statement x=x+1 is executed in the segment: for (i=1 to n) for (j=1 to i) x=x+1 i j 1 11 2 12 3 13 : n 1n cn=1+2+3+…+n =n(n+1) / 2 O(n2) 18

19. e.g.12: find a O-notation in terms of n for the number of times the statement x=x+1 is executed in the segment: j=n while ( j >= 1){ for (i=1 to j) x=x+1 j=j/2 } j i n 1n n/2 1n/2 n/4 1n/4 : n/2k 1n/2k cn=n+n/2+n/4+…+ n/2k =n(1+1/2+1/22+…+ 1/2k) =n(1/(1-0.5)) O(n) 19

20. e.g.7 (revisited):Obtain asymptotic O bound for recursive functions??? Solving Recurrence Relations-Repeated Substitution T(n)=a if n=0 T(n)=T(n-1)+b if n>0 for some constants a, b 20

21. An Asymptotic Lower Bound-Omega • Let f and g be nonnegative functions on the positive integers • We write • f(n)= (g(n)) • And say that • f(n) is of order at least g(n) or, • f(n) is omega of g(n) or, • g is an asymptotic lower bound for f • if there exist constants C2>0 and N2 such that • f(n)  C2g(n), • for all n  N2 21

22. 2n+13  O( ? ) Also, O(n2), … Can always weaken the bound. O(n) 2n+13  ( ? ) (n), also (log n), (1), … 2n O(n) ? (n) ? (n), not O(n). nlog n O(n5) ? No. Thus, (n5).

23. Let f and g be nonnegative functions on the positive integers We write f(n)= (g(n)) And say that f(n) is of order g(n) or, f(n) is theta of g(n) or, g is an asymptotic tight bound for f if f(n)=O(g(n)) and f(n)= (g(n)) 23

24. e.g. T(n) = 32n2 + 17n + 32 we can ignore the multiplicative constants and the lower order terms T(n)=O(n2), O(n3), not O(n) NOTE: O(n2) tight bound, O(n3) not tight bound, Since 32n2 + 17n + 32 T(n)= (n2), (n), not (n3)  T(n)= (n2), not (n), not (n3)

25. Properties Transitivity f(n) = (g(n)) & g(n) = (h(n))  f(n) = (h(n)) f(n) = O(g(n)) & g(n) = O(h(n))  f(n) = O(h(n)) f(n) = (g(n)) & g(n) = (h(n))  f(n) = (h(n)) Symmetry f(n) = (g(n)) if and only if g(n) = (f(n)) Transpose Symmetry f(n) = O(g(n)) if and only if g(n) = (f(n)) 25