1 / 42

CSE 3101

CSE 3101. Prof. Andy Mirzaian. Machine Model & Time Complexity. STUDY MATERIAL:. [CLRS] chapters 1, 2, 3 Lecture Note 2. Example. Time complexity shows dependence of algorithm’s running time on input size.

alyn
Download Presentation

CSE 3101

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CSE 3101 Prof. Andy Mirzaian Machine Model & Time Complexity

  2. STUDY MATERIAL: • [CLRS] chapters 1, 2, 3 • Lecture Note 2

  3. Example Time complexityshows dependence of algorithm’s running time on input size. Let’s assume: Computer speed = 106 IPS, Input: a data base of size n = 106

  4. Machine Model • Algorithm Analysis: • should reveal intrinsic properties of the algorithm itself. • should not depend on any computing platform, programminglanguage, compiler, computer speed, etc. • Random Access machine (RAM): an idealized computer model • Elementary steps: • arithmetic: + –   • logic: and or not • comparison:       • assigning a value to a scalar variable:  • input/output a scalar variable • following a pointer or array indexing • on rare occasions: sin, cos, …

  5. Time Complexity • Time complexityshows dependence of algorithm’s running time on input size. • Worst-case • Average or expected-case • Amortized (studied in CSE 4101) • What is it good for? • Tells us how efficient our design is before its costly implementation. • Reveals inefficiency bottlenecks in the algorithm. • Can use it to compare efficiency of different algorithms that solve the same problem. • Is a tool to figure out the true complexity of the problem itself! How fast is the “fastest” algorithm for the problem? • Helps us classify problems by their time complexity.

  6. Input size & Cost measure • Input size: • Bit size: # bits in the input. This is often cumbersome but sometimes necessary when running time depends on numeric data values, e.g., factoring an n-bit integer. • Combinatorial size: e.g., # items in an array, a list, a tree, # vertices and # edges in a graph, … • Cost measure: • Bit cost:charge one unit of time to each bit operation • Arithmetic cost:charge one unit of time to each elementary RAM step • Complexity: • Bit complexity • Arithmetic complexity

  7. We choose these unless stated otherwise Input size & Cost measure • Input size: • Bit size: # bits in the input. This is often cumbersome but sometimes necessary when running time depends on numeric data values, e.g., factoring an n-bit integer. • Combinatorial size: e.g., # items in an array, a list, a tree, # vertices and # edges in a graph, … • Cost measure: • Bit cost:charge one unit of time to each bit operation • Arithmetic cost:charge one unit of time to each elementary RAM step • Complexity: • Bit complexity • Arithmetic complexity

  8. Time Complexity Analysis Worst-case time complexity is a function T: N + T(n) = maximum # steps by the algorithm on any input of size n. AlgorithmInversionCount(A[1..n]) count  0 for i  1 .. n do for j  i+1 .. n doif A[i] > A[j] then count++ return count end T(n) = 3n2 + 6n + 4 = Q(n2). Which line do you prefer? The 2nd is simpler and platform independent. Incidentally, this can be improved to Q(n log n) time by divide-&-conquer

  9. Caution! AlgorithmSUM(A[1..n]) S  0for i  1 .. n do S  S + A[i]return S end AlgorithmPRIME(P) § integer P > 1 for i  2 .. P – 1 do ifP mod i = 0 then return NO return YES end Worst Case: Q(P) time. Linear? # input bits n  log P  P  2n  T(n) = Q(P) = Q(2n). EXPONENTIAL! Worst Case: Q(n) time LINEAR PRIMALITY TESTING:used in cryptography … It was a long standing open problem whether there is any deterministic algorithm that solves this problem in time polynomial in the input bit size. This was eventually answered affirmatively by: M. Agrawal, N. Kayal, N. Saxena, “PRIMES is in P,” Annals of Mathematics 160, pp: 781-793, 2004.

  10. Asymptotic Notations

  11. drop lower order terms drop constant multiplicative factor T(n) = Q( f(n) ) T(n) = 23 n3 + 5 n2 log n + 7 n log2 n + 4 log n + 6. T(n) = Q(n3) • Why do we want to do this? • Asymptotically (at very large values of n) the leading term largely determines function behaviour. • With a new computer technology (say, 10 times faster) the leading coefficient will change (be divided by 10). So, that coefficient is technology dependent any way! • This simplification is still capable of distinguishing between important but distinct complexity classes, e.g., linear vs. quadratic, or polynomial vs exponential.

  12. Asymptotic Notations: Q O W o w Rough, intuitive meaning worth remembering:

  13. Asymptotics by ratio limit L = lim n f(n)/g(n). If L exists, then:

  14.  &  quantifiers • Logic: “and” commutes with “and” , “or” commutes with “or” , but “and” & “or” do not commute. • Similarly, same type quantifiers commute, but different types do not:x y: P(x,y) = y x: P(x,y) = x,y: P(x,y) x y: P(x,y) = y x: P(x,y) = x,y: P(x,y) x y: P(x,y)  y x: P(x,y) • Counter-example for the 3rd:LHS: for every person x, there is a date y, such that x is born on date y. RHS: there is a date y, such that for every person x, x is born on date y.Each person has a birth date, but not every person is born on the same date! • Give natural examples for the following to show their differences: • x y z: P(x,y,z) • x z y: P(x,y,z) • y x z: P(x,y,z)

  15. f(n) c2g(n) f(n) = Q(g(n)) g(n) c1g(n) n0 n c1, c2,n0 >0 : n  n0 ,c1g(n) f(n)  c2g(n). + Theta : Asymptotic Tight Bound

  16. f(n) = Q(g(n)) c1, c2 , n0 >0 : n  n0 ,c1g(n) f(n)  c2g(n) Theta : Example f(n) = 23n3 – 10 n2 log n+ 7n + 6 f(n) = [23 – (10 log n)/n + 7/n2 + 6/n3]n3factor out leading term n  10 : f(n)  (23 + 0 + 7/100 + 6/1000)n3 = (23 + 0 + 0.07 + 0.006)n3 = 23.076 n3 < 24 n3 n  10 :f(n)  (23 – log10 +0 +0)n3 > (23 – log16)n3=19n3 n  10 : 19 n3 f(n)  24 n3 (Take n0= 10, c1= 19, c2 = 24, g(n) = n3) Now you see why we can drop lower order terms & the constant multiplicative factor. f(n) = Q(n3)

  17. f(n) c g(n) f(n) = O(g(n)) g(n) n0 n c, n0 >0 : n  n0 ,f(n)  c g(n). + Big Oh: Asymptotic Upper Bound

  18. f(n) f(n) = W(g(n)) g(n) cg(n) n0 n c, n0 >0 : n  n0 ,cg(n) f(n). + Big Omega : Asymptotic Lower Bound

  19. cg(n) f(n) = o(g(n)) f(n) n0 n c>0, n0 >0 : n  n0 ,f(n) < c g(n). No matter how small + Little oh : Non-tight Asymptotic Upper Bound

  20. f(n) f(n) = w(g(n)) cg(n) n0 n c>0, n0 >0 : n  n0 ,f(n) > c g(n). No matter how large + Little omega : Non-tight Asymptotic Lower Bound

  21. f(n) = W(g(n)) f(n) = Q(g(n)) f(n) = O(g(n)) f(n) = w(g(n)) f(n) = o(g(n)) c1,c2>0, n0>0: n  n0 ,c1g(n) f(n)  c2g(n) c>0, n0>0: n  n0 ,f(n)  cg(n) c>0, n0 >0: n  n0 ,cg(n) f(n) c>0, n0>0: n  n0 ,f(n)< cg(n) c>0, n0>0: n  n0 ,cg(n) < f(n) Definitions of Asymptotic Notations

  22. f(n) = o(g(n))  c>0, n0>0: n  n0 ,f(n)< cg(n) n  c n0 >0, n  max{n0 , c} n (n  n0 and n  c) choose c=1, then  n0 >0, choose n= max{n0 , c} Example Proof CLAIM: n2 o(n). Proof: Need to show:  ( c>0 , n0>0: n  n0 (n2 < c n) ). Move  inside:  (c> 0 , n0>0, n  n0 (n2  c n ) ). Work inside-out: Now browse outside-in.Everything OK!

  23. f(n) = o(g(n))  c>0, n0>0: n  n0 ,f(n)< cg(n) Any short cuts? If you are a starter on these notations, you are advised to exercise your predicate logic muscles for a while to get the hang of it! However, in the next few slides, we will study some FACTS & RULES that help us manipulate and reason about asymptotic notations in most common situationswith much ease.

  24. A Classification of Functions • o(1)e.g., ()– , – , –, • O(1)asymptotically upper-bounded by a constant • Q(1)e.g., • Poly-Logarithmic:e.g., • Polynomial: e.g., • Super-Polynomialbut Sub-Exponential:e.g., ,, • Exponential or more:e.g., .

  25. Order of Growth Rules Below X Y means X = o(Y) : o(1) Q(1) Poly-Log Polynomial Exponential or more o(1) Q(1) • Examples: • = o(

  26. Asymptotic Relation Rules • Skew Symmetric:f(n) = O(g(n))  g(n) = W(f(n))f(n) = o(g(n))  g(n) = w(f(n)) • f(n) = Q(g(n))  f(n) = O(g(n)) & f(n) = W(g(n))  f(n) = O(g(n)) & g(n) = O(f(n)) • Symmetric [only Q]:f(n) = Q(g(n))  g(n) = Q(f(n)) • Reflexive:f(n) = Q(f(n)), f(n) = O(f(n)), f(n) = W(f(n)), f(n)  o(f(n)), f(n)  w(f(n)). • Transitive:f(n) = O(g(n)) & g(n) = O(h(n))  f(n) = O(h(n))works for Q, W, o, w too. • f(n) = o(g(n))  f(n) = O(g(n)) & f(n)  W(g(n))  f(n)  Q(g(n)) • f(n) = w(g(n))  f(n) = W(g(n)) & f(n)  O(g(n))  f(n)  Q(g(n)) • g(n) + o(g(n)) = Q(g(n)) [can hide lower order terms]

  27. Arithmetic Rules Given: f(n) = O(F(n)), g(n) = O(G(n)), h(n) = Q(H(n)), constant d > 0. Sum:f(n) + g(n) = O(F(n)) + O(G(n)) = O(max{ F(n), G(n) }).Product:f(n) · g(n) = O(F(n)) · O(G(n)) = O(F(n)) · G(n) ). f(n) · h(n) = O(F(n)) · Q(H(n)) =O( F(n) · H(n) ).Division: Power: f(n)d = O( F(n)d ). In addition to O , these rules also apply to Q, W, o, w. • Examples: • . • . • Q

  28. Proof of Rule of Sum f(n) = O(F(n)) & g(n) = O(G(n)) f(n)+g(n) = O(max{F(n),G(n)}) Proof: From the premise we have:n1, c1 >0: n  n1 ,f(n)  c1 F(n)n2, c2 >0: n  n2 ,g(n)  c2 G(n) Now define n0 = max{n1 ,n2} > 0 & c = c1 +c2 > 0. We get: n  n0 ,f(n)+g(n)  c1 F(n) + c2 G(n)  c1max{F(n), G(n)} + c2 max{F(n), G(n)} = (c1 + c2) max{F(n), G(n)} = c max{F(n), G(n)} We have shown: n0 , c > 0:n  n0 ,f(n) + g(n)  c max{F(n), G(n)}. Therefore, f(n) + g(n) = O( max{F(n), G(n)} ).

  29. Where did we go wrong! Do you buy this? Where did we go wrong? META RULE: Within an expression, you can apply the asymptotic simplification rules only a constant number of times!WHY?

  30. Algorithm ComplexityvsProblem Complexity

  31. Algorithm Time Complexity • T(n) = worst-case time complexity of algorithm ALG. • T(n) = O(f(n)):You must prove that on every input of size n, for all sufficiently largen, ALG takes at most O(f(n)) time. That is, even the worst input of size n cannot force ALG to require more than O(f(n)) time. • T(n) = W(f(n)):Demonstrate the existence of an input instance In of size n for infinitely manyn, for which ALG requires at least W(f(n)) time. Just one n or a finite # of sample n’s won’t do, because for all n beyond those samples, ALG could go on cruise speed! • T(n) = Q(f(n)): Do both!

  32. Problem Time Complexity • T(n) = worst-case time complexity of problem P is the time complexity of the “best” algorithm that solves P. • T(n) = O(f(n)):Need to demonstrate (the existence of) an algorithm ALG that correctly solves problem P, and that worst-case time complexity of ALG is at most O(f(n)). • T(n) = W(f(n)):Prove that for every algorithm ALG that correctly solves every instance of problem P, worst-case time complexity of ALG must be at least W(f(n)). • T(n) = Q(f(n)): Do both! This is usually the much harder task

  33. Example Problem: Sorting Some sorting algorithms and their worst-case time complexities: Quick-Sort: Q(n2) Insertion-Sort: Q(n2) Selection-Sort: Q(n2) Merge-Sort: Q(n log n) Heap-Sort: Q(n log n) there are infinitely many sorting algorithms! Shown in Slide 5: essentially every algorithm that solves theSORTING Problem requires at least W(n log n) time in the worst-case. So, Merge-Sort and Heap-Sort are worst-case optimal, and SORTING complexity is Q(n log n).

  34. 11 4 5 2 15 7 4 11 5 2 15 7 4 5 11 2 15 7 4 5 2 11 15 7 4 5 2 11 15 7 4 2 5 7 11 15 Insertion Sortan incremental algorithm

  35. Insertion Sort: Time Complexity • AlgorithmInsertionSort(A[1..n]) • Pre-Condition: A[1..n] is an array of n numbers • Post-Condition: A[1..n] permuted in sorted order • for i  2 .. n do • LI: A[1..i –1] is sorted, A[i..n] is untouched.§ insert A[i] into sorted prefix A[1..i–1] by right-cyclic-shift: • key  A[i] • j  i –1 • while j > 0 and A[j] > key do • A[j+1]  A[j] • j  j –1 • end-while • A[j+1]  key • 9. end-for • end

  36. Exercises

  37. We listed a number of facts and showed the proofs of some of them. Prove the rest. • For each f(n) below, give the most simplified answer to f(n) = Q(?) & a brief explanation. • Are there any differences between Q(1)n and Q(2n) and 2Q(n)? Explain. • Are any of the following implications always true? Prove or give a counter-example. a) f(n) = Q(g(n))  f(n) = cg(n) + o(g(n)), for some real constant c > 0. b) f(n) = Q(g(n))  f(n) = cg(n) + O(g(n)), for some real constant c > 0. • Show that 2O(log log n) = logO(1) n . • Prove or disprove: n5= Q( n5+o(1) ).

  38. [CLRS: Problem 3-4, p. 62] Asymptotic notation propertiesLet f(n) and g(n) be asymptotically positive functions. Prove or disprove each of the following conjectures.(a) f(n) = O(g(n)) implies g(n) = O(f(n)).(b) f(n) + g(n) = ( min ( f(n) , g(n) ) ).(c) f(n) = O(g(n)) implies log(f(n)) = O(log (g(n))), where log(g(n)) ≥ 1 and f(n)≥ 1 for all sufficiently large n.(d) f(n) = O(g(n)) implies 2f(n) = O( 2g(n) ).(e) f(n) = O( (f(n))2 ).(f) f(n) = O(g(n)) implies g(n) = (f(n)).(g) f(n) = ( f(n/2) ).(h) f(n) + o(f(n)) = (f(n)).

  39. We have: So, is equal to or ? Which one? Prove or disprove: The following implications hold. (a) f(n) = Q(g(n)) f(n)h(n) = Q( g(n)h(n) ) .(b) f(n) = Q(g(n)) and h(n) = Q(1) f(n)h(n) = Q( g(n)h(n) ) .[Note: Compare with “Power Rule”.] Is it true that for every pair of functions f(n) and g(n) we must have at least one of: f(n) = O(g(n)) or g(n) = O(f(n))?

  40. Demonstrate two functions f: NN and g: NN with the following properties:Both f and g are strictly monotonically increasing, and f  O(g), and g  O(f). • Show that . • Rank the following functions by asymptotic order of growth. Put them in equivalenceclasses based on Q-equality. That is, f and g are in the same class iff f = Q(g), and f is in an earlier class than g if f = o(g).

  41. We say a function f: N+ is asymptotically additive if f(n) + f(m) = Q(f(n+m)). Which of the following functions are asymptotically additive? Justify your answer. i) Logarithm: log n, (Assume n > 1.) ii) Monomial: nd, for any real constant d  0 (this includes, e.g., n with d=0.5.) iii) Harmonic: 1/n. (Assume n > 0.) iv) Exponential: an, for any real constant a > 1.Prove or disprove:If f and g are asymptotically additive and are positive. then so is: i) a · f , for any real constant a > 0, ii) f + g, iii) f – g, assuming (f – g)(n) is always positive. Extra credit is given if f – g is monotonically increasing. iv) f · g . v) f g. • The Set Disjointness Problem is defined as follows: We are given two sets A and B, each containing n arbitrary numbers. We want to determine whether their intersection is empty, i.e., whether they have any common element. • Design the most efficient algorithm you can to solve this problem and analyze its worst-case time complexity. • What is the time complexity of the Set Disjointness Problem itself? [You are not yet equipped to answer part (b). The needed methods will be covered in Lecture Slide 5.]

  42. END

More Related