1 / 51

Algorithm Analysis

Algorithm Analysis. 1. Purpose. Why bother analyzing code; isn’t getting it to work enough? Estimate time and memory in the average case and worst case Identify bottlenecks, i.e., where to reduce time Compare different approaches Speed up critical algorithms.

oshin
Download Presentation

Algorithm Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Algorithm Analysis Cpt S 223. School of EECS, WSU 1

  2. Purpose • Why bother analyzing code; isn’t getting it to work enough? • Estimate time and memory in the average case and worst case • Identify bottlenecks, i.e., where to reduce time • Compare different approaches • Speed up critical algorithms Cpt S 223. School of EECS, WSU

  3. Problem – Algorithm – Data Structures & Techniques When designing algorithms,we may end up breakingthe problem into smaller“sub-problems” Many algorithms can exist to solve oneproblem Algorithm (e.g., binary search) Problem (e.g., searching) solves contains uses Data structures & Techniques (e.g., sorting is a techniquearray is a data struct.) Specification (Input => Output) Cpt S 223. School of EECS, WSU

  4. Algorithm Analysis • Predict resource utilization of an algorithm • Running time • Memory • Dependent on architecture & model • Serial • Parallel • Quantum • DNA • … Cpt S 223. School of EECS, WSU

  5. Model that we will assume • Simple serial computing model CPU RAM (mainmemory) control Secondary storage(disk) I/O ALU INPUT Keyboard, mouse devices OUTPUT Monitor, printer registers Cpt S 223. School of EECS, WSU

  6. Other Models • Multi-core models • Co-processor model • Multi-processor model • Shared memory machine • Multiple processors sharing common RAM • Memory is cheap ($20 per GB) • Memory bandwidth is NOT • Cray XMT Supercomputer • Up to 64 TB (65,536 GB) shared memory • Up to 8000 processors • 128 independent threads per processor • $150M Cpt S 223. School of EECS, WSU

  7. Other Models Distributed memory model www.top500.org Supercomputer speed is measured in number offloating point operations per sec (or FLOPS) • Fastest supercomputer (as of June 2012): • Sequoia @ Lawrence Livermore National Lab • 16.32 PetaFlop/s (16.32 x 1015 floating point ops per sec) • IBM BlueGene/Q architecture • #processing cores: 1.5M cores • #aggregate memory: 1,572 TB • Price tag: millions of $$$ Cpt S 223. School of EECS, WSU 9

  8. What to Analyze: T(n) • Running time T(n) • N or n is typically used to denote the size of the input • Sorting? • Multiplying two integers? • Multiplying two matrices? • Traversing a graph? • T(n) measures number of primitive operations performed • E.g., addition, multiplication, comparison, assignment Cpt S 223. School of EECS, WSU

  9. How to Analyze T(n)? • As the input (n) grows what happens to the time T(n)? Cpt S 223. School of EECS, WSU

  10. Big-O O() Omega Ω() small-O o() small-omega () Theta () Worst-case Average-case Best-case “Algorithms” Lower-bound Upper-bound Tight-bound “Optimality” Algorithmic Notation & Analysis Describes the input Describes the problem & its solution Asymptotic notations that help us quantify & compare costsof different solutions Cpt S 223. School of EECS, WSU

  11. Asymptotic Notation • Theta • Tight bound • Big-Oh • Upper bound • Omega • Lower bound The main idea: Express cost relative to standard functions (e.g., lg n, n, n2, etc.) Cpt S 223. School of EECS, WSU

  12. Some Standard Function Curves cubic quadratic linear (curves not to scale) Initial aberrations do NOT matter! Cpt S 223. School of EECS, WSU 17

  13. Big-oh : O() What you want to measure A standard function (e.g., n, n2) ≤ • f(N) = O(g(N)) if there exist positive constants c and n0 such that: • f(N) ≤ c*g(N), for all N>n0 • Asymptotic upper bound, possibly tight Upper bound for f(N) Cpt S 223. School of EECS, WSU

  14. c g(n) = n2 f(n) = 10 n n0 Up & down fluctuation allowed in this zone Example for big-Oh • E.g., let f(n)=10 n • ==> f(n) = O(n2) • Proof: • If f(n) = O(g(n)) • ==> f(n) ≤ c g(n), for all n>n0 • (show such a <c,n0> combination exists) • ==> c=1, n0 = 9 • (Remember: always try to find the lowest possible n0) c=1 Breakevenpoint (n0) Cpt S 223. School of EECS, WSU

  15. Ponder this • If f(n) = 10 n, then: • Is f(n) = O(n)? • Is f(n) = O(n2)? • Is f(n) = O(n3)? • Is f(n) = O(n4)? • Is f(n) = O(2n)? • … • If all of the above, then what is the best answer? • f(n) = O(n) Yes: e.g., for <c=10, n0=1> Yes: e.g., for <c=1, n0=9> Cpt S 223. School of EECS, WSU

  16. Little-oh : o() < • f(N) = o(g(N)) if there exist positive constants c and n0 such that: • f(N)< c*g(N) when N>n0 • E.g., f(n)=10 n; g(n) = n2 • ==> f(n) = o(g(n)) Strict upper bound for f(N) Cpt S 223. School of EECS, WSU

  17. Big-omega : Ω () ≥ • f(N) = Ω(g(N)) if there exist positive constants c and n0 such that: • f(N) ≥ c*g(N), for all N>n0 • E.g., f(n)= 2n2; g(n) =n log n ==> f(n) = Ω (g(n)) Lower bound for f(N), possibly tight Cpt S 223. School of EECS, WSU

  18. Little-omega : () > • f(N) = (g(N)) if there exist positive constants c and n0 such that: • f(N) > c*g(N) when N>n0 • E.g., f(n)= 100 n2; g(n) =n ==> f(n) =  (g(n)) Strict lower bound for f(N) Cpt S 223. School of EECS, WSU

  19. Theta : Θ() f(N) = Θ(h(N)) if there exist positive constants c1,c2 and n0 such that: c1h(N) ≤f(N) ≤ c2h(N) for all N>n0 (same as) f(N) = Θ(h(N)) if and only if f(N) = O(h(N)) and f(N) = Ω(h(N)) ≡ Tight bound for f(N) Cpt S 223. School of EECS, WSU 24

  20. Example (for theta) • f(n) = n2 – 2n  f(n) = Θ(?) • Guess:f(n) = Θ(n2) • Verification: • Can we find valid c1, c2, and n0? • If true:  c1n2≤f(n) ≤ c2n2  c1n2≤ n2 – 2n ≤ c2n2  … Cpt S 223. School of EECS, WSU

  21. The Asymptotic Notations Cpt S 223. School of EECS, WSU

  22. Asymptotic Growths c2 g(n) c g(n) f(n) f(n) f(n) c1 g(n) c g(n) n n n n0 n0 n0 f(n) = ( g(n) ) f(n) = O( g(n) )f(n) = o( g(n) ) f(n) = ( g(n) ) f(n) =  ( g(n) ) Cpt S 223. School of EECS, WSU

  23. Rules of Thumb while using Asymptotic Notations Algorithm’s complexity: • When asked to analyze an algorithm’s complexities: 1st Preference: Whenever possible, use () 2nd Preference: If not, use O() - or o() 3rd Preference: If not, use () - or () Tight bound Upper bound Lower bound Cpt S 223. School of EECS, WSU

  24. Rules of Thumb while using Asymptotic Notations… Algorithm’s complexity: • Unless otherwise stated, express an algorithm’s complexity in terms of its worst-case Cpt S 223. School of EECS, WSU

  25. Rules of Thumb while using Asymptotic Notations… Problem’s complexity • Ways to answer a problem’s complexity: Q1) This problem is at least as hard as … ? Use lower bound here Q2) This problem cannot be harder than … ? Use upper bound here Q3) This problem is as hard as … ? Use tight bound here Cpt S 223. School of EECS, WSU

  26. A few examples • N2 = O(N2) = O(N3) = O(2N) • N2 = Ω(1) = Ω(N) = Ω(N2) • N2 = Θ(N2) • N2 = o(N3) • 2N2 + 1 = Θ(?) • N2 + N = Θ(?) • N3 - N2 = Θ(?) • 3N3 – N3 = Θ(?) Cpt S 223. School of EECS, WSU

  27. Reflexivity • Is f(n) = Θ(f(n))? • Is f(n) = O(f(n))? • Is f(n) = Ω(f(n))? • Is f(n) = o(f(n))? • Is f(n) = (f(n))? yes reflexive yes yes no Not reflexive no Cpt S 223. School of EECS, WSU

  28. Symmetry • f(n) = Θ(g(n)) “iff” g(n) = Θ(f(n)) “If and only if” Cpt S 223. School of EECS, WSU

  29. Transpose symmetry • f(n) = O(g(n)) iff g(n) = Ω(f(n)) • f(n) = o(g(n)) iff g(n) = (f(n)) Cpt S 223. School of EECS, WSU

  30. Transitivity • f(n) = Θ(g(n)) andg(n) = Θ(h(n)) f(n) = Θ(h(n)) • f(n) = O(g(n)) andg(n) = O(h(n)) ? • f(n) = Ω(g(n)) andg(n) = Ω(h(n)) ? • … Cpt S 223. School of EECS, WSU

  31. additive multiplicative More rules… • Rule 1: If T1(n) = O(f(n)) and T2(n) = O(g(n)), then • T1(n) + T2(n) = O(f(n) + g(n)) • T1(n) * T2(n) = O(f(n) * g(n)) • Rule 2: If T(n) is a polynomial of degree k, then T(n) = Θ(nk) • Rule 3: logk n = O(n) for any constant k • Rule 4: logan = (logbn) for any constants a & b Cpt S 223. School of EECS, WSU

  32. Some More Proofs • Prove that: n lg n = O(n2) • We know lg n ≤ n, for n≥1 ( n0=1) • Multiplying n on both sides: n lg n ≤ n2 n lg n ≤ 1. n2 Cpt S 223. School of EECS, WSU

  33. Some More Proofs… • Prove that: 6n3≠O(n2) • By contradiction: • If 6n3=O(n2) • 6n3≤c n2 • 6n ≤c • It is not possible to bound a variable with a constant • Contradiction Cpt S 223. School of EECS, WSU

  34. Maximum subsequence sum problem • Given N integers A1, A2, …, AN, find the maximum value (≥0) of: • Don’t need the actual sequence (i,j), just the sum • If final sum is negative, output 0 • E.g., [1, -4, 4, 2, -3, 5, 8, -2] 16 is the answer j i Cpt S 223. School of EECS, WSU

  35. MaxSubSum: Solution 1 • Compute for all possible subsequence ranges (i,j) and pick the maximum range MaxSubSum1 (A) maxSum = 0 for i = 1 to N for j = i to N sum = 0 for k = i to j sum = sum + A[k] if (sum > maxSum) then maxSum = sum return maxSum (N) All possible start points (N) All possible end points (N) Calculate sum for range [ i .. j ] Total run-time = (N3) Cpt S 223. School of EECS, WSU

  36. Solution 1: Analysis • More precisely = (N3) Cpt S 223. School of EECS, WSU

  37. New code: MaxSubSum: Solution 2 New sum A[j] Old sum • Observation: • ==> So, re-use sum from previous range Old code: MaxSubSum2 (A) maxSum = 0 for i = 1 to N sum = 0 for j = ito N sum = sum + A[j] if (sum > maxSum) then maxSum = sum return maxSum MaxSubSum1 (A) maxSum = 0 for i = 1 to N for j = i to N sum = 0 for k = i to j sum = sum + A[k] if (sum > maxSum) then maxSum = sum return maxSum (N) (N) Total run-time = (N2) Sum (new k) = sum (old k) + A[k] => So NO need to recompute sum for range A[i..k-1] Cpt S 223. School of EECS, WSU

  38. Solution 2: Analysis Can we do better than this? Use a Divide & Conquertechnique? Cpt S 223. School of EECS, WSU

  39. MaxSubSum: Solution 3 • Recursive, “divide and conquer” • Divide array in half • A1..center and A(center+1)..N • Recursively compute MaxSubSum of left half • Recursively compute MaxSubSum of right half • Compute MaxSubSum of sequence constrained to use Acenter and A(center+1) • Return max { left_max, right_max, center_max } • E.g., <1, -4, 4, 2, -3, 5, 8, -2> maxleft maxright maxcenter Cpt S 223. School of EECS, WSU

  40. MaxSubSum: Solution 3 MaxSubSum3 (A, i, j) maxSum = 0 if (i = j) then if A[i] > 0 then maxSum = A[i] else k = floor((i+j)/2) maxSumLeft = MaxSubSum3(A,i,k) maxSumRight = MaxSubSum3(A,k+1,j) compute maxSumThruCenter maxSum = Maximum (maxSumLeft, maxSumRight, maxSumThruCenter) return maxSum Cpt S 223. School of EECS, WSU

  41. Cpt S 223. School of EECS, WSU

  42. // how to find the max that passes through the center Keep right end fixed at center and vary left end Keep left end fixed at center+1 and vary right end Add the two to determine max through center Cpt S 223. School of EECS, WSU

  43. Cpt S 223. School of EECS, WSU

  44. Solution 3: Analysis • T(1) = Θ(1) • T(N) = 2T(N/2) + Θ(N) • T(N) = Θ(?) Can we do even better? T(N) = (N log N) Cpt S 223. School of EECS, WSU

  45. Observation Any negative subsequence cannot be a prefix to the maximum sequence Or, only a positive, contiguous subsequence is worth adding MaxSubSum: Solution 4 1 3 6 8 0 4 16 14 E.g., <1, -4, 4, 2, -3, 5, 8, -2> MaxSubSum4 (A) maxSum = 0 sum = 0 for j = 1 to N sum = sum + A[j] if (sum > maxSum) then maxSum = sum else if (sum < 0) then sum = 0 return maxSum Can we do even better? T(N) = (N) Cpt S 223. School of EECS, WSU

  46. MaxSubSum: Solution 5 (another (N) algorithm) • Let’s define: Max(i) <= maxsubsum for range A[1..i] • ==> Max(N) is the final answer • Let; Max’(i) <= maxsubsum ending at A[i] • Base case: • Max(1) = = Max’(1) = max { 0, A[1]} • Recurrence (i.e., FOR i=2 to N): • Max(i) = ? • = max { Max’(i), Max(i-1) } • Max’(i) = ? = max { Max’(i-1) + A[i], A[i], 0} Case:Max ends at i Case:Max does not end at i This technique is called “dynamic programming” Cpt S 223. School of EECS, WSU

  47. Algorithm 5 pseudocode Dynamic programming: (re)use the optimal solution for a sub-problem to solve a larger problem MaxSubSum5 (A) if N==0 return 0 max = MAX(0,A[1]) max’ = MAX(0,A[1]) for j = 2 to N max’ = MAX(max’+A[j], A[j], 0) max = MAX(max, max’) return max T(N) = (N) Cpt S 223. School of EECS, WSU

  48. What are the space complexities of all 5 versions of the code? MaxSubSum Running Times (n) and 5 Do not include array read times. 38 min 26 days Cpt S 223. School of EECS, WSU

  49. MaxSubSum Running Times Cpt S 223. School of EECS, WSU

  50. Logarithmic Behavior • T(N) = O(log2 N) • Usually occurs when • Problem can be halved in constant time • Solutions to sub-problems combined in constant time • Examples • Binary search • Euclid’s algorithm • Exponentiation For off-class reading: - read book - slides for lecture notes from course website Cpt S 223. School of EECS, WSU

More Related