510 likes | 764 Views
Algorithm Analysis. 1. Purpose. Why bother analyzing code; isn’t getting it to work enough? Estimate time and memory in the average case and worst case Identify bottlenecks, i.e., where to reduce time Compare different approaches Speed up critical algorithms.
E N D
Algorithm Analysis Cpt S 223. School of EECS, WSU 1
Purpose • Why bother analyzing code; isn’t getting it to work enough? • Estimate time and memory in the average case and worst case • Identify bottlenecks, i.e., where to reduce time • Compare different approaches • Speed up critical algorithms Cpt S 223. School of EECS, WSU
Problem – Algorithm – Data Structures & Techniques When designing algorithms,we may end up breakingthe problem into smaller“sub-problems” Many algorithms can exist to solve oneproblem Algorithm (e.g., binary search) Problem (e.g., searching) solves contains uses Data structures & Techniques (e.g., sorting is a techniquearray is a data struct.) Specification (Input => Output) Cpt S 223. School of EECS, WSU
Algorithm Analysis • Predict resource utilization of an algorithm • Running time • Memory • Dependent on architecture & model • Serial • Parallel • Quantum • DNA • … Cpt S 223. School of EECS, WSU
Model that we will assume • Simple serial computing model CPU RAM (mainmemory) control Secondary storage(disk) I/O ALU INPUT Keyboard, mouse devices OUTPUT Monitor, printer registers Cpt S 223. School of EECS, WSU
Other Models • Multi-core models • Co-processor model • Multi-processor model • Shared memory machine • Multiple processors sharing common RAM • Memory is cheap ($20 per GB) • Memory bandwidth is NOT • Cray XMT Supercomputer • Up to 64 TB (65,536 GB) shared memory • Up to 8000 processors • 128 independent threads per processor • $150M Cpt S 223. School of EECS, WSU
Other Models Distributed memory model www.top500.org Supercomputer speed is measured in number offloating point operations per sec (or FLOPS) • Fastest supercomputer (as of June 2012): • Sequoia @ Lawrence Livermore National Lab • 16.32 PetaFlop/s (16.32 x 1015 floating point ops per sec) • IBM BlueGene/Q architecture • #processing cores: 1.5M cores • #aggregate memory: 1,572 TB • Price tag: millions of $$$ Cpt S 223. School of EECS, WSU 9
What to Analyze: T(n) • Running time T(n) • N or n is typically used to denote the size of the input • Sorting? • Multiplying two integers? • Multiplying two matrices? • Traversing a graph? • T(n) measures number of primitive operations performed • E.g., addition, multiplication, comparison, assignment Cpt S 223. School of EECS, WSU
How to Analyze T(n)? • As the input (n) grows what happens to the time T(n)? Cpt S 223. School of EECS, WSU
Big-O O() Omega Ω() small-O o() small-omega () Theta () Worst-case Average-case Best-case “Algorithms” Lower-bound Upper-bound Tight-bound “Optimality” Algorithmic Notation & Analysis Describes the input Describes the problem & its solution Asymptotic notations that help us quantify & compare costsof different solutions Cpt S 223. School of EECS, WSU
Asymptotic Notation • Theta • Tight bound • Big-Oh • Upper bound • Omega • Lower bound The main idea: Express cost relative to standard functions (e.g., lg n, n, n2, etc.) Cpt S 223. School of EECS, WSU
Some Standard Function Curves cubic quadratic linear (curves not to scale) Initial aberrations do NOT matter! Cpt S 223. School of EECS, WSU 17
Big-oh : O() What you want to measure A standard function (e.g., n, n2) ≤ • f(N) = O(g(N)) if there exist positive constants c and n0 such that: • f(N) ≤ c*g(N), for all N>n0 • Asymptotic upper bound, possibly tight Upper bound for f(N) Cpt S 223. School of EECS, WSU
c g(n) = n2 f(n) = 10 n n0 Up & down fluctuation allowed in this zone Example for big-Oh • E.g., let f(n)=10 n • ==> f(n) = O(n2) • Proof: • If f(n) = O(g(n)) • ==> f(n) ≤ c g(n), for all n>n0 • (show such a <c,n0> combination exists) • ==> c=1, n0 = 9 • (Remember: always try to find the lowest possible n0) c=1 Breakevenpoint (n0) Cpt S 223. School of EECS, WSU
Ponder this • If f(n) = 10 n, then: • Is f(n) = O(n)? • Is f(n) = O(n2)? • Is f(n) = O(n3)? • Is f(n) = O(n4)? • Is f(n) = O(2n)? • … • If all of the above, then what is the best answer? • f(n) = O(n) Yes: e.g., for <c=10, n0=1> Yes: e.g., for <c=1, n0=9> Cpt S 223. School of EECS, WSU
Little-oh : o() < • f(N) = o(g(N)) if there exist positive constants c and n0 such that: • f(N)< c*g(N) when N>n0 • E.g., f(n)=10 n; g(n) = n2 • ==> f(n) = o(g(n)) Strict upper bound for f(N) Cpt S 223. School of EECS, WSU
Big-omega : Ω () ≥ • f(N) = Ω(g(N)) if there exist positive constants c and n0 such that: • f(N) ≥ c*g(N), for all N>n0 • E.g., f(n)= 2n2; g(n) =n log n ==> f(n) = Ω (g(n)) Lower bound for f(N), possibly tight Cpt S 223. School of EECS, WSU
Little-omega : () > • f(N) = (g(N)) if there exist positive constants c and n0 such that: • f(N) > c*g(N) when N>n0 • E.g., f(n)= 100 n2; g(n) =n ==> f(n) = (g(n)) Strict lower bound for f(N) Cpt S 223. School of EECS, WSU
Theta : Θ() f(N) = Θ(h(N)) if there exist positive constants c1,c2 and n0 such that: c1h(N) ≤f(N) ≤ c2h(N) for all N>n0 (same as) f(N) = Θ(h(N)) if and only if f(N) = O(h(N)) and f(N) = Ω(h(N)) ≡ Tight bound for f(N) Cpt S 223. School of EECS, WSU 24
Example (for theta) • f(n) = n2 – 2n f(n) = Θ(?) • Guess:f(n) = Θ(n2) • Verification: • Can we find valid c1, c2, and n0? • If true: c1n2≤f(n) ≤ c2n2 c1n2≤ n2 – 2n ≤ c2n2 … Cpt S 223. School of EECS, WSU
The Asymptotic Notations Cpt S 223. School of EECS, WSU
Asymptotic Growths c2 g(n) c g(n) f(n) f(n) f(n) c1 g(n) c g(n) n n n n0 n0 n0 f(n) = ( g(n) ) f(n) = O( g(n) )f(n) = o( g(n) ) f(n) = ( g(n) ) f(n) = ( g(n) ) Cpt S 223. School of EECS, WSU
Rules of Thumb while using Asymptotic Notations Algorithm’s complexity: • When asked to analyze an algorithm’s complexities: 1st Preference: Whenever possible, use () 2nd Preference: If not, use O() - or o() 3rd Preference: If not, use () - or () Tight bound Upper bound Lower bound Cpt S 223. School of EECS, WSU
Rules of Thumb while using Asymptotic Notations… Algorithm’s complexity: • Unless otherwise stated, express an algorithm’s complexity in terms of its worst-case Cpt S 223. School of EECS, WSU
Rules of Thumb while using Asymptotic Notations… Problem’s complexity • Ways to answer a problem’s complexity: Q1) This problem is at least as hard as … ? Use lower bound here Q2) This problem cannot be harder than … ? Use upper bound here Q3) This problem is as hard as … ? Use tight bound here Cpt S 223. School of EECS, WSU
A few examples • N2 = O(N2) = O(N3) = O(2N) • N2 = Ω(1) = Ω(N) = Ω(N2) • N2 = Θ(N2) • N2 = o(N3) • 2N2 + 1 = Θ(?) • N2 + N = Θ(?) • N3 - N2 = Θ(?) • 3N3 – N3 = Θ(?) Cpt S 223. School of EECS, WSU
Reflexivity • Is f(n) = Θ(f(n))? • Is f(n) = O(f(n))? • Is f(n) = Ω(f(n))? • Is f(n) = o(f(n))? • Is f(n) = (f(n))? yes reflexive yes yes no Not reflexive no Cpt S 223. School of EECS, WSU
Symmetry • f(n) = Θ(g(n)) “iff” g(n) = Θ(f(n)) “If and only if” Cpt S 223. School of EECS, WSU
Transpose symmetry • f(n) = O(g(n)) iff g(n) = Ω(f(n)) • f(n) = o(g(n)) iff g(n) = (f(n)) Cpt S 223. School of EECS, WSU
Transitivity • f(n) = Θ(g(n)) andg(n) = Θ(h(n)) f(n) = Θ(h(n)) • f(n) = O(g(n)) andg(n) = O(h(n)) ? • f(n) = Ω(g(n)) andg(n) = Ω(h(n)) ? • … Cpt S 223. School of EECS, WSU
additive multiplicative More rules… • Rule 1: If T1(n) = O(f(n)) and T2(n) = O(g(n)), then • T1(n) + T2(n) = O(f(n) + g(n)) • T1(n) * T2(n) = O(f(n) * g(n)) • Rule 2: If T(n) is a polynomial of degree k, then T(n) = Θ(nk) • Rule 3: logk n = O(n) for any constant k • Rule 4: logan = (logbn) for any constants a & b Cpt S 223. School of EECS, WSU
Some More Proofs • Prove that: n lg n = O(n2) • We know lg n ≤ n, for n≥1 ( n0=1) • Multiplying n on both sides: n lg n ≤ n2 n lg n ≤ 1. n2 Cpt S 223. School of EECS, WSU
Some More Proofs… • Prove that: 6n3≠O(n2) • By contradiction: • If 6n3=O(n2) • 6n3≤c n2 • 6n ≤c • It is not possible to bound a variable with a constant • Contradiction Cpt S 223. School of EECS, WSU
Maximum subsequence sum problem • Given N integers A1, A2, …, AN, find the maximum value (≥0) of: • Don’t need the actual sequence (i,j), just the sum • If final sum is negative, output 0 • E.g., [1, -4, 4, 2, -3, 5, 8, -2] 16 is the answer j i Cpt S 223. School of EECS, WSU
MaxSubSum: Solution 1 • Compute for all possible subsequence ranges (i,j) and pick the maximum range MaxSubSum1 (A) maxSum = 0 for i = 1 to N for j = i to N sum = 0 for k = i to j sum = sum + A[k] if (sum > maxSum) then maxSum = sum return maxSum (N) All possible start points (N) All possible end points (N) Calculate sum for range [ i .. j ] Total run-time = (N3) Cpt S 223. School of EECS, WSU
Solution 1: Analysis • More precisely = (N3) Cpt S 223. School of EECS, WSU
New code: MaxSubSum: Solution 2 New sum A[j] Old sum • Observation: • ==> So, re-use sum from previous range Old code: MaxSubSum2 (A) maxSum = 0 for i = 1 to N sum = 0 for j = ito N sum = sum + A[j] if (sum > maxSum) then maxSum = sum return maxSum MaxSubSum1 (A) maxSum = 0 for i = 1 to N for j = i to N sum = 0 for k = i to j sum = sum + A[k] if (sum > maxSum) then maxSum = sum return maxSum (N) (N) Total run-time = (N2) Sum (new k) = sum (old k) + A[k] => So NO need to recompute sum for range A[i..k-1] Cpt S 223. School of EECS, WSU
Solution 2: Analysis Can we do better than this? Use a Divide & Conquertechnique? Cpt S 223. School of EECS, WSU
MaxSubSum: Solution 3 • Recursive, “divide and conquer” • Divide array in half • A1..center and A(center+1)..N • Recursively compute MaxSubSum of left half • Recursively compute MaxSubSum of right half • Compute MaxSubSum of sequence constrained to use Acenter and A(center+1) • Return max { left_max, right_max, center_max } • E.g., <1, -4, 4, 2, -3, 5, 8, -2> maxleft maxright maxcenter Cpt S 223. School of EECS, WSU
MaxSubSum: Solution 3 MaxSubSum3 (A, i, j) maxSum = 0 if (i = j) then if A[i] > 0 then maxSum = A[i] else k = floor((i+j)/2) maxSumLeft = MaxSubSum3(A,i,k) maxSumRight = MaxSubSum3(A,k+1,j) compute maxSumThruCenter maxSum = Maximum (maxSumLeft, maxSumRight, maxSumThruCenter) return maxSum Cpt S 223. School of EECS, WSU
// how to find the max that passes through the center Keep right end fixed at center and vary left end Keep left end fixed at center+1 and vary right end Add the two to determine max through center Cpt S 223. School of EECS, WSU
Solution 3: Analysis • T(1) = Θ(1) • T(N) = 2T(N/2) + Θ(N) • T(N) = Θ(?) Can we do even better? T(N) = (N log N) Cpt S 223. School of EECS, WSU
Observation Any negative subsequence cannot be a prefix to the maximum sequence Or, only a positive, contiguous subsequence is worth adding MaxSubSum: Solution 4 1 3 6 8 0 4 16 14 E.g., <1, -4, 4, 2, -3, 5, 8, -2> MaxSubSum4 (A) maxSum = 0 sum = 0 for j = 1 to N sum = sum + A[j] if (sum > maxSum) then maxSum = sum else if (sum < 0) then sum = 0 return maxSum Can we do even better? T(N) = (N) Cpt S 223. School of EECS, WSU
MaxSubSum: Solution 5 (another (N) algorithm) • Let’s define: Max(i) <= maxsubsum for range A[1..i] • ==> Max(N) is the final answer • Let; Max’(i) <= maxsubsum ending at A[i] • Base case: • Max(1) = = Max’(1) = max { 0, A[1]} • Recurrence (i.e., FOR i=2 to N): • Max(i) = ? • = max { Max’(i), Max(i-1) } • Max’(i) = ? = max { Max’(i-1) + A[i], A[i], 0} Case:Max ends at i Case:Max does not end at i This technique is called “dynamic programming” Cpt S 223. School of EECS, WSU
Algorithm 5 pseudocode Dynamic programming: (re)use the optimal solution for a sub-problem to solve a larger problem MaxSubSum5 (A) if N==0 return 0 max = MAX(0,A[1]) max’ = MAX(0,A[1]) for j = 2 to N max’ = MAX(max’+A[j], A[j], 0) max = MAX(max, max’) return max T(N) = (N) Cpt S 223. School of EECS, WSU
What are the space complexities of all 5 versions of the code? MaxSubSum Running Times (n) and 5 Do not include array read times. 38 min 26 days Cpt S 223. School of EECS, WSU
MaxSubSum Running Times Cpt S 223. School of EECS, WSU
Logarithmic Behavior • T(N) = O(log2 N) • Usually occurs when • Problem can be halved in constant time • Solutions to sub-problems combined in constant time • Examples • Binary search • Euclid’s algorithm • Exponentiation For off-class reading: - read book - slides for lecture notes from course website Cpt S 223. School of EECS, WSU