1 / 37

Theory of Algorithms: Fundamentals of the Analysis of Algorithm Efficiency

Theory of Algorithms: Fundamentals of the Analysis of Algorithm Efficiency. James Gain and Edwin Blake {jgain | edwin} @cs.uct.ac.za Department of Computer Science University of Cape Town August - October 2004. Objectives. To outline a general Analytic Framework

fthorne
Download Presentation

Theory of Algorithms: Fundamentals of the Analysis of Algorithm Efficiency

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Theory of Algorithms:Fundamentals of the Analysis of Algorithm Efficiency James Gain and Edwin Blake {jgain | edwin} @cs.uct.ac.za Department of Computer Science University of Cape Town August - October 2004

  2. Objectives • To outline a general Analytic Framework • To introduce the conceptual tools of: • Asymptotic Notations • Base Efficiency Classes • To cover Techniques for the Analysis of Algorithms: • Mathematical (non-recursive and recursive) • Empirical • Visualisation

  3. Analysis of Algorithms • Issues: • Correctness (Is it guaranteed to produce a correct correspondence between inputs and outputs?) • Time efficiency (How fast does it run?) • Space efficiency (How much extra space does it require?) • Optimality (Is this provably the best possible solution?) • Approaches: • Theoretical analysis • Empirical analysis • Visualisation

  4. input size running time Number of times basic operation is executed execution time for basic operation Theoretical analysis of time efficiency • Time efficiency is analyzed by determining the number of repetitions of the basic operation as a function of input size • Basic operation: the operation that contributes most towards the running time of the algorithm. • T(n) ≈copC(n)

  5. Input size and basic operations

  6. Best-, Average- and Worst-cases • For some algorithms efficiency depends on type of input: • Worst case: W(n) – max over inputs of size n • Best case: B(n) – min over inputs of size n • Average case: A(n) – “avg” over inputs of size n • Number of times the basic operation will be executed on typical input • NOT the average of worst and best case • Expected number of basic operations repetitions considered as a random variable under some assumption about the probability distribution of all possible inputs of size n

  7. Amortized Efficiency • From a data structures perspective the total time of a sequence of operations is important • Real-time apps are an exception • Amortized efficiency: time of an operation averaged over a worst-case sequence of operations • Desire “self-organizing” data structures that adapt to the context of use, e.g. red-black trees

  8. Exercise: Sequential search • Problem: Given a list of n elements and a search key K, find an element equal to K, if any. • Algorithm: Scan the list and compare its successive elements with K until either a matching element is found (successful search) of the list is exhausted (unsuccessful search) • Calculate the: • Worst case? • Best case? • Average case?

  9. Types of Formulas for Operation Counts • Exact formula • e.g., C(n) = n(n-1)/2 • Formula indicating order of growth with specific multiplicative constant • e.g., C(n) ≈ 0.5 n2 • Formula indicating order of growth with unknown multiplicative constant • e.g., C(n) ≈cn2

  10. Order of Growth • Most important: Order of growth within a constant multiple as n∞ • Example: • How much faster will algorithm run on computer that is twice as fast? • How much longer does it take to solve problem of double input size?

  11. Asymptotic Notations • A way of comparing functions that ignores constant factors and small input sizes • O(g(n)): • class of functions f(n) that grow no faster than g(n) • f(n) ≤c g(n) for all n ≥n0 • Θ(g(n)): • class of functions f(n) that grow at same rate as g(n) • c2 g(n) ≤ f(n) ≤ c1 g(n) for all n ≥ n0 • Ω(g(n)): • class of functions f(n) that grow at least as fast as g(n) • f(n) ≥ c g(n) for all n ≥ n0

  12. Big-Oh: t(n)  O(g(n))

  13. Big-Omega: t(n)  Ω(g(n))

  14. Big-Theta: t(n)  Θ(g(n))

  15. Establishing Rate of Growth: using Limits • limn→∞ t(n)/g(n) = • 0 implies order of t(n) < order of g(n) • c implies order of t(n) = order of g(n) • ∞ implies order of t(n) > order of g(n) Example: • 10n vs. 2n2 • limn→∞ 10n/2n2 = 5 limn→∞ 1/n = 0 • Exercises: • n(n+1)/2 vs. n2 • logb n vs. logc n

  16. L’Hôpital’s rule • If limn→∞ f(n) = limn →∞ g(n) = ∞ and the derivatives f ', g'exist, Then limn→∞ f(n) / g(n) = limn →∞ f '(n) / g'(n) • Example: • log n vs. n • limn→∞ log n / n = limn→∞ 1/n log e = log e limn→∞ 1/n = 0 • So, order log n < order n

  17. Establishing Rate of Growth: using Definitions • f(n) is O(g(n)) if order of growth of f(n) ≤ order of growth of g(n) (within constant multiple) • There exist positive constant c and non-negative integer n0 such that • f(n) ≤ c g(n) for every n ≥ n0 • This needs to be mathematically provable • Examples: • 10n is O(2n2) because 10n < 2n2for n > 5 • 5n+20 is O(10n) because 5n+20 < 10n for n > 4

  18. Basic Asymptotic Efficiency Classes

  19. Time Efficiency of Non-recursive Algorithms • Steps in mathematical analysis of nonrecursive algorithms: • Decide on parameter n indicating input size • Identify algorithm’s basic operation • Determine worst, average, and best case for input of size n • Set up summation for C(n) reflecting algorithm’s loop structure • Simplify summation using standard formulas (see Appendix A of textbook)

  20. Examples: Analysing non-recursive algorithms • Matrix multiplication • Multiply two square matrices of order n using dot product of rows and columns • Selection sort • Find smallest element in remainder of list and swap with current element • Insertion sort • Assume sub-list is sorted an insert current element • Mystery Algorithm • Exercise

  21. Matrix Multiplication • n = matrix order • Basic op = multiplication or addition • Best case = worst case = average case

  22. Selection Sort • n = number of list elements • basic op = comparison • best case = worst case = average case

  23. Insertion Sort • n = number of list elements • basic op = comparision • best case != worst case != average case • best case: A[j] > v executed only once on each iteration

  24. Exercise: Mystery Algorithm • What does this algorithm compute? • What is its basic operation? • What is its efficiency class? • Suggest an improvement and analyse your improvement // Input: A matrix A[0..n-1, 0..n-1] of real numbers for i  0 to n - 1 do • for j i to n-1 do if A[i,j]  A[j,i] return false return true

  25. Factorial: a Recursive Function • Recursive Evaluation of n! • Definition: • n ! = 1*2*…*(n-1)*n • Recursive form for n!: • F(n) = F(n-1) * n, F(0) = 1 • Algorithm: • IF n=0 THEN F(n)  1 ELSE F(n) F(n-1) * n • RETURN F(n) • Need to find the number of multiplications M(n) to compute n!

  26. Recurrence Relations • Definition: • An equation or inequality that describes a function in terms of its value on smaller inputs • Recurrence: • The recursive step • E.g., M(n) = M(n-1) + 1 for n > 0 • Initial Condition: • The terminating step • E.g., M(0) = 0 [call stops when n = 0, no mults when n = 0] • Must differentiate the recursive function (factorial calculation) from the recurrence relation (number of basic operations)

  27. Recurrence Solution for n! • Need an analytic solution for M(n) • Solved by Method of Backward Substitution: • M(n) = M(n-1) + 1 • Substitute M(n-1) = M(n-2) + 1 • = [M(n-2) + 1] +1 = M(n-2) + 2 • Substitute M(n-2) = M(n-3) + 1 • = [M(n-3) + 1] + 2 = M(n-3) + 3 • Pattern: M(n) = M(n-i) + i • Ultimately: M(n) = M(n-n)+n = M(0) + n = n

  28. Plan for Analysis of Recursive Algorithms • Steps in mathematical analysis of recursive algorithms: • Decide on parameter n indicating input size • Identify algorithm’s basic operation • Determine worst, average, and best case for input of size n • (a) Set up a recurrence relation and initial condition(s) for C(n)-the number of times the basic operation will be executed for an input of size n OR (b) Alternatively, count recursive calls • (a) Solve the recurrence to obtain a closed form OR (b) Estimate the order of magnitude of the solution (see Appendix B)

  29. Iterative Methods for Solving Recurrences • Method of Forward Substitution: • Starting from the initial condition generate the first few terms • Look for a pattern expressible as a closed formula • Check validity by direct substitution or induction • Limited because pattern is hard to spot • Method of Backward Substitution: • Express x(n-1) successively as a function of x(n-2), x(n-3), … • Derive x(n) as a function of x(n-i) • Substitute n-i = base condition • Surprisingly successful

  30. Templates for Solving Recurrences • Decrease by One Recurrence • One (constant) operation reduces problem size by one • Examples: Factorial • T(n) = T(n-1) + c T(1) = d • Solution: T(n) = (n-1)c + d • Linear Order • A pass through input reduces problem size by one • Examples: Insertion Sort • T(n) = T(n-1) + cn T(1) = d • Solution: T(n) = [n(n+1)/2 – 1] c + d • Quadratic Order

  31. More Templates for Solving Recurrences • Decrease by a Constant Factor • One (constant) operation reduces problem size by half • Example: binary search • T(n) = T(n/2) + c T(1) = d • Solution: T(n) = c lg n + d • Logarithmic Order • A pass through input reduces problem size by half • Example: Merge Sort • T(n) = 2T(n/2) + cn T(1) = d • Solution: T(n) = cn lg n + d n • n log n order

  32. Master Method for Solving Recurrences • Divide and Conquer Style Recurrence • T(n) = aT(n/b) + f (n)where • f (n).(nk)is the time spent in dividing and merging • a >= 1, b >= 2 • a < bk T(n) .(nk) • a = bk T(n) .(nk lg n ) • a > bk T(n) .(nlog b a) • Note: the same results hold with O instead of .

  33. Example: Fibonacci Numbers • The Fibonacci sequence: • 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, … • Describes the growth pattern of Rabbits. Just short of exponential, ask Australia! • Fibonacci recurrence: • F(n) = F(n-1) + F(n-2) • F(0) = 0 • F(1) = 1 • Another example: • A(n) = 3A(n-1) - 2A(n-2) A(0) = 1 A(1) = 3 • 2ndorder linear homogeneous recurrence relation • with constant coefficients

  34. n F(n-1) F(n) 0 1 = F(n) F(n+1) 1 1 Computing Fibonnaci Numbers Algorithm Alternatives: • Definition based recursive algorithm • Nonrecursive brute-force algorithm • Explicit formula algorithm F(n) = 1/√5 n where is the golden ratio (1 + √5) / 2 • Logarithmic algorithm based on formula: • for n≥1, assuming an efficient way of computing matrix powers.

  35. Empirical Analysis of Time Efficiency • Sometimes a mathematical analysis is difficult for even simple algorithms (limited applicability) • Alternative is to measure algorithm execution • PLAN: • Understand the experiment’s purpose • Are we checking the accuracy of a theoretical result, comparing efficiency of different algorithms or implementations, hypothesising the efficiency class? • Decide on an efficiency measure • Use physical units of time (e.g., milliseconds) • OR • Count actual number of basic operations

  36. Plan for Empirical Analysis • Decide on Characteristics of the Input Sample • Can choose Pseudorandom inputs, Patterned inputs or a Combination • Certain problems already have benchmark inputs • Implement the Algorithm • Generate a Sample Set of Inputs • Run the Algorithm and Record the Empirical Results • Analyze the Data • Regression Analysis is often used to fit a function to the scatterplot of results

  37. Algorithm Visualisation • Definition: • Algorithm Visualisation seeks to convey useful algorithm information through the use of images • Flavours: • Static algorithm visualisation • Algorithm animation • Some new insights: e.g., odd and even disks in Towers of Hanoi • Attributes of Information Visualisation apply - zoom and filter, detail on demand, etc. • Websource: • The Complete Collection of Algorithm Animation • http://www.cs.hope.edu/~alganim/ccaa/

More Related