Computer Algorithms Lecture 6 Recurrence Ch. 4 (till Master Theorem)

Computer AlgorithmsLecture 6Recurrence Ch. 4 (till Master Theorem) Some of these slides are courtesy of D. Plaisted et al, UNC and M. Nicolescu, UNR 1

Merge Sort Sorting Problem:Sort a sequence of n elements into non-decreasing order. • Divide: Divide the n-element sequence to be sorted into two subsequences of n/2 elements each • Conquer: Sort the two subsequences recursively using merge sort. • Combine: Merge the two sorted subsequences to produce the sorted answer.

Analysis of Merge Sort • Running time T(n) of Merge Sort: • Divide: computing the middle takes (1) • Conquer: solving 2 subproblems takes 2T(n/2) • Combine: merging n elements takes (n) • Total: • T(n) = (1)if n = 1 • T(n) = 2T(n/2) + (n)if n > 1  T(n) = (n lg n)

Recurrences Running time of algorithms with recursive callscan be described using recurrences A recurrence is an equation or inequality that describes a function in terms of its value on smaller inputs. For divide-and-conquer algorithms: Example: Merge Sort 4

Recurrence Relations • Equation or an inequality that characterizes a function by its values on smaller inputs. • Solution Methods (Chapter 4) • Iteration Method • Substitution Method. • Recursion-tree Method. • Master Method. • Recurrence relations arise when we analyze the running time of iterative or recursive algorithms. • Ex: Divide and Conquer. T(n) = (1)if n  c T(n) = a T(n/b) + D(n) + C(n)otherwise 5

Iteration Method T(n) = c + T(n/2) T(n) = c + T(n/2) = c + c + T(n/4) = c + c + c + T(n/8) Assume n = 2k T(n) = c + c + … + c + T(1) = clgn + T(1) = Θ(lgn) T(n/2) = c + T(n/4) T(n/4) = c + T(n/8) k times

Iteration Method – Example T(n) = n + 2T(n/2) T(n) = n + 2T(n/2) = n + 2(n/2 + 2T(n/4)) = n + n + 4T(n/4) = n + n + 4(n/4 + 2T(n/8)) = n + n + n + 8T(n/8) … = in + 2iT(n/2i) = kn + 2kT(1) = nlgn + nT(1) = Θ(nlgn) Assume: n = 2k T(n/2) = n/2 + 2T(n/4) 7

Iteration Method – Example T(n) = n + T(n-1) T(n) = n + T(n-1) = n + (n-1) + T(n-2) = n + (n-1) + (n-2) + T(n-3) … = n + (n-1) + (n-2) + … + 2 + T(1) = n(n-1)/2 - 1 + T(1) = n2/2 -n/2 -1 + T(1) = Θ(n2) 8

Substitution method Two steps Guess the form of the solution Use mathematical induction to find the constants and show the solution works Useful when it is easy to guess the form of the answer Can be used to establish either upper or lower bound on a recurrence Example: determine upper bound for guess: T(n) = O(n lg n) prove: T(n)  cn lg n for some c 9

Substitution method • Guess a solution • T(n) = O(g(n)) • Induction goal: apply the definition of the asymptotic notation • T(n) ≤ d g(n), for some d > 0 and n ≥ n0 • Induction hypothesis: T(k) ≤ d g(k) for all k < n • Prove the induction goal • Use the induction hypothesis to find some values of the constants d and n0 for which the induction goal holds

Example – Exact Function Recurrence: T(n) = 1 if n = 1 T(n) = 2T(n/2) + n if n > 1 • Guess:T(n) = n lgn + n. • Induction: • Basis: n = 1  n lgn + n = 1 = T(n). • Hypothesis:n = k/2: T(k/2) = (k/2) lg (k/2)+ (k/2) • Inductive Step: n = k: T(k) = 2 T(k /2) + k • = 2 ((k /2)lg(k /2) + (k /2)) + k • = k(lg(k /2)) + 2 k • = k lgk– k + 2 k • = klgk+ k

Example T(n) = T(n-1) + n • Guess: T(n) = O(n2) • Induction goal: T(n) ≤ c n2, for some c and n ≥ n0 • Induction hypothesis: T(n-1) ≤ c(n-1)2 • Proof of induction goal: T(n) = T(n-1) + n ≤ c (n-1)2 + n = cn2 – (2cn – c - n) ≤ cn2 if: 2cn – c – n ≥ 0  c ≥ n/(2n-1)  c ≥ 1/(2 – 1/n) • For n ≥ 1  2 – 1/n ≥ 1  any c ≥ 1 will work

Substitution Method. Summary • Guess the form of the solution, then use mathematical induction to show it correct. • Substitute guessed answer for the function when the inductive hypothesis is applied to smaller values – hence, the name. • Works well when the solution is easy to guess. • No general way to guess the correct solution.

Making a good guess Need experience and creativity Use recursion trees to generate good guesses If a recurrence is similar to one you are familiar, then guessing a similar solution is reasonable T(n) = O(nlgn)  Why? The additional term (17) cannot substantially affect the solution to the recurrence (when n is large) 14

Recursion-tree Method • Making a good guess is sometimes difficult with the substitution method. • Use recursion trees to devise good guesses. • Convert the recurrence into a tree: • Each node represents the cost incurred at that level of recursion • Sum up the costs of all levels • Recursion Trees • Show successive expansions of recurrences using trees. • Keep track of the time spent on the subproblems of a divide and conquer algorithm. • Help organize the algebraic bookkeeping necessary to solve a recurrence.

Example • Running time of Merge Sort: T(n) = (1)if n = 1 T(n) = 2T(n/2) + (n)if n > 1 • Rewrite the recurrence as T(n) = cif n = 1 T(n) = 2T(n/2) + cnif n > 1 c > 0:Running time for the base case and time per array element for the divide and combine steps.

cn Cost of divide and merge. cn cn/2 cn/2 T(n/2) T(n/2) T(n/4) T(n/4) T(n/4) T(n/4) Cost of sorting subproblems. Recursion Tree for Merge Sort For the original problem, we have a cost of cn, plus two subproblems each of size (n/2) and running time T(n/2). Each of the size n/2 problems has a cost of cn/2 plus two subproblems, each costing T(n/4).

cn cn/2 cn/2 cn/4 cn/4 cn/4 cn/4 c c c c c c Recursion Tree for Merge Sort Continue expanding until the problem size reduces to 1. cn cn lg n cn cn Total : cnlgn+cn

cn cn/2 cn/2 cn/4 cn/4 cn/4 cn/4 c c c c c c Recursion Tree for Merge Sort Continue expanding until the problem size reduces to 1. • Each level has total cost cn. • Each time we go down one level, the number of subproblems doubles, but the cost per subproblem halves  cost per level remains the same. • There are lgn + 1 levels, height is lgn. (Assuming n is a power of 2.) • Can be proved by induction. • Total cost = sum of costs at each level = (lgn + 1)cn = cnlgn + cn = (n lgn).

T(n) = 2T(n/2) + n2 Subproblem size at level i is: n/2i Subproblem size hits 1 when 1 = n/2i  i = lgn Cost of the problem at level i = (n/2i)2 No. of nodes at level i = 2i Total cost:  T(n) = O(n2) T(n/2) T(n/2) T(n/4) T(n/4) T(n/4) T(n/4) T(n/2)=2(T/4)+(n/2)2 T(n/4)=2(T/8)+(n/4)2 Recursion trees? T(1) T(1) T(1)T(1) T(1)

Example of recursion tree Solve T(n) = T(n/4) + T(n/2) + n2:

Example of recursion tree Solve T(n) = T(n/4) + T(n/2) + n2: T(n)

T(n/2) T(n/4) Example of recursion tree Solve T(n) = T(n/4) + T(n/2) + n2: n2

Example of recursion tree Solve T(n) = T(n/4) + T(n/2) + n2: n2 (n/2)2 (n/4)2 T(n/8) T(n/4) T(n/16) T(n/8)

Example of recursion tree Solve T(n) = T(n/4) + T(n/2) + n2: n2 (n/2)2 (n/4)2 (n/8)2 (n/4)2 (n/16)2 (n/8)2 … Q(1)

Example of recursion tree Solve T(n) = T(n/4) + T(n/2) + n2: n2 (n/2)2 (n/4)2 (n/8)2 (n/4)2 (n/16)2 (n/8)2 … … Q(1)

Example of recursion tree Solve T(n) = T(n/4) + T(n/2) + n2: n2 (n/2)2 (n/4)2 (n/8)2 (n/4)2 (n/16)2 (n/8)2 … … Q(1) Total = = Q(n2) geometric series

for x¹ 1 for |x| < 1 Appendix: geometric series Return to last slide viewed.

T(n) = 3T(n/4) + cn2 • Subproblem size at level i is: n/4i • Subproblem size hits 1 when 1 = n/4i  i = log4n • Cost of a node at level i = c(n/4i)2 • Number of nodes at level i = 3i last level has 3log4n = nlog43 nodes • Total cost:  T(n) = O(n2)

T(n) = T(n/3) + T(2n/3) + n for all levels • The longest path from the root to a leaf is: n  (2/3)n  (2/3)2 n  …  1 • Subproblem size hits 1 when 1 = (2/3)in  i=log3/2n • Cost of the problem at level i = n • Total cost:  T(n) = O(nlgn) T(1) T(1) T(1) T(1)

Other Examples • Use the recursion-tree method to determine a guess for the recurrences • T(n) = 3T(n/4) + (n2). • T(n) = T(n/3) + T(2n/3) + O(n).

Recursion Trees – Caution Note • Recursion trees only generate guesses. • Verify guesses using substitution method. • A small amount of “sloppiness” can be tolerated. Why? • If careful when drawing out a recursion tree and summing the costs, can be used as direct proof.

Recursion-Tree Method. Summary Recursion tree Each node represents the cost of a single subproblem somewhere in the set of recursive function invocations Each level denotes iteration Sum the costs within each level of the tree to obtain a set of per-level costs Sum all the per-level costs to determine the total cost of all levels of the recursion Useful when the recurrence describes the running time of a divide-and-conquer algorithm Useful for generating a good guess, which is then verified by the substitution method Use the recursion-tree method to “guess” the solution (rather than for proving) 35

Example Recurrences • T(n) = T(n-1) + n Θ(n2) • Recursive algorithm that loops through the input to eliminate one item • T(n) = T(n/2) + c Θ(lgn) • Recursive algorithm that halves the input in one step • T(n) = T(n/2) + n Θ(n) • Recursive algorithm that halves the input but must examine every item in the input • T(n) = 2T(n/2) + 1 Θ(n) • Recursive algorithm that splits the input into 2 halves and does a constant amount of other work 36

Three methods for solving recurrences (obtaining asymptotic Θ or O bounds on the solution) Substitution method Recursion-tree method Master method Notes – some technical details are neglected Assumption of integer arguments to functions: T(n) Boundary condition for small n  T(n) is constant for small n Floors, ceilings Mathematical induction is used to prove our solution for recurrence Showing base case holds Assume the nth step is true Prove the (n+1)th step by the result before the nth step 37

Computer Algorithms Lecture 6 Recurrence Ch. 4 (till Master Theorem)