Divide and Conquer Algorithms

1. Divide and Conquer Algorithms CS 4102: Algorithms Spring 2011 Aaron Bloomfield

2. Recurrences and Divide & Conquer • First design strategy: Divide and Conquer • Examples… • Recursive algorithms • Counting basic operations in recursive algorithms • Solving recurrence relations • By iteration method • Recursion trees (quick view) • The “Main” and “Master” Theorems • Mergesort • Trominos

3. Recursion

4. Recursion: Basic Concepts and Review • Recursive definitions in mathematics • Factorial: n! = n (n-1)! and 0! = 1! = 1 • Fibonacci numbers: F(0) = F(1) = 1 F(n) = F(n-1) + F(n-2) for n > 1 • Note base case • In programming, recursive functions can be implemented • First, check for simple solutions and solve directly • Then, solve simpler subproblem(s) by calling same function • Must make progress towards base cases • Design strategy: method99 “mental trick”

5. Designing Recursive Procedures • Think Inductively! • Converging to a base case (stopping the recursion) • identify some unit of measure (running variable) • identify base cases • How to solve p for all inputs from size 0 through 100 • Assume method99 solves sub-problem all sizes 0 through 99 • if p detect a case that is not base case it calls method99 • method99 works and is called when: 1. The sub-problem size is less than p’s problem size 2. The sub-problem size is not below the base case 3. The sub-problem satisfies all other preconditions of method99 (which are the same as the preconditions of p)

6. Recursion: Good or Evil? • It depends… • Sometimes recursion is an efficient design strategy, sometimes not • Important! we can define recursively and implement non-recursively • Note that many recursive algorithms can be re-written non-recursively • Use an explicit stack • Remove tail-recursion (compilers often do this for you) • Consider: factorial, binary search, Fibonacci • Let’s consider Fibonacci carefully…

7. Implement Fibonacci numbers • It’s beautiful code, no?long fib(int n) { assert(n >= 0); if ( n == 0 ) return 1; if ( n == 1 ) return 1; return fib(n-1) + fib(n-2);} • Let’s run and time it. • Let’s trace it.

9. Towers of Hanoi • Ah, the legend: • 64 golden disks • Those diligent priests • The world ends!

10. Towers of Hanoi • Back in the commercial Western world… • Game invented by the French mathematician, Edouard Lucas, in 1883. • Now, for only $19.95, call now!

11. Your turn to design! • Write a recursive function for the Towers of Hanoi. • Number each peg: 1, 2, 3 • Function signature:hanoi ( n, source, dest, aux)where: n is number of disks (from the top), and other parameters are peg valuesIn function body print: Move a disk from <peg> to <peg> • Do this in pairs. Then pairs group and compare. Find bugs, issues, etc. Explain to each other.Turn in one sheet with all four names.

12. Divide & Conquer

13. Divide and Conquer: A Strategy • Our first design strategy: Divide and Conquer • Often recursive, at least in definition • Strategy: • Break a problem into 1 or more smaller subproblems that are identical in nature to the original problem • Solve these subproblems (recursively) • Combine the results for the subproblems (somehow) to produce a solution to original problem • Note the assumption: • We can solve original problem given subproblems’ solutions

14. Design Strategy: Divide and Conquer • It is often easier to solve several small instances of a problem than one large one. • divide the problem into smaller instances of the same problem • solve (conquer) the smaller instances recursively • combine the solutions to obtain the solution for original input • Must be able to solve one or more small inputs directly • Solve(I) n = size(I) if (n <= smallsize) solution = directlySolve(I); else divide I into I1, …, Ik. for each i in {1, …, k} Si = solve(Ii); solution = combine(S1, …, Sk); return solution;

15. Why Divide and Conquer? • Sometimes it’s the simplest approach • Divide and Conquer is often more efficient than “obvious” approaches • E.g. Mergesort, Quicksort • But, not necessarily efficient • Might be the same or worse than another approach • Must analyze cost • Note: divide and conquer may or may not be implemented recursively

16. Cost for a Divide & Conquer Algorithm • Perhaps there is… • A cost for dividing into sub problems • A cost for solving each of several subproblems • A cost to combine results • So (for n > smallSize) • T(n) = D(n) + ΣT(size(Ii) + C(n)) • often rewritten as: T(n) = a T(n/b) + f(n) • These formulas are recurrence relations

17. Mergesort is Classic Divide & Conquer • Mergesort Strategy:

18. Algorithm: Mergesort • Specification: • Input: Array E and indexes first and last, such that the elements E[i] are defined for first <= i <= last. • Output: E[first], …, E[last] is sorted rearrangement of the same elements • Algorithm: def mergesort(list, first, last): if first < last: mid = (first+last)/2 mergesort(list, first, mid) mergesort(list, mid+1, last) merge(list, first, mid, last) # merge 2 halves return

19. Exercise: Find Max and Min • Given a list of elements, find both the maximum element and the minimum element • Obvious solution: • Consider first element to be max • Consider first element to be min • Scan linearly from 2nd to last, and update if something larger then max or if something smaller than min • Another way: • Write a recursive function that solves this using divide and conquer. • Prototype: void maxmin (list, first, last, max, min); • Base case(s)? Subproblems? How to combine results?

20. Recurrence Relations

21. Solving Recurrence Relations • Several methods: • Substitution method, AKA iteration method, AKA method of backwards substitutions • We’ll do this in class • Recurrence trees • We won’t see this in great detail, but a graphical view of the recucrrence • Sometimes a picture is worth 210 words! • “Main” Theorem and the “Master” theorem • Easy to find Order-Class for a number of common cases • Different variations are called different things, depending on the source

22. Iteration or Substitution Method • Strategy • Write out recurrence, e.g. W(n) = W(n/2) + 1 • BTW, this is a recurrence for binary search • Substitute for the recursive definition on the right-hand side by re-applying the general formula with the smaller value • In other words, plug the smaller value back into the main recurrence • So now: W(n) = ( W(n/4) + 1 ) + 1 • Repeat this several times and write it in a general form (perhaps using some index i to show how often it’s repeated) • So now: W(n) = W(n/2i) + i

23. Substitution Method (cont’d) • So far we have: W(n) = W(n/2i) + i • This is the form after we repeat i times. How many times can we repeat? • Use base case to solve for i • Base case is W(1) = 1 • So we set 1 = n/2i • Solve for i: so i = lg n • Plug this value of i back into the general recurrence: W(n) = W(n/2i) + i = W(n/n) + lg n = lg n + 1 • Note: We assume n is some power of 2, right? • That’s OK. There is a theorem called the smoothness rule that states that we’ll have the correct order-class

24. Examples Using the Substitution Method • Practice with the following: • Finding max and min W(1) = 0, W(n) = 2 W(n/2) + 2 • Is this better or worse than the “scanning” approach? • Mergesort W(1) = 0, W(n) = 2 W(n/2) + n - 1 • Towers of Hanoi • Write the recurrence. (Now, in class.) • Solve it. (At home!)

25. Return to Fibonacci… • Can we use the substitution method to find out the W(n) for our recursive implementation of fib(n)? • Nope. There’s another way to solve recurrence, which we won’t do in this class • homogenous second-order linear recurrence with constant coefficients • This method allows us to calculate F(n) “directly”: • F(n) = (1 / sqrt(5) ) n rounded to nearest int, where  is the Golden Ratio, about 1.618 • Isn’t this (1), whereas a loop is (n)? (Just punch buttons on my calculator!) • Without a table or a calculator, finding n is linear (just like finding F(n) with a loop)

26. Evaluate recursive equationusing Recursion Tree • Evaluate: T(n) = T(n/2) + T(n/2) + n • Work copy: T(k) = T(k/2) + T(k/2) + k • For k=n/2, T(n/2) = T(n/4) + T(n/4) + (n/2) • [size| non-recursive cost]

27. Fibonacci upper bound • Fibonacci recurrence: • F(n) = F(n-1) + F(n-2) • F(0) = F(1) = 1 • We’ll upper bound it by using: • F(n) = 2 F(n-1) • Since F(n-1) > F(n-2) • This is easier to solve via the iteration method

28. Recursion Tree: Total Cost • To evaluate the total cost of the recursion tree • sum all the non-recursive costs of all nodes • = Sum (rowSum(cost of all nodes at the same depth)) • Determine the maximum depth of the recursion tree: • For our example, at tree depth d the size parameter is n/(2d) • the size parameter converging to base case, i.e. case 1 • such that, n/(2d) = 1, • d = lg(n) • The rowSum for each row is n • Therefore, the total cost, T(n) = n lg(n)

29. The Master Theorem • Given: a divide and conquer algorithm • An algorithm that divides the problem of size n into asubproblems, each of size n/b • Let the cost of each stage (i.e., the work to divide the problem + combine solved subproblems) be described by the function f(n) • Then, the Master Theorem gives us a cookbook for the algorithm’s running time • Some textbooks has a simpler version they call the “Main Recurrence Theorem” • We’ll splits it into individual parts

30. The Master Theorem (from Cormen) • If T(n) = aT(n/b) + f(n) • then let k = lg a / lg b = logb a (critical exponent) • Then three common cases based on how quickly f(n) grows • If f(n)  O(nk-) for some positive , then T(n) (nk) • If f(n) (nk) then T(n) ( f(n) log(n) ) = (nk log(n)) • If f(n) (nk-) for some positive , and f(n)  O(nk+) for some positive  >= , then T(n) (f(n)) • Note: none of these cases may apply

31. The Main Recurrence Theorem • A somewhat simpler version of the master theorem • If T(n) = aT(n/b) + f(n) and f(n) = (nk) • Cases for exact bound: • T(n) (nk) if a < bk • T(n) ( nk log(n) ) if a = bk • T(n) (nE) where E=logb(a) if a > bk • Note f(n) is polynomial • This is less general than earlier Master Theorem

32. Using these two methods • T(n) = 9T(n/3) + n • A = 9, b = 3, f(n) = n • Main Recurrence Theorem • f(n) = n = n1, thus k=1 • a ? bk 9 > 31, so (nE) where E=log3(9) = 2, (n2) • Master Theorem • k = lg 9 / lg 3 = log3 9 = 2 • Since f(n) = O(nlog3 9 - ), where =1, case 1 applies:T(n) (nE) • Thus the solution is T(n) = (n2) since E=2

33. Problems to Try • Can you use a theorem on these? • Can you successfully use the iteration method? • Assume T(1) = 0 • T(n) = T(n/2) + lg n • T(n) = T(n/2) + n • T(n) = 2T(n/2) + n (like Mergesort) • T(n) = 2T(n/2) + n lg n

34. Common Forms of Recurrence Equations • Recognize these: • Divide and conquer T(n) = bT(n/c) + f(n) • Solve through iteration or main/master theorems • Chip and conquer: T(n) = T(n-c) + f(n) • Note: One sub-problem of lesser cost! • Running time is n*f(n) • Chip and Be Conquered: T(n) = b T(n-c) + f(n) where b > 1 • Like Towers of Hanoi

35. Back to Towers of Hanoi • Recurrence: W(1) = 1; W(n) = 2 W(n-1) +1 • Closed form solution: W(n) = 2n – 1 • Original “legend” says the monks moves 64 golden disks • And then the world ends! (Uh oh.) • That’s 18,446,744,073,709,551,615 moves! • If one move per second, day and night, then580 billion years • Whew, that’s a relief!

36. Closest Pairs

37. Problem: Find Closest Pair of Points • Given a set of points in 2-space, find a pair that has the minimum distance between them • Distance is Euclidean distance • A computational geometry problem… • And other applications where distance is some similarity measure • Pattern recognition problems • Items identified by a vector of scores • Graphics • VLSI • Etc.

38. Obvious Solution: Closest Pair of Points • For the complete set of n(n-1)/2 pairings, calculate the distances and keep the smallest • (n2)

39. An aside: k Nearest Neighbors problem • How to find the “k nearest neighbors” of a given point X? • Pattern recognition problem • All points belong to a category, say “cancer risk” and “not at risk”. • Each point has a vector of size n containing values for some set of features • Given an new unclassified point, find out which category it is most like • Find its k nearest neighbors and use their classifications to decide (i.e. they “vote”) • If k=1 then this is the closest point problem for n=2

40. Solving k-NN problem • Obvious solution: • Calculate distance from X to all other points • Store in a list, sort the list, choose the k smallest • Better solution, better data structure? (think back to CS2150) • Keep a max-heap with the k smallest values seen so far • Calculate distance from X to the next point • If smaller than the heap’s root, remove the root and insert that point into the heap • Why a max-heap?

41. Back to Closest Pairs • How’s it work? • See class notes (done on board), or the textbook

42. Summary of the Algorithm • Strategy: • Sort points by x-coordinate • Divide into two halves along x-coordinate. • Get closest pair in first-half, closest-pair in second-half. Let  be value of the closest of these two. • In recursion, if 3 points or fewer, solve directly to find closest pair. • Gather points in strip of width 2 into an array Sy • For each point in Sy • Look at the next 7 (or 15) points in Sy to see if they closer than 

43. Analysis: Closest Pairs • What are we counting exactly? • Several parts of this algorithm. No single basic-operation for the whole thing • (1) Sort all points by x-coordinate: Θ(n lgn) • (2) Recurrence: T(3) = k T(n) = 2T(n/2) + cn • Checking the strip is clearly O(n) • This is Case 2 of the Main Theorem, so the recursive part is also (n log n) • T(n) = 2T(n/2) + n + c

44. Mergesort

45. New Problem: Sorting a Sequence • The problem: • Given a sequence a0 … anreorder them into a permutation a’0 … a’nsuch that a’i <= a’i+1 for all pairs • Specifically, this is sorting in non-descending order… • Basic operation • Comparison of keys. Why? • Controls execution, so total operations often proportional • Important for definition of a solution • Often an expensive operation (say, large strings are keys) • However, swapping items is often expensive • We can apply same techniques to count swapping in a separate analysis

46. Why Do We Study Sorting? • An important problem, often needed • Often users want items in some order • Required to make many other algorithms work well. Example: For searching on sorted data by comparing keys, optimal solutions require (log n) comparisons using binary search • And, for the study of algorithms… • A history of solutions • Illustrates various design strategies and data structures • Illustrates analysis methods • Can prove something about optimality

47. Mergesort is Classic Divide & Conquer • Strategy

48. Algorithm: Mergesort • Specification: • Input: Array list and indexes first, and Last, such that the elements list[i] are defined for first <= i <= last. • Output: list[first], …, list[last] is sorted rearrangement of the same elements • Algorithm: def mergesort(list, first, last): if first < last: mid = (first+last)/2 mergesort(list, first, mid) mergesort(list, mid+1, last) merge(list, first, mid, last) return

49. Exercise: Trace Mergesort Execution • Can you trace MergeSort() on this list?A = {8, 3, 2, 9, 7, 1, 5, 4};

50. Efficiency of Mergesort • Cost to divide in half? No comparisons • Two subproblems: each size n/2 • Combining results? What is the cost of merging two lists of size n/2 • Soon we’ll see it’s n-1 in the worst-case • Recurrence relation: W(1) = 0 W(n) = 2 W(n/2)+ merge(n) = 2 W(n/2) + n-1You can now show that this W(n)  (n log n)