CSC401 – Analysis of Algorithms Chapter 5 – 2. Divide-and-Conquer

1. CSC401 – Analysis of Algorithms Chapter 5 – 2. Divide-and-Conquer • Objectives: • Introduce the Divide-and-conquer paradigm and variants: Decrease-and-Conquer and Transform-and-Conquer • Review the Merge-sort algorithm • Solve recurrence equations • Iterative substitution • Recursion trees • Guess-and-test • The master method • Discuss integer and matrix multiplications CSC401: Analysis of Algorithms

2. Divide-and conquer is a general algorithm design paradigm: Divide: divide the input data S in two or more disjoint subsets S1, S2, … Recur: solve the subproblems recursively Conquer: combine the solutions for S1,S2, …, into a solution for S The base case for the recursion are subproblems of constant size Analysis can be done using recurrence equations Divide-and-Conquer CSC401: Analysis of Algorithms

3. Merge-sort on an input sequence S with n elements consists of three steps: Divide: partition S into two sequences S1and S2 of about n/2 elements each Recur: recursively sort S1and S2 Conquer: merge S1and S2 into a unique sorted sequence 7 2  9 4 2 4 7 9 7  2 2 7 9  4 4 9 7 7 2 2 9 9 4 4 Merge-Sort Review AlgorithmmergeSort(S, C) Inputsequence S with n elements, comparator C Outputsequence S sorted • according to C ifS.size() > 1 (S1, S2)partition(S, n/2) mergeSort(S1, C) mergeSort(S2, C) Smerge(S1, S2) CSC401: Analysis of Algorithms

4. Recurrence Equation Analysis • The conquer step of merge-sort consists of merging two sorted sequences, each with n/2 elements and implemented by means of a doubly linked list, takes at most bn steps, for some constant b. • Likewise, the basis case (n< 2) will take at b most steps. • Therefore, if we let T(n) denote the running time of merge-sort: • We can therefore analyze the running time of merge-sort by finding a closed form solution to the above equation. • That is, a solution that has T(n) only on the left-hand side. CSC401: Analysis of Algorithms

5. Iterative Substitution • In the iterative substitution, or “plug-and-chug,” technique, we iteratively apply the recurrence equation to itself and see if we can find a pattern • Note that base, T(n)=b, case occurs when 2i=n. That is, i = log n. • So, • Thus, T(n) is O(n log n). CSC401: Analysis of Algorithms

6. The Recursion Tree • Draw the recursion tree for the recurrence relation and look for a pattern: Total time = bn + bn log n (last level plus all previous levels) CSC401: Analysis of Algorithms

7. Guess-and-Test Method • In the guess-and-test method, we guess a closed form solution and then try to prove it is true by induction: • Guess: T(n) < cnlogn. • Wrong: we cannot make this last line be less than cn log n CSC401: Analysis of Algorithms

8. Guess-and-Test Method: Part 2 • Recall the recurrence equation: • Guess #2: T(n) < cn log2 n. • if c > b. • So, T(n) is O(n log2 n). • In general, to use this method, you need to have a good guess and you need to be good at induction proofs. CSC401: Analysis of Algorithms

9. Master Method • Many divide-and-conquer recurrence equations have the form: • The Master Theorem: CSC401: Analysis of Algorithms

10. Master Method, Examples • The form: • The Master Theorem: • Example 1: • Solution: a=4, b=2, logba=2, f(n)=n, so case 1 says T(n) is O(n2). • Example 2: • Solution: a=b=2, logba=1, f(n)=nlogn, so case 2 says T(n) is O(n log2 n). CSC401: Analysis of Algorithms

11. Master Method, Examples • The form: • The Master Theorem: • Example 3: • Solution: a=1, b=3, logba=0, f(n)=nlogn, so case 3 says T(n) is O(nlogn). • Example 4: • Solution: a=2, b=8, logba=3, f(n)=n2, so case 1 says T(n) is O(n3). CSC401: Analysis of Algorithms

12. Master Method, Examples • The form: • The Master Theorem: • Example 5: • Solution: a=9, b=3, logba=2, f(n)=n3, so case 3 says T(n) is O(n3). • Example 6: (binary search) • Solution: a=1, b=2, logba=0, f(n)=1, so case 2 says T(n) is O(log n). • Example 7: (heap construction) • Solution: a=2, b=2, logba=1, f(n)=logn, so case 1 says T(n) is O(n). CSC401: Analysis of Algorithms

13. Iterative “Proof” of the Master Theorem • Using iterative substitution, let us see if we can find a pattern: • We then distinguish the three cases as • The first term is dominant • Each part of the summation is equally dominant • The summation is a geometric series CSC401: Analysis of Algorithms

14. Integer Multiplication • Algorithm: Multiply two n-bit integers I and J. • Divide step: Split I and J into high-order and low-order bits • We can then define I*J by multiplying the parts and adding: • So, T(n) = 4T(n/2) + n, which implies T(n) is O(n2). • But that is no better than the algorithm we learned in grade school. CSC401: Analysis of Algorithms

15. An Improved Integer Multiplication Algorithm • Algorithm: Multiply two n-bit integers I and J. • Divide step: Split I and J into high-order and low-order bits • Observe that there is a different way to multiply parts: • So, T(n) = 3T(n/2) + n, which implies T(n) is O(nlog23), by the Master Theorem. • Thus, T(n) is O(n1.585). CSC401: Analysis of Algorithms

16. Matrix Multiplication Algorithm • Problem: Given n×n matrices X and Y, find their product Z=X×Y, • General algorithm: O(n3) • Strassen’s Algorithm: • Rewrite the matrix product as: • Where I=AE+BG, J=AF+BH K=CE+DG, L=CF+DH • Define 7 submatrix products: • S1=A(F-H), S2=(A+B)H, S3=(C+D)E, S4=D(G-E) • S5=(A+D)(E+H), S6=(B-D)(G+H), S7=(A-C)(E+F) • Construct: • I=S4+S5+S6-S2, J=S1+S2, K=S3+S4, L=S1-S7-S3-S5 • Running time: T(n) = 7T(n/2)+bn2 for some constant b, which gives: O(nlog7), better than O(n3) CSC401: Analysis of Algorithms

17. Decrease and Conquer • Reduce problem instance to smaller instance of the same problem and extend solution • Solve smaller instance • Extend solution of smaller instance to obtain solution to original problem • Also referred to as inductive or incremental approach CSC401: Analysis of Algorithms

18. Examples of Decrease and Conquer • Decrease by one: • Insertion sort • Graph search algorithms: • DFS • BFS • Topological sorting • Algorithms for generating permutations, subsets • Decrease by a constant factor • Binary search • Fake-coin problems • multiplication à la russe • Josephus problem • Variable-size decrease • Euclid’s algorithm • Selection by partition CSC401: Analysis of Algorithms

19. Decrease by One • Generating all subsets • Given n elements in the set • Take one element from the set • Generate all subsets of remaining elements • Insert the element taken to the subsets generated • Base case: empty set only has a single subset – itself • Time efficiency? • Generating permutations • Given n symbols • Take one symbol from the set • Generate all permutations of remaining symbols • Insert the symbol taken to the permutations generated • Base case: a single symbol has only one permutation -- itself • Time efficiency? CSC401: Analysis of Algorithms

20. Variable-size-decrease: Binary search trees • Arrange keys in a binary tree with the binary search tree property: k Example 1: 5, 10, 3, 1, 7, 12, 9 Example 2: 4, 5, 7, 2, 1, 3, 6 <k >k • What about repeated keys? CSC401: Analysis of Algorithms

21. Searching and insertion in binary search trees • Searching – straightforward • Insertion – search for key, insert at leaf where search terminated • All operations: worst case # key comparisons = h + 1 • log n≤ h ≤ n – 1 with average (random files) 1.41log n • Thus all operations have: • worst case: Θ(n) • average case: Θ(lgn) • Bonus: inorder traversal produces sorted list (treesort) CSC401: Analysis of Algorithms

22. Transform and Conquer Solve problem by transforming into: • a more convenient instance of the same problem (instance simplification) • presorting • Gaussian elimination • a different representation of the same instance (representation change) • balanced search trees • heaps and heapsort • polynomial evaluation by Horner’s rule • Fast Fourier Transform • a different problem altogether (problem reduction) • reductions to graph problems • linear programming CSC401: Analysis of Algorithms

23. Instance simplification - Presorting Solve instance of problem by transforming into another simpler/easier instance of the same problem Presorting: Many problems involving lists are easier when list is sorted. • searching • computing the median (selection problem) • computing the mode • finding repeated elements CSC401: Analysis of Algorithms

24. Selection Problem Find the kth smallest element in A[1],…,A[n]. Special cases: • minimum: k = 1 • maximum: k = n • median: k = n/2 • Presorting-based algorithm • sort list • return A[k] • Partition-based algorithm (Variable decrease & conquer): • pivot/split at A[s] using partitioning algorithm from quicksort • if s=k return A[s] • else if s<k repeat with sublist A[s+1],…A[n]. • else if s>k repeat with sublist A[1],…A[s-1]. CSC401: Analysis of Algorithms

25. Notes on Selection Problem • Presorting-based algorithm: Ω(nlgn) + Θ(1) = Ω(nlgn) • Partition-based algorithm (Variable decrease & conquer): • worst case: T(n) =T(n-1) + (n+1) -> Θ(n2) • best case: Θ(n) • average case: T(n) =T(n/2) + (n+1) -> Θ(n) • Bonus: also identifies the k smallest elements (not just the kth) • Special cases max, min: better, simpler linear algorithm (brute force) • Conclusion: Presorting does not help in this case. CSC401: Analysis of Algorithms

26. Finding repeated elements • Presorting-based algorithm: • use mergesort (optimal): Θ(nlgn) • scan array to find repeated adjacent elements: Θ(n) • Brute force algorithm: Θ(n2) • Conclusion: Presorting yields significant improvement • Similar improvement for mode • What about searching? Θ(nlgn) CSC401: Analysis of Algorithms

27. Taxonomy of Searching Algorithms • Elementary searching algorithms • sequential search • binary search • binary tree search • Balanced tree searching • AVL trees • red-black trees • multiway balanced trees (2-3 trees, 2-3-4 trees, B trees) • Hashing • separate chaining • open addressing CSC401: Analysis of Algorithms