Instructor: Shengyu Zhang

1. CSC3160: Design and Analysis of Algorithms Week 7: Divide and Conquer Instructor: ShengyuZhang

2. Example 1: Merge sort

3. Starting example • Sorting: • We have a list of numbers . • We want to sort them in the increasing order.

4. An algorithm: merge sort • Merge sort: • Cut it into two halves with equal size. • Suppose divides for simplicity. • Suppose the two halves are sorted: Merge them. • Use two pointers, one for each half, to scan them, during which course do the appropriate merge. • How to sort each of the two halves? Recursively.

5. Complexity? • Suppose this algorithm takes time for an input with numbers. • Thus each of the two halves takes time. • The merging? • Scanning elements, an time operation needed for each. • Total amount of time: .

6. How to solve/bound this recurrence relation?

7. A general method for designing algorithm: Divide and conquer • Breaking the problem into subproblems • that are themselves smaller instances of the same type of problem • Recursively solving these subproblems • Appropriately combining their answers

8. Complexity • Running time on an input of size : • Break problem into subproblems, each of the same size . • In general, is not necessarily equal to . • Time to recursively solve each subproblem: • Time for breaking problem (into subproblems) and combining the answers:

9. Master theorem • , , and are all constants. • Then • Proof in textbook. Not required. • But you need to know how to apply it.

10. Merge sort: . • . So . • By the master theorem: .

11. Example 2: Selection

12. Selection • Problem: Given a list of numbers, find the -thsmallest. • We can sort the list, which needs . • Can we do better, say, linear time? • After all, sorting gives a lot more information than we requested. • Not always a waste: consider dynamic programming where solutions to subproblems are also produced.

13. Idea of divide and conquer • Divide the numbers into 3 parts , , • Depending on the size of each part, we know which part the -th element lies in. • Then search in that part. • Question: Which to choose?

14. Pivot • Suppose we use a number in the given list as a pivot. • As said, we divide the list into three parts. • : Those numbers smaller than • : Those numbers equal to • : Those numbers larger than

15. After the partition • The division is simple: just scan the list and put elements into the corresponding part. • time. • To select the -thsmallest value, it becomes • Complexity?

16. Divide and conquer • Note: though there are two subproblems (of sizes and ), we need to solve only oneof them. • Compare: in quicksort, we need to sort both substrings! • Complexity: • A new issue: and are not determined • Depends on the pivot.

17. If the pivot is the median: • Thus finally , better than by sorting.

18. If the pivot is at one end (say, the smallest) • What’s the complexity? • Complexity:

19. The similarity to quicksort tells us: a random pivot performs well • It’s away from either end by with const. prob. • To be more precise, it’s in with probability . • And in this case, the recursion becomes

20. Recall: • So

21. Thus we can use the following simple strategy: • Pick a random pivot, • do the recursion • Each random pivot falls in w/ prob. . • . • It is enough to getgood pivots to make the problem size to drop to 1. • Thus .

22. Example 3: Matrix multiplication

23. Matrix multiplication • Recall: the product of two matrices is another matrix. • Question: how fast can we multiply two matrices? • Recall: • : the entry in the matrix . Similar for ,

24. This takes multiplications (of numbers). • For a long time, people thought this was the best possible. • Until Straussen came up with the following.

25. If we break the matrix into blocks • Then the product is just block multiplication • 8 matrix multiplications of dimension • Plus additions.

26. Thus the recurrence is • This gives exactly the same , not interesting.

27. However, Straussen observed that we can actually use only 7 (instead of 8) multiplications of matrices with dimension .

28. God knows how he came up with it. • And here is how: • where

29. Thus the recurrence becomes • And solving this gives

30. Best in theory? . • Conjecture: ! • In practice? People still use the algorithm most of the time. (For example, in matlab.) • It’s simple and robust.

31. Fast Fourier Transform (FFT)

32. Multiplication of polynomials • Many applications need to multiply two polynomials. • e.g. signal processing. • A degree- polynomial is • The degree is at most . (The degree is exactly if .) • The summation of two polynomials is easy ---------------------------------------------------------------- • which still has degree at most .

33. Multiplication • Multiplication of two polynomials: a different story. ) --------------------------------------------------------------- • where (convolution) • Try an example: • The multiplication can have degree . • If we directly use this formula to multiply two polynomials, it takes time.

34. Better? YES! • By FFT: We can do it in time . • FFT: Fast Fourier Transform

35. What determines/represents a polynomial? • Let’s switch to degree-() for later simplicity. • We already used coefficients to represent a polynomial. • Coefficient representation • Other ways? • Yes: By giving values on (distinct) points. • Point-value representation. • [Fact] A degree-() polynomial is uniquely determined by values on distinct points.

36. Reason • We have coefficients to determine • We know values on distinct points. • How to get the coefficients? Just solve the system of equations:

37. unknowns • In matrix form: • The matrix is Vandermondematrix, which is invertible. Thus it has a unique solution for the coefficients .

38. Advantage of point-value representation? • It makes the multiplication extremely easy: • Though is hard • For any fixed point , is simply multiplication of two numbers. • So given and , it’s really easy to get . • time. • Thus we have the following interesting idea for polynomial multiplication…

39. Go to an easy world and come back • HK used to have many industries. • Later found mainland has less expensive labor. So: • moved the companies to mainland, • did the work there, • and then shipped the products back to HK to sell • This is worth doing if: it’s cheaper in mainland, and the traveling/shipping is not expensive either. • which turned out to be true.

40. In our case • We need to investigate: the cost of traveling. • Both way.

41. Traveling • From coefficient representation to point-value representation: • Evaluate two polynomials both on points. --- evaluation. • From point-value representation to coefficient representation: • Compute one polynomial back from point values. --- interpolation. • Both can be done in time.

42. Evaluation • One point? • Directly by the above: multiplications • Horner’s rule: --- multiplications.

43. Number of points • How many points we need to evaluate on? • Since we later want to reconstruct the product polynomial , which has degree . • So (point,value) pairs are enough to recover . • We’ll evaluate and on points , and get . • points are enough. We use for convenience. • Then recover from .

44. Now evaluation on one point needs , so evaluations on points need . • Too bad. We want . • Important fact: cost(evaluations on points)can be cheaper than cost(evaluation on point). • If we choose the points carefully. • And it turns out that the chosen points also make the (later) interpolation easier. • This powerful tool is called Fourier Transform.

45. Complex roots of unity • 1 has a complex root , satisfing . • Discrete Fourier Transform (DFT):where • Try now. • Note: exactly evaluates polynomial on . • So our evaluation task is just DFT.

46. All the essences are here… • We want to evaluate and on , , …, . • Suppose for simplicity that is a power of . • For , define two new polynomials: • Then • Divide and Conquer. • So we can evaluate by evaluating and on .

47. Distinct points • Note: are not all distinct. • Only values, each repeating twice!

48. Thus we only evaluate points. • Recursion: • ? To compute for . • Rewriting recursion: • . • Applying master theorem: . • Since , the cost of evaluating is , as claimed.

49. Interpolation • How about interpolation? • i.e., to get the coefficients by point values. • Almost the same process! • .

50. DFT matrix: . • What’s the inverse of the matrix ? • Pretty much the same matrix replace with . • …and then divide the whole matrix by for normalization.