6.006- Introduction to Algorithms

1. 6.006-Introduction to Algorithms Lecture 8 Prof. Silvio Micali CLRS: chapter 4.

2. Menu • Sorting! • Insertion Sort • Merge Sort • Recurrences • Master theorem

3. The problem of sorting Input: array A[1…n] of numbers. Output: permutation B[1…n] of Asuch that B[1] £ B[2] £ …£ B[n]. e.g. A = [7, 2, 5, 5, 9.6] → B = [2, 5, 5, 7, 9.6] How can we do it efficiently ?

4. Insertion sort INSERTION-SORT (A, n) ⊳ A[1 . . n] forj← 2ton insert key A[j]into the (already sorted) sub-array A[1 .. j-1] by pairwise key-swaps down to its right position Illustration of iteration j 1 i j n A: new location of key key sorted

5. Example of insertion sort 8 2 4 9 3 6

6. 8 2 4 9 3 6 Example of insertion sort 1 swap

7. 8 2 4 9 3 6 Example of insertion sort 2 8 4 9 3 6

8. 8 2 4 9 3 6 2 8 4 9 3 6 Example of insertion sort 1 swap

9. 8 2 4 9 3 6 2 8 4 9 3 6 Example of insertion sort 2 4 8 9 3 6

10. 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6 Example of insertion sort 0 swap

11. 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6 Example of insertion sort 2 4 8 9 3 6

12. 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6 Example of insertion sort 2 4 8 9 3 6 3 swaps

13. 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6 Example of insertion sort 2 4 8 9 3 6 2 3 4 8 9 6

14. 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6 Example of insertion sort 2 4 8 9 3 6 2 3 4 8 9 6 2 swaps

15. 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6 Example of insertion sort 2 4 8 9 3 6 2 3 4 8 9 6 2 3 4 6 8 9 done Running time? O(n2) e.g. when input is A = [n, n − 1, n − 2, . . . , 2, 1]

16. Meet Merge Sort MERGE-SORTA[1 . . n] • If n = 1, done (nothing to sort). divide and conquer Otherwise, recursively sort A[ 1 . . n/2 ] and A[ n/2+1 . . n]. “Merge” the two sorted sub-arrays. Key subroutine:MERGE

17. Merging two sorted arrays 20 13 7 2 12 11 9 1 … output array

18. Merging two sorted arrays 20 13 7 2 12 11 9 1 … 1 output array

19. Merging two sorted arrays 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 … 1 output array

20. Merging two sorted arrays 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 … 1 2 output array

21. Merging two sorted arrays 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 … 1 2 output array

22. Merging two sorted arrays 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 … 1 2 7 output array

23. Merging two sorted arrays 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 20 13 12 11 9 … 1 2 7 output array

24. Merging two sorted arrays 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 20 13 12 11 9 … 1 2 7 9 output array

25. Merging two sorted arrays 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 20 13 12 11 9 20 13 12 11 … 1 2 7 9 output array

26. Merging two sorted arrays 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 20 13 12 11 9 20 13 12 11 … 1 2 7 9 11 output array

27. Merging two sorted arrays 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 20 13 12 11 9 20 13 12 11 20 13 12 … 1 2 7 9 11 output array

28. 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 20 13 12 11 9 20 13 12 11 20 13 12 1 2 7 9 11 12 Merging two sorted arrays … output array

29. 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 20 13 12 11 9 20 13 12 11 20 13 12 1 2 7 9 11 12 Merging two sorted arrays … output array Time = Q(n) to merge a total of nelements (linear time).

30. T(n) = Q(1) ifn = 1; 2T(n/2) + Q(n) ifn > 1. Analyzing merge sort MERGE-SORTA[1 . . n] T(n) Q(1) 2T(n/2) Q(n) • If n = 1, done • 2. Recursively sort • A[ 1 . . n/2 ] and A[ n/2+1 . . n ] • 3. “Merge” the two sorted lists

31. Recurrence solving Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.

32. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. T(n)

33. cn T(n/2) T(n/2) Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.

34. cn cn/2 cn/2 T(n/4) T(n/4) T(n/4) T(n/4) Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.

35. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/2 cn/2 cn/4 cn/4 cn/4 cn/4 … … Q(1)

36. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/2 cn/2 cn/4 cn/4 cn/4 cn/4 … … Q(1)

37. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn cn/2 cn/2 cn/4 cn/4 cn/4 cn/4 … … Q(1)

38. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn cn/2 cn cn/2 cn/4 cn/4 cn/4 cn/4 … … Q(1)

39. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn cn/2 cn cn/2 cn/4 cn/4 cn cn/4 cn/4 … … … Q(1)

40. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn cn/2 cn cn/2 h = lgn cn/4 cn/4 cn cn/4 cn/4 … … … Q(1) Q(n) #leaves = n

41. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn cn/2 cn cn/2 h = lg n cn/4 cn/4 cn cn/4 cn/4 … … … Q(1) Q(n) Total = ? #leaves = n

42. Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn cn/2 cn cn/2 h = lgn cn/4 cn/4 cn cn/4 cn/4 … … Q(1) Q(n) Total=Q(nlgn) #leaves = n

43. The master method “One theorem for all recurrences” (sort of) It applies to recurrences of the form T(n) = aT(n/b) + f(n) , where a³ 1, b > 1, and fis positive. e.g. Mergesort: a=2, b=2, f(n)=O(n) e.g.2 Binary Search: a=1, b=2, f(n)=O(1) Basic Idea: Compare f(n) with nlogba. size of eachsubproblem #subproblems time to split into subproblems and combine results

44. h = logbn Idea of master theorem T(n) = aT(n/b) + f(n) Recursion tree: f(n) f(n) a … f(n/b) f(n/b) f(n/b) af(n/b) a … f(n/b2) f(n/b2) a2f(n/b2) f(n/b2) … … T(1) nlogbaT(1) #leaves = ah = alogbn = nlogba ?

45. Idea of master theorem T(n) = aT(n/b) + f(n) Recursion tree: f(n) f(n) a … af(n/b) f(n/b) f(n/b) f(n/b) a h = logbn … a2f(n/b2) f(n/b2) f(n/b2) f(n/b2) … … CASE 1: The weight increases geometrically from root to leaves. T(1) nlogbaT(1) Leaves hold a constant fraction of total weight! Q(nlogba) ?

46. Idea of master theorem T(n) = aT(n/b) + f(n) Recursion tree: f(n) f(n) a … f(n/b) f(n/b) f(n/b) af(n/b) a h = logbn … a2f(n/b2) f(n/b2) f(n/b2) f(n/b2) … … CASE 2: Weight approximately the same on each level. T(1) nlogbaT(1) Total weight #levels leaves’ weight! ?

47. Idea of master theorem T(n) = aT(n/b) + f(n) Recursion tree: f(n) f(n) a … f(n/b) f(n/b) f(n/b) af(n/b) a h = logbn … a2f(n/b2) f(n/b2) f(n/b2) f(n/b2) … … CASE 3: Weight decreases geometrically from root to leaves. T(1) nlogbaT(1) Root holds a constant fraction of total weight! Q(f(n)) ?

48. Three common cases Compare f(n) with nlogba: • f(n) = Θ(nlogba– e) for some constant e > 0. • I.e., f(n) grows polynomially slower than nlogba (by an ne factor). (be)i cost of level i = aif(n/bi) = Q(nlogba-e bi e ) so geometric increase of cost as we go deeper in the tree hence, leaf level cost dominates! Solution: T(n) = Q(nlogba) .

49. Three common cases (cont.) Compare f(n) with nlogba: • f(n) = Q(nlogba) for some constant k³0. • I.e., f(n) and nlogba grow at similar rates. (cost of level i) = aif(n/bi) = Q(nlogbak(n/bi)) so all levels have about the same cost Solution:

50. Three common cases (cont.) Compare f(n) with nlogba: • f(n) = Θ(nlogba+ e) for some constant e > 0. • I.e., f(n)grows polynomially faster than nlogba (by an nefactor). (cost of level i) = aif(n/bi) = Q(nlogba+e b-ie ) so geometric decrease of cost as we go deeper in the tree hence, root cost dominates! Solution: T(n) = Q(f(n)) .