1 / 57

Algorithm Design and Analysis

L ECTURES 10-11 Divide and Conquer Recurrences Nearest pair. Algorithm Design and Analysis. CSE 565. Adam Smith. Divide and Conquer. Break up problem into several parts. Solve each part recursively. Combine solutions to sub-problems into overall solution. Most common usage.

sasson
Download Presentation

Algorithm Design and Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. LECTURES 10-11 • Divide and Conquer • Recurrences • Nearest pair Algorithm Design and Analysis CSE 565 Adam Smith A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne

  2. Divide and Conquer • Break up problem into several parts. • Solve each part recursively. • Combine solutions to sub-problems into overall solution. • Most common usage. • Break up problem of size n into two equal parts of size n/2. • Solve two parts recursively. • Combine two solutions into overall solution in linear time. • Consequence. • Brute force: (n2). • Divide-and-conquer:  (n log n). Divide et impera. Veni, vidi, vici. - Julius Caesar S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  3. A L G O R I T H M S A L G O R I T H M S A G L O R H I M S T A G H I L M O R S T Mergesort • Divide array into two halves. • Recursively sort each half. • Merge two halves to make sorted whole. Jon von Neumann (1945) divide O(1) sort 2T(n/2) merge O(n) S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  4. Merging • Combine two pre-sorted lists into a sorted whole. • How to merge efficiently? • Linear number of comparisons. • Use temporary array. • Challenge for the bored: in-place merge [Kronrud, 1969] A G L O R H I M S T A G H I using only a constant amount of extra storage S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  5. T(n) = Q(1) ifn = 1; 2T(n/2) + Q(n) ifn > 1. Recurrence for Mergesort • T(n) = worst case running time of Mergesort on an input of size n. • Should be T( n/2 ) + T( n/2) , but it turns out not to matter asymptotically. • Usually omit the base case because our algorithms always run in time (1) when n is a small constant. • Several methods to find an upper bound on T(n). S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  6. Recursion Tree Method • Technique for guessing solutions to recurrences • Write out tree of recursive calls • Each node gets assigned the work done during that call to the procedure (dividing and combining) • Total work is sum of work at all nodes • After guessing the answer, can prove by induction that it works. S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  7. Recursion Tree for Mergesort Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. T(n) T(n/2) T(n/2) h = lg n T(n/4) T(n/4) T(n/4) T(n/4) … T(n / 2k) T(1) #leaves = n S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  8. Recursion Tree for Mergesort Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn T(n/2) T(n/2) h = lg n T(n/4) T(n/4) T(n/4) T(n/4) … T(n / 2k) T(1) #leaves = n S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  9. Recursion Tree for Mergesort Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/2 cn/2 h = lg n T(n/4) T(n/4) T(n/4) T(n/4) … T(n / 2k) T(1) #leaves = n S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  10. Recursion Tree for Mergesort Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/2 cn/2 h = lg n cn/4 cn/4 cn/4 cn/4 … T(n / 2k) T(1) #leaves = n S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  11. Recursion Tree for Mergesort Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn cn/2 cn cn/2 h = lg n cn/4 cn/4 cn cn/4 cn/4 … T(n / 2k) … Q(1) #leaves = n Q(n) Total = Q(n lg n) S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  12. Counting Inversions • Music site tries to match your song preferences with others. • You rank n songs. • Music site consults database to find people with similar tastes. • Similarity metric: number of inversions between two rankings. • My rank: 1, 2, …, n. • Your rank: a1, a2, …, an. • Songs i and j inverted if i < j, but ai > aj. • Brute force: check all (n2) pairs i and j. Songs A B C D E Inversions 3-2, 4-2 Me 1 2 3 4 5 You 1 3 4 2 5 S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  13. Counting Inversions: Algorithm • Divide-and-conquer 1 5 4 8 10 2 6 9 12 11 3 7 S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  14. Counting Inversions: Algorithm • Divide-and-conquer • Divide: separate list into two pieces. Divide: (1). 1 5 4 8 10 2 6 9 12 11 3 7 1 5 4 8 10 2 6 9 12 11 3 7 S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  15. Counting Inversions: Algorithm • Divide-and-conquer • Divide: separate list into two pieces. • Conquer: recursively count inversions in each half. Divide: (1). 1 5 4 8 10 2 6 9 12 11 3 7 1 5 4 8 10 2 6 9 12 11 3 7 Conquer: 2T(n / 2) 5 blue-blue inversions 8 green-green inversions 5-4, 5-2, 4-2, 8-2, 10-2 6-3, 9-3, 9-7, 12-3, 12-7, 12-11, 11-3, 11-7 S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  16. Counting Inversions: Algorithm • Divide-and-conquer • Divide: separate list into two pieces. • Conquer: recursively count inversions in each half. • Combine: count inversions where ai and aj are in different halves, and return sum of three quantities. Divide: (1). 1 5 4 8 10 2 6 9 12 11 3 7 1 5 4 8 10 2 6 9 12 11 3 7 Conquer: 2T(n / 2) 5 blue-blue inversions 8 green-green inversions 9 blue-green inversions 5-3, 4-3, 8-6, 8-3, 8-7, 10-6, 10-9, 10-3, 10-7 Combine: ??? Total = 5 + 8 + 9 = 22. S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  17. Counting Inversions: Combine Combine: count blue-green inversions • Assume each half is sorted. • Count inversions where ai and aj are in different halves. • Merge two sorted halves into sorted whole. to maintain sorted invariant 3 7 10 14 18 19 2 11 16 17 23 25 6 3 2 2 0 0 Count: (n) 13 blue-green inversions: 6 + 3 + 2 + 2 + 0 + 0 2 3 7 10 11 14 16 17 18 19 23 25 Merge: (n) T(n) = 2T(n/2) + Q (n). Solution: T(n) = Q (n log n). S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  18. Implementation • Pre-condition. [Merge-and-Count] A and B are sorted. • Post-condition. [Sort-and-Count] L is sorted. Sort-and-Count(L) { if list L has one element return 0 and the list L Divide the list into two halves A and B (rA, A)  Sort-and-Count(A) (rB, B)  Sort-and-Count(B) (rB, L)  Merge-and-Count(A, B) return r = rA + rB + r and the sorted list L } S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  19. Find an element in a sorted array: • Divide: Check middle element. • Conquer: Recursively search 1 subarray. • Combine: Trivial. Example: Find 9 3 5 7 8 9 12 15 Binary search S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  20. Find an element in a sorted array: • Divide: Check middle element. • Conquer: Recursively search 1 subarray. • Combine: Trivial. 3 5 7 8 9 12 15 Binary search Example: Find 9 S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  21. Find an element in a sorted array: • Divide: Check middle element. • Conquer: Recursively search 1 subarray. • Combine: Trivial. 3 5 7 8 9 12 15 Binary search Example: Find 9 S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  22. Find an element in a sorted array: • Divide: Check middle element. • Conquer: Recursively search 1 subarray. • Combine: Trivial. 3 5 7 8 9 12 15 Binary search Example: Find 9 S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  23. Find an element in a sorted array: • Divide: Check middle element. • Conquer: Recursively search 1 subarray. • Combine: Trivial. 3 5 7 8 9 12 15 Binary search Example: Find 9 S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  24. Find an element in a sorted array: • Divide: Check middle element. • Conquer: Recursively search 1 subarray. • Combine: Trivial. 3 5 7 8 9 12 15 Binary search Example: Find 9 S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  25. Binary Search BINARYSEARCH(b, A[1 . . n]) B find b in sorted array A • Ifn=0 thenreturn “not found” • If A[dn/2e] = b then returndn/2e • If A[dn/2e] < b then • return BINARYSEARCH(A[1 . . dn/2e]) • Else • return dn/2e + BINARYSEARCH(A[dn/2e+1 . . n]) S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  26. Recurrence for binary search T(n) = 1T(n/2) + Q(1) # subproblems work dividing and combining subproblem size S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  27. Recurrence for binary search T(n) = 1T(n/2) + Q(1) # subproblems work dividing and combining subproblem size •  T(n) = T(n/2) + c = T(n/4) + 2c • = cdlog ne= Q(lgn) . S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  28. ab/2 ×ab/2 ifbis even; ab = a(b–1)/2 ×a(b–1)/2 ×aifbis odd. Divide-and-conquer algorithm: Exponentiation Problem: Compute ab, where b Î N is n bits long Question: How many multiplications? Naive algorithm:Q(b) = (2n) (exponential in the input length!) T(b) = T(b/2) + Q(1)  T(b) = Q(logb) = (n) . S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  29. So far: 2 recurrences • Mergesort; Counting InversionsT(n) = 2 T(n/2) + (n) = (n log n) • Binary Search; ExponentiationT(n) = 1 T(n/2) + (1) = (log n) Master Theorem: method for solving recurrences. S. Raskhodnikova; based on slides by E. Demaine, C. Leiserson, A. Smith, K. Wayne

  30. ab/2 ×ab/2 ifbis even; ab = a(b–1)/2 ×a(b–1)/2 ×aifbis odd. Review Question: Exponentiation Problem: Compute ab, where b Î N is n bits long. Question: How many multiplications? Naive algorithm:Q(b) = (2n) (exponential in the input length!) Naive algorithm: Divide-and-conquer algorithm: T(b) = T(b/2) + Q(1)  T(b) = Q(logb) = (n) . A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova.

  31. So far: 2 recurrences • Mergesort; Counting InversionsT(n) = 2 T(n/2) + (n) = (n log n) • Binary Search; ExponentiationT(n) = 1 T(n/2) + (1) = (log n) Master Theorem: method for solving recurrences. A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova.

  32. The master method The master method applies to recurrences of the form T(n) = aT(n/b) + f(n) , where a³ 1, b > 1, and f is asymptotically positive, that is f(n) >0 for all n> n0. First step: compare f(n) to nlogba. A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova.

  33. Three common cases Compare f(n) with nlogba: • f(n) = O(nlogba – e) for some constant e > 0. • f(n) grows polynomially slower than nlogba (by an ne factor). • Solution: T(n) = Q(nlogba) . A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova.

  34. Three common cases Compare f(n) with nlogba: • f(n) = O(nlogba – e) for some constant e > 0. • f(n) grows polynomially slower than nlogba (by an ne factor). • Solution: T(n) = Q(nlogba) . • f(n) = Q(nlogba lgkn) for some constant k³ 0. • f(n) and nlogba grow at similar rates. • Solution: T(n) = Q(nlogba lgk+1n) . A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova.

  35. Three common cases (cont.) Compare f(n) with nlogba: • f(n) = W(nlogba+ e) for some constant e > 0. • f(n) grows polynomially faster than nlogba(by an ne factor), • andf(n) satisfies the regularity conditionthat af(n/b) £cf(n) for some constant c< 1. • Solution: T(n) = Q(f(n)) . A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova.

  36. Idea of master theorem Recursion tree: f(n) a … f(n/b) f(n/b) f(n/b) a … f(n/b2) f(n/b2) f(n/b2) … T(1) A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova.

  37. f(n) af(n/b) a2f(n/b2) … Idea of master theorem Recursion tree: f(n) a … f(n/b) f(n/b) f(n/b) a … f(n/b2) f(n/b2) f(n/b2) … T(1) A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova.

  38. f(n) af(n/b) h = logbn a2f(n/b2) Idea of master theorem Recursion tree: f(n) a … f(n/b) f(n/b) f(n/b) a … f(n/b2) f(n/b2) f(n/b2) … … T(1) A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova.

  39. h = logbn nlogbaT(1) Idea of master theorem Recursion tree: f(n) f(n) a … af(n/b) f(n/b) f(n/b) f(n/b) a … a2f(n/b2) f(n/b2) f(n/b2) f(n/b2) … #leaves = ah = alogbn = nlogba … T(1) A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova.

  40. Idea of master theorem Recursion tree: f(n) f(n) a … af(n/b) f(n/b) f(n/b) f(n/b) a h = logbn … a2f(n/b2) f(n/b2) f(n/b2) f(n/b2) … CASE 1: The weight increases geometrically from the root to the leaves. The leaves hold a constant fraction of the total weight. … T(1) nlogbaT(1) Q(nlogba) A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova.

  41. Examples EX.T(n) = 4T(n/2) + n3 a = 4, b = 2 nlogba =n2; f(n) = n3. CASE 3: f(n) = W(n2+ e) for e = 1 and4(n/2)3£cn3 (reg. cond.) for c = 1/2.  T(n) = Q(n3). A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova.

  42. Examples EX.T(n) = 4T(n/2) + n3 a = 4, b = 2 nlogba =n2; f(n) = n3. CASE 3: f(n) = W(n2+ e) for e = 1 and4(n/2)3£cn3 (reg. cond.) for c = 1/2.  T(n) = Q(n3). EX.T(n) = 4T(n/2) + n2/lgn a = 4, b = 2 nlogba =n2; f(n) = n2/lgn. Master method does not apply. In particular, for every constant e > 0, we have ne= w(lgn). A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova.

  43. Notes • Master Th’m generalized by Akra and Bazzi to cover many more recurrences:where • See http://www.dna.lth.se/home/Rolf_Karlsson/akrabazzi.ps A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova.

  44. Multiplying large integers • Given n-bit integers a, b (in binary), compute c=ab • Naïve(grade-school) algorithm: • Write a,b in binary • Compute n intermediate products • Do nadditions • Total work: (n2) an-1 an-2 … a0 £bn-1 bn-2 … b0 n bits 0 n bits … 0 0  0 n bits 2n bit output A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova.

  45. Multiplying large integers • Divide and Conquer (Attempt #1): • Write a = A1 2n/2 + A0b = B1 2n/2 + B0 • We want ab= A1B12n + (A1B0 + B1A0) 2n/2 +A0B0 • Multiply n/2 –bit integers recursively • T(n) = 4T(n/2) + (n) • Alas! this is still (n2)(Master Theorem, Case 1) A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova.

  46. Multiplying large integers • Divide and Conquer (Attempt #1): • Write a = A1 2n/2 + A0b = B1 2n/2 + B0 • We want ab= A1B12n + (A1B0 + B1A0) 2n/2 +A0B0 • Multiply n/2 –bit integers recursively • T(n) = 4T(n/2) + (n) • Alas! this is still (n2) . (Exercise: write out the recursion tree.) • Karatsuba’s idea:(A0+A1) (B0 + B1) = A0B0+ A1B1 + (A0B1 + B1A0) • We can get away with 3 multiplications! (in yellow)x= A1B1y = A0B0z = (A0+A1)(B0+B1) • Now we use ab = A1B12n + (A1B0 + B1A0) 2n/2 + A0B0 = x2n + (z–x–y)2n/2 + y A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova.

  47. Multiplying large integers MULTIPLY (n, a, b) Baand b are n-bit integers B Assume nis a power of 2 for simplicity • Ifn · 2then use grade-school algorithm else • A1Ãa div 2n/2; B1Ãb div 2n/2 ; • A0Ãa mod 2n/2 ; B0Ãb mod 2n/2 . • xà MULTIPLY(n/2 , A1, B1) • yà MULTIPLY(n/2 , A0, B0) • zà MULTIPLY(n/2 , A1+A0 , B1+B0) • Outputx2n + (z–x–y)2n/2 + y A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova.

  48. Multiplying large integers • The resulting recurrenceT(n) = 3T(n/2) + (n) • Master Theorem, Case 1: T(n) =  (nlog23) = (n1.59…) • Note: There is a (n log n) algorithm for multiplication (more on it later in the course). A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova.

  49. Matrix multiplication A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne

  50. Input:A = [aij], B = [bij]. Output:C = [cij] = A×B. i,j = 1, 2,… , n. Matrix multiplication A. Smith; based on slides by E. Demaine, C. Leiserson, S. Raskhodnikova, K. Wayne

More Related