Greedy Algorithms CSc 4520/6520

1. Greedy AlgorithmsCSc 4520/6520 Fall 2013

2. Problems Considered • Activity Selection Problem • Knapsack Problem • 0 – 1 Knapsack • Fractional Knapsack • Huffman Codes

3. Greedy Algorithms 2 techniques for solving optimization problems: 1. Dynamic Programming 2. Greedy Algorithms (“Greedy Strategy”) For the optimization problems: Greedy Approach can solve these problems: Dynamic Programming can solve these problems: • For some optimization problems, • Dynamic Programming is “overkill” • Greedy Strategy is simpler and more efficient.

4. Activity-Selection Problem For a set of proposed activities that wish to use a lecture hall, select a maximum-size subset of “compatible activities”. • Set of activities: S={a1,a2,…an} • Duration of activity ai: [start_timei, finish_timei) • Activities sorted in increasing order of finish time: i 1 2 3 4 5 6 7 8 9 10 11 start_timei 1 3 0 5 3 5 6 8 8 2 12 finish_timei 4 5 6 7 8 9 10 11 12 13 14

5. time a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Activity-Selection Problem i 1 2 3 4 5 6 7 8 9 10 11 start_timei 1 3 0 5 3 5 6 8 8 2 12 finish_timei 4 5 6 7 8 9 10 11 12 13 14 Compatible activities: {a3, a9, a11}, {a1,a4,a8,a11}, {a2,a4,a9,a11}

6. time time time a1 a1 a1 a2 a2 a2 a3 a3 a3 a4 a4 a4 a5 a5 a5 a6 a6 a6 a7 a7 a7 a8 a8 a8 a9 a9 a9 a10 a10 a10 a11 a11 a11 time a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4 5 ok ok 5 5 5 6 ok ok ok 7 ok ok 6 6 6 8 ok ok ok ok 7 7 7 9 ok ok ok 10 ok ok 8 8 8 11 ok 9 9 9 12 13 10 10 10 14 11 11 11 12 12 12 13 13 13 14 14 14 Activity-Selection ProblemDynamic Programming Solution (Step 1) Step 1. Characterize the structure of an optimal solution. S: i 1 2 3 4 5 6 7 8 9 10 11(=n) start_timei 1 3 0 5 3 5 6 8 8 2 12 finish_timei 4 5 6 7 8 9 10 11 12 13 14 Let Si,j be the set of activities that start after ai finishes and finish before aj starts. eg. S2,11= eg Definition: Sij={akS: finish_timeistart_timek<finish_timek start_timej}

7. time a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 a12 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14  Activity-Selection ProblemDynamic Programming Solution (Step 1) Add fictitious activities: a0 and an+1: S: i 0 1 2 3 4 5 6 7 8 9 10 11 12 start_timei 1 3 0 5 3 5 6 8 8 2 12  finish_timei0 4 5 6 7 8 9 10 11 12 13 14 S: i 1 2 3 4 5 6 7 8 9 10 11(=n) start_timei 1 3 0 5 3 5 6 8 8 2 12 finish_timei 4 5 6 7 8 9 10 11 12 13 14 ie. S0,n+1 ={a1,a2,a3,a4,a5,a6,a7,a8,a9,a10,a11} = S Note: If i>=j then Si,j=Ø

8. time a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 0 1 2 time a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 3 0 1 4 2 5 3 6 4 7 5 6 8 7 9 8 10 9 11 10 11 12 12 13 13 14 14 Activity-Selection ProblemDynamic Programming Solution (Step 1) The problem: For a set of proposed activities that wish to use a lecture hall, select a maximum-size subset of “compatible activities Select a maximum-size subset of compatible activities from S0,n+1. = Substructure: Suppose a solution to Si,j includes activity ak, then,2 subproblems are generated: Si,k, Sk,j The maximum-size subset Ai,j of compatible activities is: Ai,j=Ai,k U {ak} U Ak,j Suppose a solution to S0,n+1 contains a7, then, 2 subproblems are generated: S0,7 and S7,n+1

9. 0 if Si,j=Ø c(i,j) = Maxi<k<j {c[i,k] + c[k,j] + 1} if Si,jØ Activity-Selection ProblemDynamic Programming Solution (Step 2) Step 2. Recursively define an optimal solution Let c[i,j] = number of activities in a maximum-size subset of compatible activities in Si,j. If i>=j, then Si,j=Ø, ie. c[i,j]=0. Step 3. Compute the value of an optimal solution in a bottom-up fashion Step 4.Construct an optimal solution from computed information.

10. 0 if Si,j=Ø c(i,j) = Maxi<k<j {c[i,k]+c[k,j]+1} if Si,jØ time a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 0 1 2 3 4 5 ok ok 6 ok ok ok 7 ok ok 8 ok ok ok ok 9 ok ok ok 10 ok ok 11 ok 12 13 14 Activity-Selection ProblemGreedy Strategy Solution eg. S2,11={a4,a6,a7,a8,a9} Consider any nonempty subproblem Si,j, and let am be the activity in Si,j with the earliest finish time. Then, 1. Am is used in some maximum-size subset of compatible activities of Si,j. Among {a4,a6,a7,a8,a9}, a4 will finish earliest 1. A4 is used in the solution 2. After choosing A4, there are 2 subproblems: S2,4 and S4,11. But S2,4 is empty. Only S4,11 remains as a subproblem. 2. The subproblem Si,m is empty, so that choosing am leaves the subproblem Sm,j as the only one that may be nonempty.

11. time a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 a12 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14  Activity-Selection ProblemGreedy Strategy Solution Hence, to solve the Si,j: 1. Choose the activity am with the earliest finish time. 2. Solution of Si,j = {am} U Solution of subproblem Sm,j That is, To solve S0,12, we select a1 that will finish earliest, and solve for S1,12. To solve S1,12, we select a4 that will finish earliest, and solve for S4,12. To solve S4,12, we select a8 that will finish earliest, and solve for S8,12. … Greedy Choices (Locally optimal choice) To leave as much opportunity as possible for the remaining activities to be scheduled. Solve the problem in a top-down fashion

12. time a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 a12 0 1 2 3 4 5 6 {1,4,8,11} 7 8 {4,8,11} 9 10 {8,11} 11 12 {11} 13 14 Ø  Activity-Selection ProblemGreedy Strategy Solution Recursive-Activity-Selector(i,j) 1 m = i+1 // Find first activity in Si,j 2 while m < j and start_timem < finish_timei 3 do m = m + 1 4 if m < j 5 then return {am} U Recursive-Activity-Selector(m,j) 6 else return Ø m=3 Okay m=2 Okay m=4 break the loop Order of calls: Recursive-Activity-Selector(0,12) Recursive-Activity-Selector(1,12) Recursive-Activity-Selector(4,12) Recursive-Activity-Selector(8,12) Recursive-Activity-Selector(11,12)

13. time a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 a12 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14  Activity-Selection ProblemGreedy Strategy Solution Iterative-Activity-Selector() 1 Answer = {a1} 2 last_selected=1 3 for m = 2 to n 4 if start_timem>=finish_timelast_selected 5 then Answer = Answer U {am} 6 last_selected = m 7 return Answer

14. Activity-Selection ProblemGreedy Strategy Solution For both Recursive-Activity-Selector and Iterative-Activity-Selector, Running times are (n) Reason: each am are examined once.

15. Greedy Algorithm Design Steps of Greedy Algorithm Design: 1. Formulate the optimization problem in the form: we make a choice and we are left with one subproblem to solve. 2. Show that the greedy choice can lead to an optimal solution, so that the greedy choice is always safe. 3. Demonstrate that an optimal solution to original problem = greedy choice + an optimal solution to the subproblem Optimal Substructure Property Greedy-Choice Property A good clue that that a greedy strategy will solve the problem.

16. Greedy Algorithm Design Comparison: Dynamic Programming Greedy Algorithms • At each step, the choice is determined based on solutions of subproblems. • At each step, we quickly make a choice that currently looks best. --A local optimal (greedy) choice. • Sub-problems are solved first. • Greedy choice can be made first before solving further sub-problems. • Bottom-up approach • Top-down approach • Can be slower, more complex • Usually faster, simpler

17. Greedy Algorithms • Similar to dynamic programming, but simpler approach • Also used for optimization problems • Idea: When we have a choice to make, make the one that looks best right now • Make a locally optimal choice in hope of getting a globally optimal solution • Greedy algorithms don’t always yield an optimal solution • Makes the choice that looks best at the moment in order to get optimal solution.

18. Fractional Knapsack Problem • Knapsack capacity: W • There are n items: the i-th item has value vi and weight wi • Goal: • find xi such that for all 0  xi  1, i = 1, 2, .., n  wixi  W and  xivi is maximum

19. Fractional Knapsack - Example • E.g.: 20 --- 30 50 50 $80 + Item 3 30 20 Item 2 $100 + 20 Item 1 10 10 $60 $60 $100 $120 $240 $6/pound $5/pound $4/pound

20. Fractional Knapsack Problem • Greedy strategy 1: • Pick the item with the maximum value • E.g.: • W = 1 • w1 = 100, v1 = 2 • w2 = 1, v2 = 1 • Taking from the item with the maximum value: Total value taken = v1/w1 = 2/100 • Smaller than what the thief can take if choosing the other item Total value (choose item 2) = v2/w2 = 1

21. Fractional Knapsack Problem Greedy strategy 2: • Pick the item with the maximum value per pound vi/wi • If the supply of that element is exhausted and the thief can carry more: take as much as possible from the item with the next greatest value per pound • It is good to order items based on their value per pound

22. Fractional Knapsack Problem Alg.:Fractional-Knapsack (W, v[n], w[n]) • While w > 0 and as long as there are items remaining • pick item with maximum vi/wi • xi  min (1, w/wi) • remove item i from list • w  w – xiwi • w – the amount of space remaining in the knapsack (w = W) • Running time: (n) if items already ordered; else (nlgn)

23. Huffman Codes • Huffman Codes • For compressing data (sequence of characters) • Widely used • Very efficient (saving 20-90%) • Use a table to keep frequencies of occurrence of characters. • Output binary string. “Today’s weather is nice” “001 0110 0 0 100 1000 1110”

24. Huffman Code Problem • Huffman’s algorithm achieves data compression by finding the best variable length binary encoding scheme for the symbols that occur in the file to be compressed.

25. Huffman Code Problem • The more frequent a symbol occurs, the shorter should be the Huffman binary word representing it. • The Huffman code is a prefix-free code. • No prefix of a code word is equal to another codeword.

26. C: Alphabet Overview • Huffman codes: compressing data (savings of 20% to 90%) • Huffman’s greedy algorithm uses a table of the frequencies of occurrence of each character to build up an optimal way of representing each character as a binary string

27. eg. “abc” = “0101100” eg. “abc” = “000001010” Huffman Codes Example: Frequency Fixed-length Variable-length codewordcodeword ‘a’ 45000 000 0 ‘b’ 13000 001 101 ‘c’ 12000 010 100 ‘d’ 16000 011 111 ‘e’ 9000 100 1101 ‘f’ 5000 101 1100 A file of 100,000 characters. Containing only ‘a’ to ‘e’ 1*45000 + 3*13000 + 3*12000 + 3*16000 + 4*9000 + 4*5000 = 222424,000bits 1*45000 + 3*13000 + 3*12000 + 3*16000 + 4*9000 + 4*5000 = 224,000 bits 300,000bits 300,000 224,000

28. 0 0 1 1 0 0 1 1 0 0 1 1 1 1 0 0 1 1 0 0 0 0 a:45 a:45 b:13 b:13 c:12 c:12 d:16 d:16 e:9 e:9 f:5 f:5 100 1 0 100 0 0 0 1 1 1 55 a:45 1 86 86 0 14 14 0 0 0 1 1 1 0 0 0 30 25 28 28 28 14 14 14 1 58 58 58 1 0 0 1 1 1 1 1 1 0 0 0 1 1 1 0 0 0 0 0 0 d:16 b:13 14 1 0 a:45 a:45 a:45 b:13 b:13 b:13 c:12 c:12 c:12 d:16 d:16 d:16 e:9 e:9 e:9 f:5 f:5 f:5 f:5 e:9 Huffman Codes A file of 100,000 characters. The coding schemes can be represented by trees: Frequency Variable-length (in thousands) codeword ‘a’ 45 0 ‘b’ 13 101 ‘c’ 12 100 ‘d’ 16 111 ‘e’ 9 1101 ‘f’ 5 1100 Frequency Fixed-length (in thousands) codeword ‘a’ 45 000 ‘b’ 13 001 ‘c’ 12 010 ‘d’ 16 011 ‘e’ 9 100 ‘f’ 5 101 A full binary treeevery nonleaf node has 2 children Not a fullbinary tree c:12

29. 100 1 0 55 a:45 1 0 30 25 1 1 0 0 d:16 c:12 b:13 14 1 0 f::5 e:9 Huffman Codes • To find an optimal code for a file: • 1. The coding must be unambiguous. • Consider codes in which no codeword is also a prefix of other codeword. => Prefix Codes • Prefix Codes are unambiguous. • Once the codewords are decided, it is easy to compress (encode) and decompress (decode). • 2. File size must be smallest. • => Can be represented by a full binary tree. • => Usually less frequent characters are at bottom • Let C be the alphabet (eg. C={‘a’,’b’,’c’,’d’,’e’,’f’}) • For each character c, no. of bits to encode all c’s occurrences = freqc*depthc • File size B(T) = cCfreqc*depthc Frequency Codeword ‘a’ 45000 0 ‘b’ 13000 101 ‘c’ 12000 100 ‘d’ 16000 111 ‘e’ 9000 1101 ‘f’ 5000 1100 Eg. “abc” is coded as “0101100”

30. How do we find the optimal prefix code? Huffman code (1952) was invented to solve it. A Greedy Approach. Q: A min-priority queue f:5 e:9 c:12 b:13 d:16 a:45 c:12 b:13 d:16 a:45 14 100 a:45 a:45 25 30 55 f:5 e:9 55 a:45 25 30 c:12 b:13 d:16 14 14 d:16 a:45 30 25 25 c:12 b:13 d:16 14 f:5 e:9 d:16 c:12 b:13 14 f:5 e:9 c:12 b:13 f:5 e:9 f::5 e:9 Huffman Codes

31. Q: A min-priority queue f:5 e:9 c:12 b:13 d:16 a:45 c:12 b:13 d:16 a:45 14 f:5 e:9 14 d:16 a:45 25 f:5 e:9 c:12 b:13 Huffman Codes …. HUFFMAN(C) 1 Build Q from C 2 For i = 1 to |C|-1 3 Allocate a new node z 4 z.left = x = EXTRACT_MIN(Q) 5 z.right = y = EXTRACT_MIN(Q) 6 z.freq = x.freq + y.freq 7 Insert z into Q in correct position. 8 Return EXTRACT_MIN(Q) If Q is implemented as a binary min-heap, “Build Q from C” is O(n) “EXTRACT_MIN(Q)” is O(lg n) “Insert z into Q” is O(lg n) Huffman(C) is O(n lg n) How is it “greedy”?

32. Cost of a Tree T • For each character c in the alphabet C • let f(c) be the frequency of c in the file • let dT(c) be the depth of c in the tree • It is also the length of the codeword. Why? • Let B(T) be the number of bits required to encode the file (called the cost of T)

33. Huffman Code Problem In the pseudocode that follows: • we assume that C is a set of n characters and that each character c C is an object with a defined frequency f [c]. • The algorithm builds the tree T corresponding to the optimal code • A min-priority queue Q, is used to identify the two least-frequent objects to merge together. • The result of the merger of two objects is a new object whose frequency is the sum of the frequencies of the two objects that were merged.

34. Running time of Huffman's algorithm • The running time of Huffman's algorithm assumes that Q is implemented as a binary min-heap. • For a set C of n characters, the initialization of Q in line 2 can be performed in O (n) time using the BUILD-MINHEAP • The for loop in lines 3-8 is executed exactly n - 1 times, and since each heap operation requires time O (lg n), the loop contributes O (n lg n) to the running time. Thus, the total running time of HUFFMAN on a set of n characters is O (n lg n).

35. Depth of c (length of the codeword) Frequency of c Prefix Code • Prefix(-free) code: no codeword is also a prefix of some other codewords (Un-ambiguous) • An optimal data compression achievable by a character code can always be achieved with a prefix code • Simplify the encoding (compression) and decoding • Encoding: abc  0 . 101. 100 = 0101100 • Decoding: 001011101 = 0. 0. 101. 1101  aabe • Use binary tree to represent prefix codes for easy decoding • An optimal code is always represented by a full binary tree, in which every non-leaf node has two children • |C| leaves and |C|-1 internal nodes Cost:

36. Huffman Code • Reduce size of data by 20%-90% in general • If no characters occur more frequently than others, then no advantage over ASCII • Encoding: • Given the characters and their frequencies, perform the algorithm and generate a code. Write the characters using the code • Decoding: • Given the Huffman tree, figure out what each character is (possible because of prefix property)

37. How to Decode? • With fixed length code, easy: • break up into 3's, for instance • For variable length code, ensure that no character's code is the prefix of another • no ambiguity 101111110100 b d e a a

38. Huffman Algorithm correctness: Need to prove two things: Greedy Choice Property: There exists a minimum cost prefix tree where the two smallest frequency characters are indeed siblings with the longest path from root. This means that the greedy choice does not hurt finding the optimum.

39. Algorithm correctness: Optimal Substructure Property: An optimal solution to the problem once we choose the two least frequent elements and combine them to produce a smaller problem, is indeed a solution to the problem when the two elements are added.

40. Algorithm correctness: There exists a minimum cost tree where the minimum frequency elements are longest path siblings: Assume that is not the situation. Then there are two elements in the longest path. Say a,b are the elements with smallest frequency and x,y the elements in the longest path.

41. Algorithm correctness: CT We know about depth and frequency: da ≤ dy fa ≤ fy da dy a x y

42. Algorithm correctness: We also know about code tree CT: ∑fσdσ σ is smallest possible. CT da dy a x y Now exchange a and y.

43. Algorithm correctness: Cost(CT) = ∑fσdσ= σ ∑fσdσ+fada+fydy≥ σ≠a,y CT’ da dy (da ≤ dy, fa≤ fy Therefore fada ≥fydaand fydy ≥fady ) y ∑fσdσ+fyda+fady= σ≠a,y cost(CT’) x a

44. Algorithm correctness: CT Now do the same thing for b and x db dx b x a

45. Algorithm correctness: CT” And get an optimal code tree where a and b are sibling with the longest paths db dx x b a

46. Algorithm correctness: Optimal substructure property: Let a,b be the symbols with the smallest frequency. Let x be a new symbol whose frequency is fx =fa +fb. Delete characters a and b, and find the optimal code tree CT for the reduced alphabet. Then CT’ = CT U {a,b} is an optimal tree for the original alphabet.

47. Algorithm correctness: CT CT’ x fx = fa + fb x a b

48. Algorithm correctness: cost(CT’)=∑fσd’σ= ∑fσd’σ+ fad’a + fbd’b= σσ≠a,b ∑fσd’σ+ fa(dx+1) + fb (dx+1) = σ≠a,b ∑fσd’σ+(fa + fb)(dx+1)= σ≠a,b ∑fσdσ+fx(dx+1)+fx = cost(CT) + fx σ≠a,b

49. Algorithm correctness: CT CT’ x fx = fa + fb x cost(CT)+fx = cost(CT’) a b

50. Algorithm correctness: Assume CT’ is not optimal. By the previous lemma there is a tree CT” that is optimal, and where a and b are siblings. So cost(CT”) < cost(CT’)