How to Parallelize an Algorithm

1. How to Parallelize an Algorithm Lecture 2

2. Today’s Outline • Quiz • Functional Programming Review • Algorithm Parallelization • Announcements • Projects • Readings

3. Quiz • This one is graded (unlike the last one)

4. Fold & Map review Fold: foldl f z [] = z foldl f z (x:xs) = foldl f (f z x) xs [foldr f z (x:xs) = f x (foldr f z xs)] • Applies a function to every element in the list • Each iteration can access the result of the previous Map: map f [] = [] map f (x:xs) = f x : map xs • Applies a function to every element in the list • Each iteration is independent of all others How would you parallelize Map? Fold?

5. Group Exercises • See handout

6. Answers to Group Exercises Concat: concat xss = foldr (++) [] xss • Given a list of lists, concats all sublists Group: group xss = foldl group_func [] xss group_func (result) (k,v) = if (has (k,v) result) then (map (update (k,v)) result) else (k,v) :: result update (k1,v1) (k2,v_list) = if (EQ k1 k2) then (k1, v1::vlist) else (k2,v_list)

7. Why Parallelize? Reasons to parallelize: • Your reasons here Reasons to not parallelize: • Your reasons here What issues are there in parallelization?

8. Implicit Serialization Example DocInfo: f = read_file("file.txt”) words = count_uniq_words(f) spaces = count_spaces(f) f = capitalize(f) f = remove_punct(f) words2 = count_uniq_words(f) puts(“unique words: ” + words) puts(“num spaces: ” + spaces) puts(“unique scrubbed words: ” + words2) Which statements can be reordered?

9. words2 spaces words f0 f1 f2 console2 console1 console0 Data Dependency Graph f = read_file("file.txt”) words = count_uniq_words(f) spaces = count_spaces(f) f = capitalize(f) f = remove_punct(f) words2 = count_uniq_words(f) puts(“unique words” + words) puts(“num spaces” + spaces) puts(“unique scrubbed words” + words2) read_file capitalize remove_punct count_uniq_words count_spaces puts puts count_uniq_words puts Which operations can be done in parallel?

10. words2 spaces words f0 f1 f2 console2 console1 console0 Distributing Computation read_file capitalize Ram Cpu1 remove_punct Storage count_uniq_words count_spaces Cpu2 puts Cpu3 puts count_uniq_words Cpu4 puts

11. words2 spaces words f0 f1 f2 console2 console0 console1 Eliminating Dependencies 1 of 3 Synchronization points? • f[0,1,2] • console[0,1,2] read_file capitalize remove_punct count_uniq_words count_spaces Ideas for removing them: • Your ideas here puts puts count_uniq_words puts

12. Eliminating Dependencies 2 of 3 captialize, remove_punct can be combined and run first to create a copy of the data before “counting”. DocInfo 2.0: f = read_file("file.txt”) scrubbed_f = scrub_words(f) words = count_uniq_words(f) spaces = count_spaces(f) words2 = count_uniq_words(scrubbed_f) puts(“unique words” + words) puts(“num spaces” + spaces) puts(“unique scrubbed words” + words2)

13. words2 spaces words f console2 console1 console0 Dependency Graph 2.0 f = read_file("file.txt”) scrubbed_f = scrub_words(f) words = count_uniq_words(f) spaces = count_spaces(f) words2 = count_uniq_words(scrubbed_f) puts(“unique words” + words) puts(“num spaces” + spaces) puts(“unique scrubbed words” + words2) scrub_words read_file scrubbed_f count_uniq_words count_uniq_words count_spaces puts puts puts

14. words spaces words2 f console2 console1 console0 Distributing Computation 2.0 scrub_words read_file Ram scrubbed_f Cpu1 Storage count_uniq_words count_uniq_words count_spaces Cpu2 puts Cpu3 puts Cpu4 puts

15. Eliminating Dependencies 3 of 3 captialize, remove_punct only needs to be applied to each word (not the whole file) before “counting”. DocInfo 3.0: f = read_file("file.txt”) words = count_uniq_words(f) spaces = count_spaces(f) words2 = count_uniq_scrubbed_words(f) puts(“unique words” + words) puts(“num spaces” + spaces) puts(“unique scrubbed words” + words2)

16. words2 spaces words f0 console0 console1 console2 Dependency Graph 3.0 f = read_file("file.txt”) words = count_uniq_words(f) spaces = count_spaces(f) words2 = count_uniq_scrubbed_words(f) puts(“unique words” + words) puts(“num spaces” + spaces) puts(“unique scrubbed words” + words2) read_file count_uniq_words count_spaces count_uniq_scrubbed_words puts puts puts

17. words2 spaces words f0 console1 console2 console0 Distributing Computation 3.0 read_file Ram Cpu1 Storage count_uniq_words count_spaces count_uniq_scrubbed_words Cpu2 puts Cpu3 puts Cpu4 puts

18. Parallelization Summary Parallelization tradeoff: Good: better scalability Bad: less algorithmic flexibility, higher complexity Neutral: optimizes for large input over small input Why avoid data dependencies? • lowers complexity • makes parallelization possible How do you avoid data dependencies? • avoid stateful algorithms • avoid side effects (clone state instead of modifying) • avoid global variables and member variables

19. Parallelizing Map What’s required to parallelize map? • function needs to be stateless • function available to each computation unit • input accessible by all computation units • output ordering isn’t important Definition of map: map f [] = [] map f (x:xs) = f x : map xs

20. Parallelizing Fold What’s required to parallelize fold? You can’t. Why can’t you parallelize fold? Each step depends on the result of the previous. Definition of fold: foldl f z [] = z foldl f z (x:xs) = foldl f (f z x) xs How is fold useful in parallel computing then?

21. MapReduce MapReduce: mapreduce fm fr l = map (reducePerKey fr) (group (map fm l)) reducePerKey fr (k,v_list) = (k, (foldl (fr k) [] v_list)) • Assume map here is actually concatMap. • Argument l is a list documents • The result of first map is a list of key-value pairs • The function fr takes 3 arguments key, context, current. With currying, this allows for locking the value of “key” for each list during the fold. MapReduce maps a fold over the sorted result of a map!