Jimmy Lin The iSchool University of Maryland Monday, March 30, 2009

1. Fundamentals of MapReduce Jimmy Lin The iSchool University of Maryland Monday, March 30, 2009 Some material adapted from slides by Christophe Bisciglia, Aaron Kimball, & Sierra Michels-Slettvet, Google Distributed Computing Seminar, 2007 (licensed under Creation Commons Attribution 3.0 License) This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United StatesSee http://creativecommons.org/licenses/by-nc-sa/3.0/us/ for details

2. This afternoon... • Introduction to MapReduce • MapReduce “killer app” #1: text retrieval • MapReduce “killer app” #2: graph algorithms

3. Introduction to MapReduce

4. Introduction to MapReduce: Topics • Functional programming • MapReduce • Distributed file system

5. Functional Programming • MapReduce = functional programming meets distributed processing on steroids • Not a new idea… dates back to the 50’s (or even 30’s) • What is functional programming? • Computation as application of functions • Theoretical foundation provided by lambda calculus • How is it different? • Traditional notions of “data” and “instructions” are not applicable • Data flows are implicit in program • Different orders of execution are possible • Exemplified by LISP and ML

6. Overview of Lisp • Lisp ≠ Lost In Silly Parentheses • We’ll focus on particular a dialect: “Scheme” • Lists are primitive data types • Functions written in prefix notation '(1 2 3 4 5) '((a 1) (b 2) (c 3)) (+ 1 2)  3 (* 3 4)  12 (sqrt (+ (* 3 3) (* 4 4)))  5 (define x 3)  x (* x 5)  15

7. Functions • Functions = lambda expressions bound to variables • Syntactic sugar for defining functions • Above expressions is equivalent to: • Once defined, function can be applied: (define foo (lambda (x y) (sqrt (+ (* x x) (* y y))))) (define (foo x y) (sqrt (+ (* x x) (* y y)))) (foo 3 4)  5

8. Other Features • In Scheme, everything is an s-expression • No distinction between “data” and “code” • Easy to write self-modifying code • Higher-order functions • Functions that take other functions as arguments (define (bar f x) (f (f x))) Doesn’t matter what f is, just apply it twice. (define (baz x) (* x x)) (bar baz 2)  16

9. Recursion is your friend • Simple factorial example • Even iteration is written with recursive calls! (define (factorial n) (if (= n 1) 1 (* n (factorial (- n 1))))) (factorial 6)  720 (define (factorial-iter n) (define (aux n top product) (if (= n top) (* n product) (aux (+ n 1) top (* n product)))) (aux 1 n 1)) (factorial-iter 6)  720

10. Lisp  MapReduce? • What does this have to do with MapReduce? • After all, Lisp is about processing lists • Two important concepts in functional programming • Map: do something to everything in a list • Fold: combine results of a list in some way

11. Map • Map is a higher-order function • How map works: • Function is applied to every element in a list • Result is a new list f f f f f

12. Fold • Fold is also a higher-order function • How fold works: • Accumulator set to initial value • Function applied to list element and the accumulator • Result stored in the accumulator • Repeated for every item in the list • Result is the final value in the accumulator f f f f f final value Initial value

13. Map/Fold in Action • Simple map example: • Fold examples: • Sum of squares: (map (lambda (x) (* x x)) '(1 2 3 4 5))  '(1 4 9 16 25) (fold + 0 '(1 2 3 4 5))  15 (fold * 1 '(1 2 3 4 5))  120 (define (sum-of-squares v) (fold + 0 (map (lambda (x) (* x x)) v))) (sum-of-squares '(1 2 3 4 5))  55

14. Lisp  MapReduce • Let’s assume a long list of records: imagine if... • We can parallelize map operations • We have a mechanism for bringing map results back together in the fold operation • That’s MapReduce! • Observations: • No limit to map parallelization since maps are indepedent • We can reorder folding if the fold function is commutative and associative

15. Typical Problem • Iterate over a large number of records • Extract something of interest from each • Shuffle and sort intermediate results • Aggregate intermediate results • Generate final output Map Reduce Key idea:provide an abstraction at the point of these two operations

16. MapReduce • Programmers specify two functions: map (k, v) → <k’, v’>* reduce (k’, v’) → <k’, v’>* • All v’ with the same k’ are reduced together • Usually, programmers also specify: partition (k’, number of partitions ) → partition for k’ • Often a simple hash of the key, e.g. hash(k’) mod n • Allows reduce operations for different keys in parallel • Implementations: • Google has a proprietary implementation in C++ • Hadoop is an open source implementation in Java

17. k1 v1 k2 v2 k3 v3 k4 v4 k5 v5 k6 v6 map map map map a 1 b 2 c 3 c 6 a 5 c 2 b 7 c 9 Shuffle and Sort: aggregate values by keys a 1 5 b 2 7 c 2 3 6 9 reduce reduce reduce r1 s1 r2 s2 r3 s3

18. Recall these problems? • How do we assign work units to workers? • What if we have more work units than workers? • What if workers need to share partial results? • How do we aggregate partial results? • How do we know all the workers have finished? • What if workers die?

19. MapReduce Runtime • Handles scheduling • Assigns workers to map and reduce tasks • Handles “data distribution” • Moves the process to the data • Handles synchronization • Gathers, sorts, and shuffles intermediate data • Handles faults • Detects worker failures and restarts • Everything happens on top of a distributed FS (later)

20. “Hello World”: Word Count Map(String input_key, String input_value): // input_key: document name // input_value: document contents for each word w in input_values: EmitIntermediate(w, "1"); Reduce(String key, Iterator intermediate_values): // key: a word, same for input and output // intermediate_values: a list of counts int result = 0; for each v in intermediate_values: result += ParseInt(v); Emit(AsString(result));

21. UserProgram (1) fork (1) fork (1) fork Master (2) assign map (2) assign reduce worker split 0 (6) write output file 0 worker split 1 (5) remote read (3) read split 2 (4) local write worker split 3 output file 1 split 4 worker worker Input files Map phase Intermediate files (on local disk) Reduce phase Output files Redrawn from Dean and Ghemawat (OSDI 2004)

22. Bandwidth Optimization • Issue: large number of key-value pairs • Solution: use “Combiner” functions • Executed on same machine as mapper • Results in a “mini-reduce” right after the map phase • Reduces key-value pairs to save bandwidth

23. Skew Problem • Issue: reduce is only as fast as the slowest map • Solution: redundantly execute map operations, use results of first to finish • Addresses hardware problems... • But not issues related to inherent distribution of data

24. How do we get data to the workers? SAN Compute Nodes NAS What’s the problem here?

25. Distributed File System • Don’t move data to workers… Move workers to the data! • Store data on the local disks for nodes in the cluster • Start up the workers on the node that has the data local • Why? • Not enough RAM to hold all the data in memory • Disk access is slow, disk throughput is good • A distributed file system is the answer • GFS (Google File System) • HDFS for Hadoop

26. GFS: Assumptions • Commodity hardware over “exotic” hardware • High component failure rates • Inexpensive commodity components fail all the time • “Modest” number of HUGE files • Files are write-once, mostly appended to • Perhaps concurrently • Large streaming reads over random access • High sustained throughput over low latency GFS slides adapted from material by Dean et al.

27. GFS: Design Decisions • Files stored as chunks • Fixed size (64MB) • Reliability through replication • Each chunk replicated across 3+ chunkservers • Single master to coordinate access, keep metadata • Simple centralized management • No data caching • Little benefit due to large data sets, streaming reads • Simplify the API • Push some of the issues onto the client

28. Application GFS master /foo/bar (file name, chunk index) File namespace GSF Client chunk 2ef0 (chunk handle, chunk location) Instructions to chunkserver Chunkserver state (chunk handle, byte range) GFS chunkserver GFS chunkserver chunk data Linux file system Linux file system … … Redrawn from Ghemawat et al. (SOSP 2003)

29. Single Master • We know this is a: • Single point of failure • Scalability bottleneck • GFS solutions: • Shadow masters • Minimize master involvement • Never move data through it, use only for metadata (and cache metadata at clients) • Large chunk size • Master delegates authority to primary replicas in data mutations (chunk leases) • Simple, and good enough!

30. Master’s Responsibilities (1/2) • Metadata storage • Namespace management/locking • Periodic communication with chunkservers • Give instructions, collect state, track cluster health • Chunk creation, re-replication, rebalancing • Balance space utilization and access speed • Spread replicas across racks to reduce correlated failures • Re-replicate data if redundancy falls below threshold • Rebalance data to smooth out storage and request load

31. Master’s Responsibilities (2/2) • Garbage Collection • Simpler, more reliable than traditional file delete • Master logs the deletion, renames the file to a hidden name • Lazily garbage collects hidden files • Stale replica deletion • Detect “stale” replicas using chunk version numbers

32. Metadata • Global metadata is stored on the master • File and chunk namespaces • Mapping from files to chunks • Locations of each chunk’s replicas • All in memory (64 bytes / chunk) • Fast • Easily accessible • Master has an operation log for persistent logging of critical metadata updates • Persistent on local disk • Replicated • Checkpoints for faster recovery

33. Mutations • Mutation = write or append • Must be done for all replicas • Goal: minimize master involvement • Lease mechanism: • Master picks one replica as primary; gives it a “lease” for mutations • Primary defines a serial order of mutations • All replicas follow this order • Data flow decoupled from control flow

34. Parallelization Problems • How do we assign work units to workers? • What if we have more work units than workers? • What if workers need to share partial results? • How do we aggregate partial results? • How do we know all the workers have finished? • What if workers die? How is MapReduce different?

35. Questions?

36. MapReduce “killer app” #1: Text Retrieval

37. Text Retrieval: Topics • Introduction to information retrieval (IR) • Boolean retrieval • Ranked retrieval • Text retrieval with MapReduce

38. Resource Query Results Documents System discovery Vocabulary discovery Concept discovery Document discovery Information source reselection The Information Retrieval Cycle Source Selection Query Formulation Search Selection Examination Delivery

39. The Central Problem in Search Author Searcher Concepts Concepts Query Terms Document Terms “tragic love story” “fateful star-crossed romance” Do these represent the same concepts?

40. Architecture of IR Systems Documents Query online offline Representation Function Representation Function Query Representation Document Representation Index Comparison Function Hits

41. How do we represent text? • Remember: computers don’t “understand” anything! • “Bag of words” • Treat all the words in a document as index terms for that document • Assign a “weight” to each term based on “importance” • Disregard order, structure, meaning, etc. of the words • Simple, yet effective! • Assumptions • Term occurrence is independent • Document relevance is independent • “Words” are well-defined

42. What’s a word? 天主教教宗若望保祿二世因感冒再度住進醫院。這是他今年第二度因同樣的病因住院。 وقال مارك ريجيف - الناطق باسم الخارجية الإسرائيلية - إن شارون قبل الدعوة وسيقوم للمرة الأولى بزيارة تونس، التي كانت لفترة طويلة المقر الرسمي لمنظمة التحرير الفلسطينية بعد خروجها من لبنان عام 1982. Выступая в Мещанском суде Москвы экс-глава ЮКОСа заявил не совершал ничего противозаконного, в чем обвиняет его генпрокуратура России. भारत सरकार ने आर्थिक सर्वेक्षण में वित्तीय वर्ष 2005-06 में सात फ़ीसदी विकास दर हासिल करने का आकलन किया है और कर सुधार पर ज़ोर दिया है 日米連合で台頭中国に対処…アーミテージ前副長官提言 조재영 기자= 서울시는 25일 이명박 시장이 `행정중심복합도시'' 건설안에 대해 `군대라도 동원해 막고싶은 심정''이라고 말했다는 일부 언론의 보도를 부인했다.

43. McDonald's slims down spuds Fast-food chain to reduce certain types of fat in its french fries with new cooking oil. NEW YORK (CNN/Money) - McDonald's Corp. is cutting the amount of "bad" fat in its french fries nearly in half, the fast-food chain said Tuesday as it moves to make all its fried menu items healthier. But does that mean the popular shoestring fries won't taste the same? The company says no. "It's a win-win for our customers because they are getting the same great french-fry taste along with an even healthier nutrition profile," said Mike Roberts, president of McDonald's USA. But others are not so sure. McDonald's will not specifically discuss the kind of oil it plans to use, but at least one nutrition expert says playing with the formula could mean a different taste. Shares of Oak Brook, Ill.-based McDonald's (MCD: down $0.54 to $23.22, Research, Estimates) were lower Tuesday afternoon. It was unclear Tuesday whether competitors Burger King and Wendy's International (WEN: down $0.80 to $34.91, Research, Estimates) would follow suit. Neither company could immediately be reached for comment. … 16 × said 14 × McDonalds 12 × fat 11 × fries 8 × new 6 × company, french, nutrition 5 × food, oil, percent, reduce, taste, Tuesday … Sample Document “Bag of Words”

44. Boolean Retrieval • Users express queries as a Boolean expression • AND, OR, NOT • Can be arbitrarily nested • Retrieval is based on the notion of sets • Any given query divides the collection into two sets: retrieved, not-retrieved • Pure Boolean systems do not define an ordering of the results

45. aid 0 1 all 0 1 back 1 0 brown 1 0 come 0 1 dog 1 0 fox 1 0 good 0 1 jump 1 0 lazy 1 0 men 0 1 now 0 1 over 1 0 party 0 1 quick 1 0 their 0 1 time 0 1 Representing Documents Document 1 Term Document 1 Document 2 The quick brown fox jumped over the lazy dog’s back. Stopword List for is of Document 2 the to Now is the time for all good men to come to the aid of their party.

46. Term Doc 2 Doc 3 Doc 4 Doc 1 Doc 5 Doc 6 Doc 7 Doc 8 aid 0 0 0 1 0 0 0 1 all 0 1 0 1 0 1 0 0 back 1 0 1 0 0 0 1 0 brown 1 0 1 0 1 0 1 0 come 0 1 0 1 0 1 0 1 dog 0 0 1 0 1 0 0 0 fox 0 0 1 0 1 0 1 0 good 0 1 0 1 0 1 0 1 jump 0 0 1 0 0 0 0 0 lazy 1 0 1 0 1 0 1 0 men 0 1 0 1 0 0 0 1 now 0 1 0 0 0 1 0 1 over 1 0 1 0 1 0 1 1 party 0 0 0 0 0 1 0 1 quick 1 0 1 0 0 0 0 0 their 1 0 0 0 1 0 1 0 time 0 1 0 1 0 1 0 0 Inverted Index Term Postings aid 4 8 all 2 4 6 back 1 3 7 brown 1 3 5 7 come 2 4 6 8 dog 3 5 fox 3 5 7 good 2 4 6 8 jump 3 lazy 1 3 5 7 men 2 4 8 now 2 6 8 over 1 3 5 7 8 party 6 8 quick 1 3 their 1 5 7 time 2 4 6

47. Boolean Retrieval • To execute a Boolean query: • Build query syntax tree • For each clause, look up postings • Traverse postings and apply Boolean operator • Efficiency analysis • Postings traversal is linear (assuming sorted postings) • Start with shortest posting first AND ( fox or dog ) and quick quick OR fox dog dog 3 5 fox 3 5 7 dog 3 5 OR = union 3 5 7 fox 3 5 7

48. Extensions • Implementing proximity operators • Store word offset in postings • Handling term variations • Stem words: love, loving, loves …  lov

49. Strengths and Weaknesses • Strengths • Precise, if you know the right strategies • Precise, if you have an idea of what you’re looking for • Implementations are fast and efficient • Weaknesses • Users must learn Boolean logic • Boolean logic insufficient to capture the richness of language • No control over size of result set: either too many hits or none • When do you stop reading? All documents in the result set are considered “equally good” • What about partial matches? Documents that “don’t quite match” the query may be useful also

50. Ranked Retrieval • Order documents by how likely they are to be relevant to the information need • Estimate relevance(q, di) • Sort documents by relevance • Display sorted results • User model • Present hits one screen at a time, best results first • At any point, users can decide to stop looking • How do we estimate relevance? • Assume document is relevant if it has a lot of query terms • Replace relevance (q, di) with sim(q, di) • Compute similarity of vector representations