Download
1 / 23
240 likes | 513 Views
MapReduce. Simplified Data Processing on Large Clusters Google, Inc. Presented by Prasad Raghavendra. Introduction. Model for processing large data sets. Contains Map and Reduce functions.
E N D
MapReduce Simplified Data Processing on Large Clusters Google, Inc. Presented by Prasad Raghavendra
Introduction • Model for processing large data sets. • Contains Map and Reduce functions. • Runs on a large cluster of machines. • A lot of MapReduce programs are executed on Google’s cluster everyday.
Motivation • Very large data sets need to be processed. - The whole Web, billions of Pages • Lots of machines - Use them efficiently.
Processing of Large Data Sets For example: - Counting access frequency to URLs: Input: list(RequestURL) Output: list(RequestURL, total_number) - Distributed Grep - Distributed Sort
Programming modelInput & Output: each a set of key/value pairs Programmer specifies two functions: map (in_key, in_value) -> list(out_key, intermediate_value) Name comes from map function in LISPEx. (map 'list #’+ '(1 2 3) '(1 2 3)) => (2 4 6)-Processes input key/value pair -Produces set of intermediate pairsmap(document, content) {for each word in contentemit(word, “1”)}
reduce (out_key, list(intermediate_value)) -> list(out_value) Name comes from reduce function in LISPEx. (reduce #’+ '(1 2 3 4 5)) => 15- Combines all intermediate values for a particular key - Produces a set of merged output values (usually just one)reduce(word, values) {result = 0;for each value in valuesresult += valueemitString(w, result)}
ExampleThe problem of counting the number of occurrences of each word in a large collection ofdocuments. • Page 1: the weather is good • Page 2: today is good • Page 3: good weather is good
Map output • Worker 1: (the 1), (weather 1), (is 1), (good 1). • Worker 2: (today 1), (is 1), (good 1). • Worker 3: (good 1), (weather 1), (is 1), (good 1).
Reduce Input • Worker 1:(the 1) • Worker 2: (is 1), (is 1), (is 1) • Worker 3:(weather 1), (weather 1) • Worker 4:(today 1) • Worker 5:(good 1),(good 1), (good 1), (good 1)
Reduce Output • Worker 1: (the 1) • Worker 2: (is 3) • Worker 3: (weather 2) • Worker 4: (today 1) • Worker 5: (good 4)
Flow of MapReduce Operation • The MapReduce library in the user program splits the input files into M pieces(16,64 MB). • One of the copies of the program is special . The master. The rest are workers . • A worker who is assigned a map task parses key/value pairs out of the input data. • Periodically, the buffered pairs are written to local disk. • When a reduce worker is notified by the master about these locations, it uses remote procedure calls to read the buffered data. • The output of the Reduce function is appended to a final output file. • When all map tasks and reduce tasks have been completed, the master wakes up the user program.
Problem: Stragglers • Often some machines are late in their replies - slow disk, overloaded, etc • Approach: - when only few tasks left to execute, start backup tasks - a task completes when either primary or backup completes task • Performance: - without backup, sort (->) takes 44% longer
Partition Function • Defines which worker processes which keys - default: hash(key2) mod R Other partition functions useful: - sort: prefix of k bytes of line - idea: based on known/sampled distribution of key2 to evenly distribute processed keys
Combiner Function • Problem: intermediate results can be quite verbose e.g., (“the”, 1) could occur many times in previous example • Approach: perform a local reduction before writing intermediate results typically, combiner same function as reduce func This will reduce the run-time because less writing to disk and across the network
Performance • Scan 10^10 100-byte records to extract records matching a rare pattern (92K matching records) : 150 seconds. • Sort 10^10 100-byte records (modeled after TeraSort benchmark) : normal 839 seconds.
Fault Tolerance • Crash of worker all - even finished - tasks are redone • Crash of leader crash of leader process -> restart process with checkpoint crash of leader machine-> unlikely - restart computation redo computation
Conclusion • MapReduce has proven to be a useful abstraction • Easy to use • Very large variety of problems are easily expressible as MapReduce computations • Greatly simplifies large-scale computations at Google
