1 / 32

Intro to Map-Reduce

Intro to Map-Reduce. Feb 21, 2014. map-reduce?. A programming model or abstraction. A novel way of thinking about designing a solution to certain problems…. Why?. We have access to huge volumes of data. Facebook posts and photos, twitter streams, …. Easy….

alyson
Download Presentation

Intro to Map-Reduce

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Intro to Map-Reduce Feb 21, 2014

  2. map-reduce? A programming model or abstraction. A novel way of thinking about designing a solution to certain problems… CS512 | Spring 2014

  3. Why? We have access to huge volumes of data. Facebook posts and photos, twitter streams, … CS512 | Spring 2014

  4. Easy… Computational resources are cheap. Amazon EC2, Microsoft Azure, … CS512 | Spring 2014

  5. Not really! Code running on one CPU is simple, two is a headache, four is a nightmare, … You get the picture. CS512 | Spring 2014

  6. Pipe Dream Forget multiple machines. Just write code imagining one CPU. Someone else takes care of running the code on thousands of machine. Free the programmer from the unnecessary details. CS512 | Spring 2014

  7. Map-Reduce map-reduce programming model to the rescue CS512 | Spring 2014

  8. Long long ago… LISP, 1958 A programming language that introduced several innovative ideas Recursive Functions of Symbolic Expression and Their Computation by Machine, Part I John McCarthy, MIT, April 1960 CS512 | Spring 2014

  9. LISP Introduced map and reduce. CS512 | Spring 2014

  10. Map map(mf, [a1, a2, …an]) -> [b1, b2, …, bn] Accepts two arguments: a function and a list of values. Generates output by repeatedly applying the function on the list of values. CS512 | Spring 2014

  11. Reduce reduce(rf, [b1, b2, …bn]) -> c Accepts two arguments: a function and a list of values. Generates output by reducing the list of input values using the function. CS512 | Spring 2014

  12. Simple composition Map’s output is a list of values, which reduce can accept as one of its argument. CS512 | Spring 2014

  13. Analogy Break large problem into small pieces Code mf to solve one piece Run map to apply mf on the small pieces and generate nuggets of solutions Code rf to combine the nuggets Run reduce to apply rf on the nuggets to output the complete solution CS512 | Spring 2014

  14. Example 1TB file split into 100,000 chunks Count number of lines in each chunk Add counts together to output final line count CS512 | Spring 2014

  15. A slightly different map-reduce Map Copies a function on a number of machines and applies each copy on different pieces of the input Reduce Combine the map outputs from different machines into a final solution CS512 | Spring 2014

  16. Map-reduce reintroduced… Google created the awareness Hadoop made it into a sensation Hadoop is an open-source map-reduce implementation based on Google’s paper. MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean and Sanjay Ghemawat OSDI'04: Sixth Symposium on Operating System Design and Implementation. December, 2004. CS512 | Spring 2014

  17. Hadoop CS512 | Spring 2014

  18. Example CS512 | Spring 2014

  19. Terminology Mapper Instance of the map function Reducer Instance of the reduce function CS512 | Spring 2014

  20. Job User’s implementation of map and reduce functions CS512 | Spring 2014

  21. Splitting the input User submits job and specifies the input files. Input files are split into chunks. Chunks are fed to the mappers typically over a distributed file system like HDFS. CS512 | Spring 2014

  22. Copying the job JobTracker Hadoop service that copies the map and reduce code to available machines. Feeds the input to the mappers and connects their outputs to reducers. CS512 | Spring 2014

  23. Maps in parallel Maps run in parallel. Each maps operates on a set of chunks assigned to it by the job tracker. Maps write to local disk. CS512 | Spring 2014

  24. What if maps fail? Re-run failed maps. No need to re-run succeeded maps. Why does this work? Maps typically are idempotent. CS512 | Spring 2014

  25. Input to reducers #reducers(n) known a priori. #partitions equals #reducers. Hash on the keys of mapper outputs. partition = hash(key) mod n Load balancing by randomization. CS512 | Spring 2014

  26. Wait before reducing… Cannot start reducers before mappers complete. Synchronization barrier between map and reduce phases. Why? CS512 | Spring 2014

  27. Embarrassing Parallelism CS512 | Spring 2014

  28. Not a panacea! If your workload exhibits embarrassing parallelism, Hadoop might be the ideal framework. If not, look for other parallel programming paradigms. CS512 | Spring 2014

  29. Example CS512 | Spring 2014

  30. WordCount https://developer.yahoo.com/hadoop/tutorial/module4.html CS512 | Spring 2014

  31. Mapper public static class MyMapper implements Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException{ // Split the given line (in value) into words and emit for each word the tuple <word, 1> String line = value.toString(); StringTokenizeritr = new StringTokenizer(line); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); output.collect(word, one); } } } https://developer.yahoo.com/hadoop/tutorial/module4.html CS512 | Spring 2014

  32. Reducer public static class MyReducerimplements Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { int sum = 0; while (values.hasNext()) { sum += values.next().get(); } output.collect(key, new IntWritable(sum)); } } https://developer.yahoo.com/hadoop/tutorial/module4.html CS512 | Spring 2014

More Related