Sort in MapReduce

1. Sort in MapReduce

2. MapReduce Shuffle/Sort Map Block 1 Map Reduce Output 1 Block 2 Map • Block 3 Reduce Output 2 • Block 4 Map • Block 5 Map

3. How to Sort in MapReduce? Shuffle/Sort Shuffle/Sort Map Block 1 Map Reduce Output 1 Block 2 Map • Block 3 Reduce Output 2 • Block 4 Map • Block 5 Map Use the Built-In Mechanism to Sort Data

4. Mapper (1337,null) (1,1337) Mapper 1337 1103 1242 0932 (1103,null) (2,1103) (1242,null) (3,1242) (0932,null) (4,0932) Block 1 Mapper (0432,null) (5,0432) 0432 1690 1324 0232 (1690,null) (6,1690) (1324,null) (7,1324) (0232,null) (8,0232) Block 2 Intermediate Key-Value Pairs Input Key-Value Pairs Input

5. Reducer Reducer (1337,null) 0232 0432 0932 (1103,null) (1242,null) (0932,null) Output 1 1103 1242 1324 1337 1690 Reducer (0432,null) (1690,null) (1324,null) (0232,null) Output 2 Intermediate Key-Value Pairs Sorted Outputs Shuffle and Sort

6. The MapReduce Program Mapper: Input Key-Value Pairs: (k, v) Output Key-Value Pairs: (v, null) Reducer: Input Key-Value Pairs: (v, null) Output Key-Value Pairs: (v, null) • publicvoid map(Object key, Text value, Context context) throwsException • { • //Parse the value, if required. • context.write(value, null); • }

7. The MapReduce Program Mapper: Input Key-Value Pairs: (k, v) Output Key-Value Pairs: (v, null) Reducer: Input Key-Value Pairs: (v, null) Output Key-Value Pairs: (v, null) • publicvoid reduce(Text key, Iterable<Text> values, Context context) throwsIOException, InterruptedException • { • while(values.iterator().hasNext()) • { • context.write(key, values.iterator().next()); • } • }