Improving MapReduce Performance in Heterogeneous Environments

1. Improving MapReduce Performance in Heterogeneous Environments Sturzu Antonio-Gabriel, SCPD

2. Table of Contents • Intro • Scheduling in Hadoop • Heterogeneity in Hadoop • The LATE Scheduler(Longest Approximate Time to End) • Evaluation of Performance

3. Intro • The big volume of data that internet services work on has led to the need of parallel processing • The leading example is Google which uses MapReduce to process 20 petabytes of data per day • MapReduce breaks a computation into small tasks that run in parallel on multiple machines and scales easy to very large clusters • The two key benefits of MapReduce are: • Fault tolerance • Speculative execution

4. Intro • Google has noted that speculative execution improves response time by 44% • The paper shows an efficient way to do speculative execution in order to maximize performance • It also shows that Hadoop’s simple speculative algorithm based on comparing each task’s progress to the average progress brakes down in heterogeneous systems

5. Intro • The proposed scheduling algorithm increases Hadoop’s response time by a factor of two • Most of the examples and tests are done on Amazon’s Elastic Compute Cloud( EC2) • The paper adresses two important problems in speculative execution: • Choosing the best node to run the speculative task • Distinguishing between nodes slightly slower than the mean and stragglers

6. Scheduling in Hadoop • Hadoop divides each MapReduce job into tasks • The input file is split into even-sized chunks replicated for fault tolerance • Each chunk of input is first processed by a map task that outputs a set of key-value pairs • Map outputs are split into buckets based on the key

7. Scheduling in Hadoop • When all map tasks finish reducers apply a reduce function on the set of values associated with each key

8. Scheduling in Hadoop • Hadoop runs several maps and reduces concurrently on each slave in order to overlap I/O with computation • Each slave tells the master when it has empty task slots • First any failed task is given priority • Second a non-running task • Third a task to execute speculatively

9. Scheduling in Hadoop • To select speculative tasks Hadoop uses a progress score between 0 and 1 • For a map the progress score is the fraction of input data read • For a reduce task the execution is divided into 3 phases each of which accounts for 1/3 of the score: • The copy phase • The sort phase • The reduce phase

10. Scheduling in Hadoop • In each phase the score is the fraction of data processed • Hadoop calculates an average score for each category of tasks in order to define a threshold for speculative execution • When a task’s progress score is less than the average for its category minus 0.2 and the task has run for at least a minute it is marked as a straggler

11. Scheduling in Hadoop • All tasks beyond the threshold are considered equally slow and ties between them are broken by data locality • This threshold works well in homogeneous systems because tasks tend to start and finish in “waves” at roughly the same times • When running multiple jobs Hadoop uses a FIFO discipline

12. Scheduling in Hadoop • Assumptions made by Hadoop Scheduler: • Nodes can perform work at roughly the same rate • Tasks progress at a constant rate throughout time • There is no cost to launching a speculative task on a node that would otherwise have an idle slot

13. Scheduling in Hadoop • A task’s progress score is a representative of fraction of its total work that it has done • Tasks tend to finish in waves, so a task with a low progress score is likely a straggler • Tasks in the same category (map or reduce) require roughly the same amount of work

14. Heterogeneity in Hadoop • Too many speculative tasks are launched because of the fixed threshold (assumption 3 falls) • Because the scheduler uses data locality to rank candidates for speculative execution the wrong tasks may be chosen first • Assumptions 3, 4 and 5 fall on both homogeneous and heterogeneous clusters

15. The LATE Scheduler • The main idea is that it speculatively executes the task that will finish farthest in the future • Estimates progress rate as ProgressScore/T, where T is the amount of time the task has been running for • The time to completion is (1-ProgressScore)/ProgressRate

16. The LATE Scheduler • In order to get the best chance to beat the original task which was speculated the algorithm launches speculative tasks only on fast nodes • It does this using a SlowNodeThreshold which is a metric of the total work performed • Because speculative tasks cost resources LATE uses two additional heuristics: • A limit on the number of speculative tasks executed (SpeculativeCap) • A SlowTaskThreshold that determines if a task is slow enough in order to get speculated (uses progress rate for comparison)

17. The LATE Scheduler • When a node asks for a new task and the number of speculative tasks is less than the threshold: • If the node’s progress score is below SlowNodeThreshold ignore the request • Rank currently running tasks that are not being speculated by estimating completion time • Launch a copy of the highest ranked task whose progress rate is below SlowTaskThreshold • Doesn’t take into account data locality

18. The LATE Scheduler • Advantages of the algorithm: • Robust to node heterogeneity because it launches only the slowest tasks and only few of them • Prioritized among slow tasks based on how they hurt response time • Takes into account node heterogeneity when choosing on which node to run a speculative task • Executes only tasks that will improve the total response time, not any slow task

19. The LATE Scheduler • The time completion estimation can produce errors when a task’s progress rate decreases but in general gets correct approximations in typical MapReduce jobs

20. Evaluation • In order to create heterogeneity they mapped a variable number of virtual machines (from 1 to 8) on each host in the EC2 cluster • They measured the impact of contention on I/O performance and Application Level Performance

21. Evaluation

22. Evaluation • For Application Level they sorted 100 GB of random data using Hadoop’s Sort benchmark with speculative execution disabled • With isolated VM’s the job completed in 408 s and with VM’s packed densely onto physical hosts (7 VM’s per host) it took 1094s • For evaluating the scheduling algorithms they used clusters of about 200 VM’s and they performed 5-7 runs

23. Evaluation • Results for scheduling in a heterogeneous cluster of 243 VM’s using the Sort job (128 MB per host) for a total of 30GB of data:

24. Evaluation • On average LATE finished jobs 27% faster than Hadoop’s native scheduler and 31% faster than no speculation • Results for scheduling with stragglers • In order to simulate stragglers they manually slowed down eight VM’s in a cluster of 100

25. Evaluation • For each run they sorted 256 MB per host for a total of 25GB:

26. Evaluation • They also ran other two workloads on a heterogeneous cluster with stragglers: • Grep • WordCount • They used a 204 node cluster with 1 to 8 VM’s per host • For the Grep test they searched on 43GB of text data or about 200 MB per host

27. Evaluation

28. Evaluation • On average LATE finished jobs 36% faster than Hadoop’s native scheduler and 56% faster than no speculation • For the WordCount test they used a data set of 21GB or 100 MB per host

29. Evaluation

30. Evaluation • Sensitivity analysis • SpeculativeCap • SlowTaskThreshold • SlowNodeThreshold • SpeculativeCap results • They ran experiments at six SpeculativeCap values from 2.5% to 100% repeating each experiment 5 times

31. Evaluation

32. Evaluation • Sensitivity to SlowTaskThreshold • Here the idea is to not speculate tasks that are progressing fast if they are the only tasks left • They tested 6 values from 5% to 100%

33. Evaluation

34. Evaluation • We observe that values past 25% all work well with 25% being the optimum value • Sensitivity to SlowNodeThreshold

35. Evaluation