Effective Straggler Mitigation: Attack of the Clones

1. Effective Straggler Mitigation: Attack of the Clones Ganesh Ananthanarayanan, Ali Ghodsi, Srikanth Kandula, Scott Shenker, Ion Stoica

2. Small jobs increasingly important • Most jobs are small • 82% of jobs contain less than 10 tasks (Facebook’s Hadoop cluster) • Small jobs often are interactiveandlatency-constrained • Data analyst testing query on small sample • New frameworks targeted at interactive analyses

3. Stragglers in Small Jobs • Small jobs particularly sensitive to stragglers • Inordinately slow tasks that delay job completion • Straggler Mitigation: • Blacklisting: Clusters periodically diagnose and eliminate machines with faulty hardware • Speculation: LATE [OSDI’08], Mantri [OSDI’10]… • Address the non-deterministic stragglers • Complete systemic modeling is intrinsically complex

4. Despite the mitigation techniques… • LATE: The slowest task runs 8 times slower*than the median task • Mantri: The slowest task runs 6 times slower*than the median task • (…but they work well for large jobs) * progress rate of a task = input-size/duration

5. State-of-the-art Straggler Mitigation Speculative Execution: • Wait: observe relative progress rates of tasks • Speculate: launch copies of tasks that are predicted to be stragglers

6. Why doesn’t this work for small jobs? • Consist of just a few tasks • Statistically hard to predict stragglers • Need to wait longer to accurately predict stragglers • Run all their tasks simultaneously • Waiting can constitute considerable fraction of a small job’s duration Wait & Speculate is ill-suited to address stragglers in small jobs

7. CloningJobs • Proactivelylaunch clones of a job, just as they are submitted • Pick the result from the earliest clone • Probabilistically mitigates stragglers • Eschews waiting, speculation, causal analysis… Is this really feasible??

8. Heavy-tailed Distribution 90% of jobs use 6% of resources Can clone small jobs with few extra resources

9. Challenge: Avoid I/O contention • Every clone should get its own copy of data • Input data of jobs • Replicated three times (typically) • Storage crunch: Cannot increase replication • Intermediate data of jobs • Not replicated at all, to avoid overheads

10. Strawman: Job-level Cloning • Easy to implement • Directly extends to any framework M1 R1 M2 Job Earliest M1 R1 M2

11. Number of clones • Contentionfor input data by map task clones • Storage crunch  Cannot increase replication (Map-only job) >> 3clones

12. Task-level Cloning M1 Earliest M1 Earliest R1 Job R1 Earliest M2 M2

13. ≤3 clones suffices Task-level Cloning Strawman

14. Intermediate Data Contention • We would like every reduce clone to get its own copy of intermediate data (map output) • When a map clones does not straggle, use its output • When they do straggle?

15. Contention-Avoidance Cloning (CAC) M1 M1 Exclusive copy M1 R1 R1 R1 M2 M2 M2 • Jobs are more vulnerable to stragglers

16. Contention Cloning (CC) Earliest copy M1 M1 R1 R1 M2 M2 • Intermediate data transfer takes longer

17. CAC vs. CC • CACavoids contentions but makes jobs more vulnerable to stragglers • Straggler probability in a job increases by >10% • CCmitigates stragglers in jobs but causes contentions • Shuffle takes ~50% longer • Do not distinguish intrinsic variations in task durations fromstragglers

18. Delay Assignment • Small delay before contending for the available copy of the intermediate data • (Similar to delay scheduling [EuroSys’10]) • Probabilistic modelingof the delay • Expected task durations • Read bandwidths w/ and w/o contention • Happens automatically and periodically

19. Dolly: Cloning Jobs • Task-level cloning of jobs • Delay Assignment to manage intermediate data • Works within a budget • Cap on the extra cluster resources for cloning

20. Evaluation Setup • Workload derived from Facebook traces • FB: 3500 node Hadoop cluster, 375K jobs, 1 month • Prototype on top of Hadoop 0.20.2 • Experiments on 150-node cluster • Baselines: LATE and Mantri, + blacklisting • Cloning budget of 5%

21. Average job completion time Jobs are 44% and 42% faster w.r.t. LATE and Mantri Slowest task in a job now runs 1.06x times slower than median (down from 8x)

22. Delay Assignment is crucial… 1.5x – 2x better (Exclusive Copy) (Earliest Copy)

23. …and gets better with #phases in job • Dryad jobs have multiple phases in a single job Steady gains, and outperforms CAC and CC

24. Summary • Stragglers in small jobs are not well-handled by traditional mitigation strategies • Dolly: Proactive Cloning of jobs • Heavy-tail Small cloning budget (5%) suffices • Jobs improve by at least 42% w.r.t. state-of-the-art straggler mitigation strategies