CARDIO: Cost-Aware Replication for Data-Intensive workflOws

1. CARDIO: Cost-Aware Replication for Data-Intensive workflOws Presented by Chen He

2. Motivation • Is large scale cluster reliable? • 5 average worker deaths per Map-Reduce job • At least 1 disk failure in every run of a 6- hour MapReduce job on a 4000-node cluster

3. Motivation • How to prevent node failure from affecting performance? • Replication • Capacity constraint • Replication time, etc • Regeneration through re-execution • Delay program progress • Cascaded re-execution

4. Motivation COST AVAILABILITY All pictures adopted from the Internet

5. Outline • Problem Exploration • CARDIO Model • Hadoop CARDIO System • Evaluation • Discussion

6. Problem Exploration • Performance Costs • Replication cost (R) • Regeneration cost (G) • Reliability cost (Z) • Execution cost (A) • Total cost (T) • Disk cost (Y) T=A+Z Z=R+G

7. Problem Exploration • Experiment Environment • Hadoop 0.20.2 • 25 VMs • Workloads: Tagger->Join->Grep->RecordCounter

8. Problem Exploration Summary • Replication Factor for MR Stages

9. Problem Exploration Summary • Detailed Execution Time of 3 Cases

10. CARDIO Model • Block Failure Model • Output of stage i is • Replication factor is • Total block number is • Single block failure probability is • Failure probability in stage i:

11. CARDIO Model • Cost Computation Model • Total time of stage i: • Replication cost of stage i: • Expected regeneration time of stage i: • Reliability cost for all stages: • Storage Constraint C of all stages: • Choose to minimize Z

12. CARDIO Model • Dynamic Replication • Replication number x may vary during the program approaching • Job is in Step k, the replication factor at this step is:

13. CARDIO Model • Model for Reliability • Minimize • Based on • In the condition of

14. CARDIO Model • Resource Utilization Model • Model Cost = resource utilized • Resource type Q • CPU, Network, Disk, and Storage resource, etc. • Utilization of q resource in stage i: • Normalize usage by • Relative costs weights:

15. CARDIO Model • Resource Utilization Model • The cost for A is: • Total Cost: • Optimization target: • Choose to minimize T

16. CARDIO Model • Optimization Problem • Job optimality (JO) • Stage optimality (SO)

17. Hadoop CARDIO System • CardioSense • Obtain progress from JT periodically • Be triggered by pre-configured threshold-value • Collect resource usage statistics for running stages • Rely on HMon on each worker node • HMon based on Atop has low overhead

18. Hadoop CARDIO System • CardioSolve • Receive data from CardioSense • Solve SO problem • Decide the replication factors for current and previous stages

19. Hadoop CARDIO System • CardioAct • Implement the command from CardioSolve • Use HDFS API setReplication(file, replicaNumber)

20. Hadoop CARDIO System

21. Evaluation • Several Important Parameters • p is the failure rate 0.2 if not specified • is the time to replicate a data unit, 0.2 as well • is the computation resource of stage i, it follows uniform distribution U(1,Cmax),Cmax=100 in general. • is the output of stage i, it is obtained from a uniform distribution U(1, Dmax), Dmax varies within the [1,Cmax]. • C is the storage constraint for the whole process. Default value is

22. Evaluation • Effect of Dmax

23. Evaluation • Effect of Failure rate p

24. Evaluation • Effect of block size

25. Evaluation • Effect of different resource constraints ++ means over-utilzed, and this type of resource is regarded as expensive P=0.08, C=204GB, delta=0.6 S3 is CPU intensive DSK has similar performance pattern as NET CPU 0010, NET 0011, DSKIO 0011,STG0011

26. Evaluation S2 re-execute more frequently due to the failure injection. Because it has large data output. P=0.02, 0.08 and 0.1 1 , 3, 21 API reason

27. Discussion • Problems • Typos and misleading symbols • HDFS API setReplication() • Any other ideas?