Resource Management with YARN: YARN Past, Present and Future

1. Resource Management with YARN: • YARN Past, Present and Future Anubhav Dhoot Software Engineer Cloudera

2. Resource Management Map Reduce Impala Spark YARN (DYNAMIC RESOURCE MANAGEMENT)

3. YARN (Yet Another Resource Negotiator) Hadoop Traditional Operating System Storage: Hadoop Distributed File System (HDFS) Execution/ Scheduling: Yet Another Resource Negotiator (YARN) Storage: File System Execution/ Scheduling: Processes/ Kernel Scheduler

4. Overview of Talk • History of YARN • Recent features • On going features • Future

5. Why YARN

6. Traditional Distributed Execution Engines Master Worker Task Task Worker Task Client Task Client Task Worker Task

7. MapReducev1 (MR1) Job Tracker Task Tracker Map Map Task Tracker Reduce Client Map Client Map Task Tracker Reduce JobTracker tracks every task in the cluster!

8. MR1 Utilization 4 GB Map 1024 MB Map 1024 MB Reduce 1024 MB Reduce 1024 MB Fixed-size slot model forces slots large enough for the biggest task!

9. Running multiple frameworks… Master Master Master Worker Worker Worker Task Task Task Task Task Task Worker Worker Worker Task Task Task Client Client Client Task Task Task Client Client Client Task Task Task Worker Worker Worker Task Task Task

10. YARN to the rescue! • Scalability: Track only applications, not all tasks. • Utilization: Allocate only as many resources as needed. • Multi-tenancy: Share resources between frameworks and users • Physical resources – memory, CPU, disk, network

11. YARN Architecture Resource Manager Node Manager App Master Container Node Manager Client Client Applications State Cluster State Node Manager

12. MR1 to YARN/MR2 functionality mapping • JobTracker is split into • ResourceManager– cluster-management, scheduling and application state handling • ApplicationMaster – Handle tasks (containers) per application (e.g. MR job) • JobHistoryServer – Serve MR history • TaskTracker maps to NodeManager

13. Early Features

14. Handing faults on Workers Resource Manager Node Manager App Master Node Manager App Master Container Client Client Applications State Node Manager Container Cluster State

15. Master Fault-tolerance - RM Recovery Resource Manager Node Manager App Master Container Applications State Node Manager App Master Container Cluster State Client Client RM Store

16. Master Node Fault toleranceHigh Availability (Active / Standby) Active Resource Manager App Master Node Manager Elector ZK RM Store Standby Resource Manager Elector Client Client Node Manager

17. Master Node Fault toleranceHigh Availability (Active / Standby) Standby Resource Manager Node Manager Elector ZK RM Store Active Resource Manager Elector Client Client App Master Node Manager

18. Scheduler • Inside ResourceManager • Decides who gets to run when and where • Uses “Queues” to describe organization needs • Applications are submitted to a queue • Two schedulers out of the box • Fair Scheduler • Capacity Scheduler

19. Fair Scheduler Hierarchical Queues Root Mem Capacity: 12 GB CPU Capacity: 24 cores Marketing Fair Share Mem: 4 GB Fair Share CPU: 8 cores R&D Fair Share Mem: 4 GB Fair Share CPU: 8 cores Sales Fair Share Mem: 4 GB Fair Share CPU: 8 cores Jim’s Team Fair Share Mem: 2 GB Fair Share CPU: 4 cores Bob’s Team Fair Share Mem: 2 GB Fair Share CPU: 4 cores

20. Fair Scheduler Queue Placement Policies <queuePlacementPolicy> <rule name="specified" /> <rule name="primaryGroup" create="false" /> <rule name="default" /> </queuePlacementPolicy>

21. Multi-Resource Scheduling • Node capacities expressed in both memory and CPU • Memory in MB and CPU in terms of vcores • Scheduler uses dominant resource for making decisions

22. Multi-Resource Scheduling 6 cores 50% cap. 12 GB 33% cap. 10 GB 28% cap. 3 cores 25% cap. Queue 1 Usage Queue 2 Usage

23. Multi-Resource Enforcement • YARN kills containers that use too much memory • CGroups for limiting CPU

24. Recently added features

25. RM recovery without losing work • Preserving running containers on RM restart • NM no longer kills containers on resync • AM made to register on resync with RM

26. RM recovery without losing work Resource Manager Node Manager Applications State Node Manager App Master Container Cluster State Client Client RM Store

27. NM Recovery without losing work • NM stores container and its associated state in a local store • On restart reconstruct state from store • Default implementation using LevelDB • Supports rolling restarts with no user impact

28. NM Recovery without losing work Resource Manager Node Manager App Master Container Client Client Applications State State Store Cluster State

29. Fair Scheduler Dynamic User Queues Root Mem Capacity: 12 GB CPU Capacity: 24 cores Marketing Fair Share Mem: 4 GB Fair Share CPU: 8 cores R&D Fair Share Mem: 4 GB Fair Share CPU: 8 cores Sales Fair Share Mem: 4 GB Fair Share CPU: 8 cores Moe Fair Share Mem: 4 GB Fair Share CPU: 8cores Larry Fair Share Mem: 2 GB Fair Share CPU: 4 cores Moe Fair Share Mem: 2 GB Fair Share CPU: 4cores

30. On going features

31. Long Running Apps on Secure Clusters (YARN-896) • Update tokens of running applications • Reset AM failure count to allow mulitple failures over a long time • Need to access logs while application is running • Need a way to show progress

32. Application Timeline Server (YARN-321, YARN-1530) • Currently we have a JobHistoryServer for MapReduce history • Generic history server • Gives information even while job is running

33. Application Timeline Server • Store and serve generic data like when containers ran, container logs • Apps post app-specific events • e.g. MapReduce Attempt Succeeded/Failed • Pluggable framework-specific UIs • Pluggable storage backend • Default LevelDB

34. Disk scheduling (YARN-2139 ) • Disk as a resource in addition to CPU and Memory • Expressed as virtual disk similar to vcore for cpu • Dominant resource fairness can handle this on the scheduling side • Use CGroups blkio controller for enforcement

35. Reservation-based Scheduling (YARN-1051)

36. Reservation-based Scheduling

37. FUTURE FEATUREs

38. Container Resizing (YARN-1197) • Change container’s resource allocation • Very useful for frameworks like Spark that schedule multiple tasks within a container • Follow same paths as for acquiring and releasing containers

39. Admin labels (YARN-796) • Admin tags nodes with labels (e.g. GPU) • Applications can include labels in container requests I want a GPU Application Master NodeManager [Windows] NodeManager [GPU, beefy]

40. Container Delegation (YARN-1488) • Problem: single process wants to run work on behalf of multiple users. • Want to count resources used against users that use them. • E.g. Impala or HDFS caching

41. Container Delegation (YARN-1488) • Solution: let apps “delegate” their containers to other containers on the same node. • Delegated container never runs • Framework container gets its resources • Framework container responsible for fairness within itself

42. Questions?

43. Thank You! • Anubhav Dhoot, Software Engineer, Cloudera • adhoot@cloudera.com