1 / 20

YARN

Nagarjuna K. YARN. Why Next Generation MR. Reliability Availability Scalability - Clusters of 10,000 machines and 200,000 cores, and beyond. Backward (and Forward) Compatibility Ensure customers’ MapReduce applications run unchanged in the next version of the framework.

brock
Download Presentation

YARN

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Nagarjuna K YARN nagarjuna@outlook.com

  2. Why Next Generation MR • Reliability • Availability • Scalability - Clusters of 10,000 machines and 200,000 cores, and beyond. • Backward (and Forward) Compatibility • Ensure customers’ MapReduce applications run unchanged in the next version of the framework. • Evolution – Ability for customers to control upgrades to the Hadoop software stack. • Predictable Latency – A major customer concern. • Cluster utilization nagarjuna@outlook.com

  3. Why Next Generation MR • Secondary Requirements • Support for alternate programming paradigms to MapReduce. • Support for short-lived services nagarjuna@outlook.com

  4. ReArchitecure • Need • Separate the tasks of Job Tracker • Resource management • Job Scheduling / Management nagarjuna@outlook.com

  5. So, What did we come up with nagarjuna@outlook.com Resource Manager Node Manager Application Master Container

  6. Resource Manager (RM) Manages the global assignment of compute resources to applications. nagarjuna@outlook.com

  7. Resource Manager (RM) • A pure Scheduler • No monitoring, tracking status of application • No guarantee on restarting failed tasks. nagarjuna@outlook.com

  8. Resource Manager (RM) • Each client/application may request multiple resources • Memory • Network • Cpu • Disk .. • This is a significant change from static Mapper / Reducer model nagarjuna@outlook.com

  9. Application Master • A per – applicationApplicationMaster(AM) that manages the application’s life cycle(scheduling and coordination). •  An application is either a single job in the classic MapReduce jobs or a DAG of such jobs. nagarjuna@outlook.com

  10. Application Master A per – applicationApplicationMaster(AM) that manages the application’s life cycle. nagarjuna@outlook.com

  11. Application Master • Application Master has the responsibility of • negotiating appropriate resource containers from the Scheduler • launching tasks • tracking their status • monitoring for progress • handling task-failures. nagarjuna@outlook.com

  12. Node Manager • The NodeManager is the per-machine framework agent • responsible for launching the applications’ containers, monitoring their resource usage (cpu, memory, disk, network) and reporting the same to the Scheduler. nagarjuna@outlook.com

  13. Gain with New Architecture • Scalability • Availability • Wire-compatibility • Innovation & Agility • Cluster Utilization • Support for programming paradigms other than MapReduce nagarjuna@outlook.com

  14. Gain with New Architecture Scalability Availability Wire-compatibility Innovation & Agility Cluster Utilization Support for programming paradigms other than MapReduce • RM and Job manager segregated • The Hadoop MapReduceJobTracker spends a very significant portion of time and effort managing the life cycle of applications nagarjuna@outlook.com

  15. Gain with New Architecture Scalability Availability Wire-compatibility Innovation & Agility Cluster Utilization Support for programming paradigms other than MapReduce • ResourceManage • Uses ZooKeeper for fail-over. • When primary fails, secondary can quickly start using the state stored in ZK • Application Master • MapReduceNextGen supports application specific checkpoint capabilities for the ApplicationMaster. • MapReduceApplicationMaster can recover from failures by restoring itself from state saved in HDFS. nagarjuna@outlook.com

  16. Gain with New Architecture Scalability Availability Wire-compatibility Innovation & Agility Cluster Utilization Support for programming paradigms other than MapReduce • MapReduceNextGen uses wire-compatible protocols to allow different versions of servers and clients to communicate with each other. • Rolling upgrades for the cluster in future. nagarjuna@outlook.com

  17. Gain with New Architecture Scalability Availability Wire-compatibility Innovation&Agility Cluster Utilization Support for programming paradigms other than MapReduce • New framework is generic. • Can came up with non MR parallel computing techniques • Different versions of MR running in parallel  • End users can upgrade to MR versions on their own schedule nagarjuna@outlook.com

  18. Gain with New Architecture Scalability Availability Wire-compatibility Innovation & Agility Cluster Utilization Support for programming paradigms other than MapReduce • MRv2 uses a general concept of a resource for scheduling and allocating to individual applications. • Container , can be a mapper or a reducer or … ? • Stubborn notion of Mapper,Reducer abolished • Better cluster utilization nagarjuna@outlook.com

  19. Gain with New Architecture Scalability Availability Wire-compatibility Innovation & Agility Cluster Utilization Support for programming paradigms other than MapReduce • Any Doubts ? nagarjuna@outlook.com

  20. Bibilography • http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/ nagarjuna@outlook.com

More Related