1 / 27

Can’t We All Just Get Along?

Can’t We All Just Get Along?. Spark and Resource Management on Hadoop. Introductions. Software engineer at Cloudera MapReduce, YARN, Resource management Hadoop committer. Introduction. Spark as a first class data processing framework alongside MR and Impala Resource management

indira-cote
Download Presentation

Can’t We All Just Get Along?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Can’t We All Just Get Along? Spark and Resource Management on Hadoop

  2. Introductions • Software engineer at Cloudera • MapReduce, YARN, Resource management • Hadoop committer

  3. Introduction • Spark as a first class data processing framework alongside MR and Impala • Resource management • What we have already • What we need for the future

  4. Bringing Computation to the Data • Users want to • ETL a dataset with Pig and MapReduce • Fit a model to it with Spark • Have BI tools query it with Impala • Same set of machines that hold data must also host these frameworks

  5. Cluster Resource Management • Hadoop brings generalized computation to big data • More processing frameworks • MapReduce, Impala, Spark • Some workloads are more important than others • A cluster has finite resources • Limited CPU, memory, disk and network bandwidth • How do we make sure each workload gets the resources it deserves?

  6. How We See It Impala MapReduce Spark HDFS

  7. How They Want to See It Engineering - 50% Marketing - 20% Finance - 30% Spark Spark Spark MR MR MR Impala Impala Impala HDFS

  8. Central Resource Management Impala MapReduce Spark YARN HDFS

  9. YARN • Resource manager and scheduler for Hadoop • “Container” is a process scheduled on the cluster with a resource allocation (amount MB, # cores) • Each container belongs to an “Application”

  10. YARN Application Masters • Each YARN app has an “Application Master” (AM) process running on the cluster • AM responsible for requesting containers from YARN • AM creation latency is much higher than resource acquisition

  11. YARN Client JobHistoryServer ResourceManager NodeManager NodeManager Container Container Container Application Master Map Task Reduce Task

  12. YARN Queues • Cluster resources allocated to “queues” • Each application belongs to a queue • Queues may contain subqueues Root Mem Capacity: 12 GB CPU Capacity: 24 cores Marketing Fair Share Mem: 4 GB Fair Share CPU: 8 cores R&D Fair Share Mem: 4 GB Fair Share CPU: 8 cores Sales Fair Share Mem: 4 GB Fair Share CPU: 8 cores Jim’s Team Fair Share Mem: 2 GB Fair Share CPU: 4 cores Bob’s Team Fair Share Mem: 2 GB Fair Share CPU: 4 cores

  13. YARN app models • Application master (AM) per job • Most simple for batch • Used by MapReduce • Application master per session • Runs multiple jobs on behalf of the same user • Recently added in Tez • AM as permanent service • Always on, waits around for jobs to come in • Used for Impala

  14. Spark Usage Modes

  15. Spark on YARN • Developed at Yahoo • Application Master per SparkContext • Container per Spark executor • Currently useful for Spark Batch jobs • Requests all resources up front

  16. Enhancing Spark on YARN • Long-lived sessions • Multiple Jobs • Multiple Users

  17. Long-Lived Goals • Hang on to few resources when we’re not running work • Use lots of the cluster (over fair share) when it’s not being used by others • Give back resources gracefully when preempted • Get resources quickly when we need them

  18. Mesos Fine-Grained Mode • Allocate static chunks of memory at Spark app start time • Schedule CPU dynamically when running tasks

  19. Long-Lived Approach • A YARN application master per Spark application (SparkContext) • Which is to say an application master per session • One executor per application per node • One YARN container per executor • Executors can acquire and give back resources

  20. Long-Lived: YARN work • YARN-896 - long lived YARN • YARN not built with apps that would stick around indefinitely • Miscellaneous work like renewable container tokens • YARN-1197 - resizable containers

  21. Long-Lived: Spark Work • YARN fine-grained mode • Changes to support adjusting resources in Spark AM • Memory?

  22. The Memory Problem • We want to be able to have memory allocations preempted and keep running • RDDs stored in JVM memory • JVMs don’t give back memory

  23. The Memory Solutions • Rewrite Spark in C++ • Off-heap cache • Hold RDDs in executor processes in off-heap byte buffers • These can be freed and returned to the OS • Tachyon • Executor processes don’t hold RDDs • Store data in Tachyon • Punts off-heap problem to Tachyon • Has other advantages, like not losing data when executor crashes

  24. Multiple User Challenges • A single Spark application wants to run work on behalf of multiple parties • Applications are typically billed to a single queue • We’d want to bill jobs to different queues Spark App Rajat from Marketing Cluster Sylvia from Finance

  25. Multiple Users with Spark Fair Scheduler • Full-features Fair Scheduler within a Spark Application • Two level scheduling • Difficult to share dynamically between Spark and other frameworks

  26. Multiple Users with Impala • Impala has same exact problem • Solution: Llama (Low Latency Application MAster) • Adapter between YARN and Impala • Runs multiple AMs in a single process • Submits resource requests on behalf of relevant AM • Jobs billed to the YARN queues they belong in Spark App Rajat from Marketing Cluster AM for Marketing Queue Sylvia from Finance AM for Finance Queue

  27. Spark Other Hadoop processing frameworks

More Related