1 / 33

CPU scheduling and enforcement

CPU scheduling and enforcement. (screenshots of live demo). Andrew Ferguson July 10, 2012. Cores in the web GUI. Cores in the web GUI. Cores in the web GUI. Unfair sharing. S tart four threads spinning. Unfair sharing. S tart four threads spinning. Then, a second 16 threads spinning.

luisa
Download Presentation

CPU scheduling and enforcement

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CPU scheduling and enforcement (screenshots of live demo) Andrew Ferguson July 10, 2012

  2. Cores in the web GUI

  3. Cores in the web GUI

  4. Cores in the web GUI

  5. Unfair sharing Start four threads spinning

  6. Unfair sharing Start four threads spinning Then, a second 16 threads spinning

  7. Unfair sharing Start four threads spinning Second process gets more CPU Then, a second 16 threads spinning

  8. Let’s use Kitten!

  9. Let’s use Kitten! Same programs as before

  10. Let’s use Kitten! Same programs as before Added: CPU cores request

  11. Playing with YARN Launch one application in YARN

  12. Playing with YARN Launch one application in YARN Gets all cores, even though requested only one

  13. Playing with YARN Launch second application, again with more threads

  14. Playing with YARN Now CPU is shared equally! Launch second application, again with more threads

  15. Playing with YARN

  16. Playing with YARN

  17. Request all the cores two cores per container x two containers = 4 cores

  18. Request all the cores First application is running

  19. Request all the cores First application is running Second application Is waiting

  20. Request all the cores First application is running First application has all the cores Second application Is waiting

  21. Request all the cores

  22. How does this work?

  23. Pieces of YARN Node Manager Resource Manager Node Manager Application Master Node Manager

  24. Pieces of YARN Node Manager Resource Manager Node Manager Application Master Node Manager

  25. Pieces of YARN Node Manager Resource Manager Node Manager Application Master Node Manager

  26. Node Manager Internals Node Manager Launched by ContainerExecutor

  27. Node Manager Internals Node Manager Monitors memory usage, and kills tasks over limit Reports killed or failed tasks back to ResourceManager Deletes temporary files created by executor Containers Monitor Launched by ContainerExecutor

  28. Node Manager Internals Node Manager Monitors memory usage, and kills tasks over limit Reports killed or failed tasks back to ResourceManager Deletes temporary files created by executor Containers Monitor Launched by ContainerExecutor Made pluggable, like executor, in MAPREDUCE-4351

  29. Node Manager Internals CgroupsCM in MAPREDUCE-4334 Node Manager Creates Cgroup for container, places processes in it Reports killed or failed tasks back to ResourceManager Deletes temporary files created by executor Deletes Cgroup when finished Containers Monitor Launched by ContainerExecutor

  30. Cgroups hierarchy Controller cpu memory (find by reading mount table) $ sudo mount -t cgroup -o memory none /cgroups/mem

  31. Cgroups hierarchy Controller cpu memory (find by reading mount table) . . . hadoop-yarn . . . hadoop-yarn Top of Hierarchy yarn.nodemanager. cgroups.path $ sudocgcreate -a hadoop_user_name -g memory:hadoop-yarn

  32. Cgroups hierarchy Controller cpu memory (find by reading mount table) . . . hadoop-yarn . . . hadoop-yarn Top of Hierarchy yarn.nodemanager. cgroups.path . . . . . . container_..._001 container_..._002 container_..._003 container_..._001 container_..._002 container_..._003

  33. Container Cgroup CPU process from before Isolated CPU cgroup

More Related