1 / 19

What’s New in Work Queue

What’s New in Work Queue. Michael Albrecht, University of Notre Dame CCL Workshop, June 2012. Overview. New “Batch Job” Systems Moab / Cluster MPI Queue Work Queue Enhancements Hierarchical Work Queue. Batch Job Systems. Abstraction Layer Library for generic task submission

michaelbsmith
Download Presentation

What’s New in Work Queue

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. What’s New in Work Queue Michael Albrecht, University of Notre Dame CCL Workshop, June 2012

  2. Overview New “Batch Job” Systems Moab / Cluster MPI Queue Work Queue Enhancements Hierarchical Work Queue

  3. Batch Job Systems Abstraction Layer Library for generic task submission Used by Makeflow

  4. Moab & other Clusters Similar to “SGE” Explicitly support Moab scheduler Support other, similar schedulers Set name, submit and remove commands moab

  5. MPI Queue Workflow W W W W W W W W W W W W W W

  6. MPI Queue Workflow W W W W F W W W W W

  7. MPI Queue Enables arbitrary computation on “MPI-only” clusters Ranks 1-N talk to Rank 0, which acts as a “foreman” Assumes shared parallel filesystem for cluster API very similar to Work Queue Fully supported by Batch Job/Makeflow

  8. Work Queue

  9. Work Queue is Wonderful Easily harness 100’s-1000’s of cores Combine multiple resources for one project Dynamically scale computational resources Private Cluster Shared SGE Cluster Makefile Makeflow Campus Condor Pool Public Cloud Provider Local Files and Programs sge_submit_workers Wv W Hundreds of Workers in a Personal Cloud W W submit tasks W W W W W W W W W ssh condor_submit_workers

  10. Work Queue has Limits Bandwidth, file size, and computation length constrain potential number of workers

  11. Work Queue has Limits Natural parallelism of a workflow leaves extra workers idle T1 T2 T3 T4 T0 ??? W1 W2 W3 W4 W5 T5

  12. Work Queue has Limits Synchronous transfer leaves network resources idle, increases dispatch time

  13. Add more indirection! “All problems in computer science can be solved by another layer of indirection” -David Wheeler “…except for the problem of too many layers of indirection” Kevlin Henney

  14. Hierarchical Work Queue Master (Makeflow) Worker Worker Worker Worker Worker Worker Worker Worker

  15. Hierarchical Work Queue Master (Makeflow) Foreman Foreman Worker Worker Worker Worker Worker Worker Worker Worker

  16. Hierarchical Work Queue Master (Makeflow) Shared FS Foreman Foreman Worker Worker Worker Worker Worker Worker Worker Worker

  17. Hierarchical Work Queue Master (Makeflow) Shared FS Shared FS Foreman Foreman Worker Worker Worker Worker Worker Worker Worker Worker

  18. Hierarchical Work Queue Master (Makeflow) Shared FS Foreman Foreman Shared FS Shared FS Foreman Foreman Worker Worker Worker Worker Worker Worker Worker Worker

  19. Hierarchical Work Queue Coming Soon! http://www.nd.edu/~ccl 19

More Related