1 / 12

Queue Scheduling and Advance Reservations with COSY

Queue Scheduling and Advance Reservations with COSY. Junwei Cao Falk Zimmermann C&C Research Laboratories NEC Europe Ltd. http://www.ccrl-nece.de/. Outline. Related Work COSY Structure COSY Job Scheduling Scheduling with Advance Reservations Performance Evaluation Experimental Results

balin
Download Presentation

Queue Scheduling and Advance Reservations with COSY

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Queue Scheduling and Advance Reservations with COSY Junwei Cao Falk Zimmermann C&C Research Laboratories NEC Europe Ltd. http://www.ccrl-nece.de/ IPDPS 2004

  2. Outline • Related Work • COSY Structure • COSY Job Scheduling • Scheduling with Advance Reservations • Performance Evaluation • Experimental Results • Ongoing Work IPDPS 2004

  3. Related Work • Job Scheduling Systems • Catalina, Condor, EASY, LoadLeveler, LSF, Maui, PBS, SGE, STORM, Titan … • Scheduling with Advance Reservations • W. Smith, I. Foster, and V. Taylor, “Scheduling with Advanced Reservations”, IPDPS 2000. • Q. Snell, M. Clement, D. Jackson, and C. Gregory, “The Performance Impact of Advance Reservation Metascheduling”, JSSPP 2000. IPDPS 2004

  4. Users / Applications Generic User Interfaces (submit, release, queue, confirm,…) Specific User Interfaces (mpirun …) Job Scheduling Log Management Configurations Resource Monitoring Database Management Job Execution Operating Systems, MPI Environments (NEC, SCore …), … Plugin Scripts Resources COSY Structure IPDPS 2004

  5. COSY Job Scheduling • Queue Scheduling: • First-come-first-served algorithm • Aggressive backfilling • Multiple queue support • Queue and user priority • Resource limitation • Access control list • Batch and interactive support • Advance Reservations: • Exact start time • Latest end time • Two phase commitment IPDPS 2004

  6. Scheduling with Advance Reservations • Mean queue wait time increases • Resource utilization decreases • Schedulable with limit percentage of advance reservations IPDPS 2004

  7. Challenges … • How to prevent advance reservations from taking start time advantages over queued jobs? • How to satisfy advance reservations as far as possible without sacrificing queue efficiency? • Additional scheduling policies have to be defined (in academic/research environments) … IPDPS 2004

  8. COSY Strategies • Defining mandatory shortest notice time for advance reservations • Using current mean wait time of queued jobs • Using times of current mean wait time of queued jobs IPDPS 2004

  9. Performance Evaluation • Performance Metrics • Queue wait time • Resource utilization • Rejection rate • Workload Modeling • Cirne comprehensive model (ANL arrival pattern) • 100 requests -> 32 node cluster • Time reduced by 180 • Experimental Design • COSY running with job execution disabled • Using different percentages of advance reservations • Using different shortest notice time of advance reservations IPDPS 2004

  10. Experimental Results • Queue wait time • Resource utilization IPDPS 2004

  11. Solutions … • Policies applied in COSY • COSY takes the current mean wait time of queued jobs as the mandatory AR shortest notice time so that ARs cannot take start time advantages. • The mandatory AR shortest notice time is applied in COSY as 1 time of current mean wait time of queued jobs and increases to 4 linearly as the AR percentage increases from 0% (not inclusive) to 15% so that ARs can be satisfied as far as possible without increasing the queue wait time. • When the AR percentage is more than 15%, COSY will stop accepting ARs temporarily in order to guarantee a proper queue scheduling. • Summary • Advantage: adaptive to queue workload. • Disadvantage: additional calculation of statistics. IPDPS 2004

  12. Ongoing Work • COSY performance evaluation in a larger scale and a longer time period • Integration of COSY in GEMSS grid infrastructure to support QoS negotiation • Extension of COSY to support standard languages and protocols (e.g. GRAAP and JSDL) • COSY-G: job scheduling and workload balancing in grid environments IPDPS 2004

More Related