1 / 22

Resource Selection Services for a Single Job Execution

Resource Selection Services for a Single Job Execution. Soonwook Hwang National Institute of Informatics/NAREGI OGSA F2F RSS Session Sunnyvale, CA, US Aug 15, 2005. Job Manager. What are the services OGSA-RSS wants to define?. Candidate Set Generator (Work -Resource mapping).

redell
Download Presentation

Resource Selection Services for a Single Job Execution

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Resource Selection Services for a Single Job Execution Soonwook Hwang National Institute of Informatics/NAREGI OGSA F2F RSS Session Sunnyvale, CA, USAug 15, 2005

  2. Job Manager What are the services OGSA-RSS wants to define? Candidate Set Generator (Work -Resource mapping) RSS Services to be defined Provisioning Deployment Configuration Execution Planning Services Information Services Reservation Data Container Service Container Accounting Services

  3. What are the OGSA-EMS Problems? • According to the OGSA Architecture v1.0 documents • Finding execution candidate locations (CSG) • What are the locations at which a job can execution? • Selecting execution location (EPS) • Once where a job can execute is known, where should it actually run? • Preparing for execution (the focus of ACS and CDDLM-WG) • Deployment and configuration of binaries and libraries, staging data • Initiating and managing the execution (the focus of BES- WG)

  4. RSS Problems: Essential, but Challenging • Defining the interfaces and protocols of the OGSA Resource Selection Services (RSS) is an integral part of the OGSA-EMS architecture, but seems to be very challenging • Interactions with many other OGSA EMS components • Job Manger (JM), Information Service (IS), Service Container (SC) , Reservation Service (RS) , etc • Various job types • Single, atomic job • A set of jobs w/o dependency • e.g., embarrassingly-parallel applications • A set of jobs w/ dependency • e.g., workflows

  5. Let’s focus on a simple case to identify some interactions between RSS and other EMS Services • Focusing on resource selection services for the execution of a single, atomic job • A single job that exactly fits into the JSDL v1.0 spec • Not considering complex jobs such as workflows and embarrassingly parallel applications • JSDL v1.0 specification • Job identification requirements • Resource requirements • Data requirements

  6. Resource Selection Scenarios • JM sends to EPS a JSDL document with a job’s resource requirements written in it • EPS hands off the JSDL document to CSG • CSG creates a candidate set of resources, returning it to EPS • CSG looks up the Information Service to find resources that match the job’s resource requirements • EPS selects the best one among the candidate resources • EPS perhaps uses some selection policy to select the best one • JM submit the JSDL document to the se

  7. Concrete JSDL Abstract JSDL Interactions between RSS and other EMS Services • Abstract JSDL • Selection Policy User submits job as abstract JSDLdocument along with selection policy Job Manager EPS • Concrete JSDL • EPR to the SC • Concretized JSDL • Candidate EPRs CSG CSG mightinteract withSC directly Info Service Service Container SC publishes information about resource attributes

  8. Abstract and Concrete JSDL • Abstract JSDL • a JSDL document where some of elements necessary for initiating the execution of a job have yet to be filled • E.g., a JSDL document with no hostname specified in it • Concrete JSDL • a self-sufficient JSDL document for job execution

  9. Job Manager (JM) • Responsible for managing and controlling of jobs during their entire lifetime • Submits jobs described in a JSDL document to a Service Container which the EPR returned from EPS points to • uses the protocol and interfaces defined in the OGSA BES specification

  10. Information Service (IS) • A placeholder to keep attributes metadata about resources • Service Containers publish their resourceattributes along with their own EPRs to IS • Static information • Architecture, OS version, Hostname, etc • Dynamic information • CPU load, Queue length, etc

  11. Candidate Set Generator (CSG) • Performs a simple, dumb matchmaking with the IS to identify a list of container services that have resource capabilities that meet the job’s resource requirements • Concretizes the abstract input JSDL document using resource information retrieved from the IS • e.g., might fill in the “CandidateHost” element • Passes EPRs to candidate hosts and the more concretized JSDL document back to EPS • Discussion • Should CSG be dumb or intelligent? • E.g., Should selection policy be reflected in CSG as well when CSG locating candidate resources?

  12. Execution Planning Service (EPS) • Select the best one among the candidate resources sent from CSG • If needed (?), may want to interact with the IS to get additional resource information that could be used to more concretize the JSDL document returned from CSG • Uses the selection policy which is given as input to select the best one • Discussion • Is it a good idea to allow EPS to interact with IS as well? • What kinds of selection policies should be considered? • Where should selection policy be put, EPS, or CSG, or both?

  13. Selection Policy • Some user preference or priority to be allowed to put on each of resource requirements specified in the JSDL document • e.g., Condor ClassAD • Some objective function • e.g., minimum execution time, lowest cost, reservation, etc • Application-specific selection algorithm • Discussion • Is working on the Condor ClassAD-like user preference mechanism within this OGSA-RSS-WG as an example of selection policy worth trouble to do?

  14. Reservation Service (RS) • Will probably reside on local resources, directly interacting with local schedulers (e.g., maui, PBSPro) to get and manage reservations on locally reservable resources • Will probably provide a common interface • “MakeReservation” • “CancelReservation” • Maybe some advanced service container could act as the RS by providing such a common reservation interface

  15. Concrete JSDL • EPR to the SC (w/ Reservation) Concrete JSDL Abstract JSDL Reservation as Selection Policy Make Reservation from 21:00 – 23:00 August 15, 2005 • Abstract JSDL • Selection Policy User submits job as abstract JSDLdocument along with selection policy Job Manager EPS • Concretized JSDL • Candidate EPRs CSG RS Info Service Service Container SC publishes information about resource attributes

  16. Concrete JSDL • EPR to the SC (w/ Reservation) Concrete JSDL Abstract JSDL EPS makes reservations Make Reservation from 21:00 – 23:00 August 15, 2005 • Abstract JSDL • Selection Policy User submits job as abstract JSDLdocument along with selection policy Job Manager EPS • Concretized JSDL • Candidate EPRs EPS makes reservation(s) CSG RS Info Service Service Container SC publishes information about resource attributes

  17. Concrete JSDL • EPR to the SC (w/ Reservation) Concrete JSDL CSG makes reservations Make Reservation from 21:00 – 23:00 August 15, 2005 • Abstract JSDL • Selection Policy User submits job as abstract JSDLdocument along with selection policy Job Manager EPS • Concretized JSDL • Candidate EPRs • Abstract JSDL • Selection Policy EPS confirms reservation(s) (w/ Reservation) CSG Make Reservation from 21:00 – 23:00 August 15, 2005 CSG makes reservations RS Info Service Service Container

  18. Discussions on Reservation Service • Which one should interact with the RS, EPS, or CSG, or perhaps JM? • How about some kind of a global RS that interacts with the local RS to make reservation on behalf of either ESP or CSG? • How does the RS have an effect on the design of EPS and CSG?

  19. Summery of Discussion Topics • Should CSG perform a simple matchmaking or have some intelligence in locating available resources? • Is it a good idea to allow EPS to interact with IS as well e.g., to have it concretize the input JSDL document? Is there any use case for this? • What kinds of selection policy should be considered? • Where should selection policy be put, only EPS, or CSG as well? • Is working on the Condor ClassAD-like user preference mechanism as an example of selection policy worth trouble to do? • Which one should interact with the local RS, EPS or CSG? • Is it a good idea to regard Reservation as a selection policy? • How about some global reservation service? Do we need it? How does it affect the interface design of EPS and CSG? • In General, how the RS affect on the design of EPS and CSG? • Any other important issues to discuss together?

  20. Thank you

  21. What is the Architecture User submits jobas JSDL document Job Manager asksEPS where to run EPS commits thereservation(s) to be used Job Manager EPS Reservation Service CSG gets potentialatomic reservationsfrom reservationservice EPS asks CSG forcandidate executionplans (concretized JSDL) CSG CSG mightinspectservicecontainersdirectly CSG retrieves service info(info model not important) Info Service Service Container Service container publishes information about itself

  22. Concrete JSDL Abstract JSDL JM makes reservations • Abstract JSDL • Selection Policy User submits job as abstract JSDLdocument along with selection policy Job Manager EPS • Concrete JSDL • EPS to the SC • Concretized JSDL • Candidate EPRs JM makes reservations CSG RS Info Service Service Container SC publishes information about resource attributes

More Related