Resource Brokering on Complex Grids EUROGRID and GRIP

1. Presented by John Brooke ESNW October 3/4 UK/Japan N+N Resource Brokering on Complex Grids EUROGRID and GRIP

2. Contents Aims of the resource broker Functionality of the ancestral broker EUROGRID broker Interoperability architecture (UNICORE-Globus) Towards a resource description ontology Relation to the GGF and OGSA January 17, 2003

3. Brokers as VOs Users VirtualOrganisationBrokers OrganizationFirewalls SystemBrokers ComputeResources

4. Resource Client Broker Service Broker Service Resource Broker Service Client Resource Broker Service Client Broker Service Resource Broker Service Client Broker Service Replication Resource Client Broker Service Resource VO Layer Specialist Layer Site Layer Federated Brokering

5. Interoperability for Brokering We want to broker on Grids controlled by either UNICORE or Globus. In GRIP we developed two methods Bifurcation, separate “sub-brokers” for a Globus or a Unicore Grid. This is achieved and is extensible to a limited extent. Constructing an extendable resource broker utilising a Grid Resource Ontology to handle mappings of resource terms.

6. Execution NJS 2 CheckQoS Broker NJS 1 CheckQoS 3 CheckQoS_Outcome 2 CheckQoS User 4 CheckQoS_Outcome Execution NJS 3 CheckQoS_Outcome Ancestral EUROGRID Broker The API allows two levels of operation: Resource Checking: Static requirements, capability and capacity. QoS Checking: Performance vs cost. Tickets can be issued as a “guarantee”. Protocol can be used symmetrically by Broker. 1

7. Bezier SGI Onyx @ Manchester Vtk + VizServer UNICORE Gateway and NJS Manchester Laptop SHU Conference Centre UNICORE Gateway and NJS QMUL Dirac SGI Onyx @ QMUL LB3D with RealityGrid Steering API Brokering for Workflows Firewall SGI OpenGL VizServer Simulation Data • VizServer client • Steering GUI • The Mind Electric GLUE web service hosting environment with OGSA extensions • Single sign-on using UK e-Science digital certificates Steering (XML)

8. Interoperable Broker – Method 1 The Network Job Supervisor (NJS) delegates the Resource Check to the Broker at the Vsite. The UNICORE brokering track utilises the IDB exactly as for the ancestral broker. The Globus track uses a translator of the QoS check object. The translation service is extendable. The results of the translation are used to drive the LDAP search and the Globus broker then utilises MDS to perform this. UNICORE NJS 4.0 gave much greater power and flexibility in brokering for complex workflows.

9. NJS Broker Unicore Broker Globus Broker Translator Filter IDB MDS(GRIIS/GRIS) Basic Translator Architecture – Method 1 Diagram Of Broker Architecture Delegates resource check Delegates translation Uses to drive LDAP search Lookup resources Performs LDAP search

10. Pros and Cons • A nice feature of Method 1 is that no alteration needs to be made to the client side of UNICORE, thus no alteration for application plugins or “expert” brokers • Also no alterations need to be made to Globus. • However the UNICORE description of Grid resources is very different from the MDS-2 description. MDS-2 does not publish software resource and user environment, Unicore does not check dynamic resource, e.g. machine loading. • The need for resource description translation is thus highlighted.

11. NJS Resource Discovery Service Broker Unicore Broker Globus Broker Translator Filter IDB Ontology engine Filter Resource Discovery Service Architecture – Method 2 Diagram Of Broker Architecture Delegates resource check Other Brokers Delegates translation Uses to drive MDS search Lookup resources Uses to Drive MDS Search Hierarchical Grid Search Hierarchical Grid Search

12. Ontologies • Defines knowledge domain and allows reasoning on this domain. • If we can create a Grid Resource Ontology, creation of specialist translation classes from basic Grid translator becomes possible. • IDB at sites can be created via ontology, it contains site specific information which the clients job specification cannot do. • So brokers take client request formulated in RR space, at each site use translator to convert to RR space, offers come back with capability and QoS.

13. Client Client Gateway Gateway Broker NJS Broker R-GMA NJS NJS NJS IDB TSI/Host GT3 Host Host Host Local Brokering Configurations Site-Wide Brokering Normal EUROGRID/GRIP Brokering

14. Persistent Virtual Environments Clients Other Brokers Banking Services MetaschedulingService Broker Site Feedback Policy Manager Chargeable Schedulable GridServices Workflow Manager Resource Usage Monitor Brokering and OGSA Services

15. Relevant GGF work • Grid Protocol Architecture-RG : Core Grid Functions • Grid Resource Allocation Agreement Protocol-WG :advanced reservation, co-allocation • CIM-based schema-WG : successor to LDAP • GESA-WG looking at economic issues of scheduling The recently-formed Semantic Grid RG is very interested in the Grid Resource Ontology idea. 1

16. Points for Discussion • What is the relationship between brokering and scheduling? • How to deal with legacy (not Grid-aware) schedulers? • How to relate the ontologies from the application side (Resource Requestor) to the service provision side (Resource Provider)? • How does a broker estimate upper and lower bounds for turnaround time? • How far does the broker trust information from the service provider. Should it monitor running workflows? 1

17. request RR space RP space request B A sync RP space RR space RP space Request referral C D Figure 1: Request from RR space at A mapped into resource providers at B and C, with C forwarding a request formulated in RR space to RP space at D. B and C synchronize at end of workflow before results returned to the initiator A. RR and RP Spaces January 17, 2003 GRIP First Project Review 1