WS-Based Advance Reservation and Co-allocation Architecture Proposal T.Ferrari, E.Ronchieri JRA1

1. WS-Based Advance Reservation and Co-allocation ArchitectureProposalT.Ferrari, E.RonchieriJRA1 JRA1 all-hands meeting, June 29 2004 www.eu-egee.org EGEE is a project funded by the European Union under contract IST-2003-508833

2. Contents • Use Cases • DataGrid WP1 legacy architecture • GGF (Grid Resource Allocation and Agreement Protocol) WS-Agreement Specification • Web services-based proposal

3. Use Cases • NETWORK: data replication. • Data replication: guaranteed, dedicated bandwidth • to optimize the network performance of a data transfer session (that otherwise would compete with other streams and would get and amount of bandwidth that could greatly vary over time) • to support file transfer with deadline (to synchronize job execution with input file transfer) • COMPUTING: • to reserve resources computing resources (eg. worker nodes) in presence of a large number of other competing jobs • STORAGE: • to be guaranteed that after the computation phase, a sufficient amount of space is present in a “close SE” to save the output data

4. The EDG legacy architecture

5. The EDG WP1 legacy architecture 4 Submits reservation request Creation of reservation 1

6. The EDG WP1 legacy architecture 5 Starts the discovery phase contacting RB, which returns an ordered list of suitable resources Creation of reservation 1

7. The EDG WP1 legacy architecture 6 RA iterates through the list representing suitable resources, and contacts the correspondent RM, until it succeeds Creation of reservation 1

8. GGF GRAAP (Grid Resource Allocation and Agreement Protocol) WS-Agreement Conceputal Layered Service Model Reservation Agent Resource Manager

9. GGF GRAAP (Grid Resource Allocation and Agreement Protocol) • WS-Agreement defines a language and a protocol for Advertising the capabilities of providers Creating agreements based on creational offers Monitoring agreement compliance • Agreement Layer (work in progress): Provides a web service based interface to be used to represent and monitor agreements with respect to provisioning of services implemented in the service layer • Service Layer (out of the GRAAP scope): Is an application-specific layer of a provided service The interface to this layer is domain-specific May or may not be exposed as a web service interface 1

10. Name Context Terms GRAAP Agreement structure Name: optional identificator Context: participants’ names, lifetime, links to other agreements related to this (co-allocation) Terms: Service Description Terms: - provides information needed to instantiate or identify a service to which this agreement pertains - describes the functionality that will be delivered under an agreement Guaranteed Terms: specify the service level that the parties are agreeing to Terms have to extended for specific usage domains. Agreement Service Description Terms Guarantee Terms

11. Agreement status • SATISFIED • VIOLATED at least one term not respected by service provider • INACTIVE terms not guaranted • ACTIVE mechanism to guarantee the terms in place and running • OBSERVED all terms agreed • CONSIDERED at least one term under negotiation

12. WS-based proposal user Workload Manager Logging & Bookkeeping UI MM resource ID list Co-allocation Agreement Provider CE ID list Network Resource ID list SE ID list Computing Agreement Provider Network Agreement Provider Storage Agreement Provider SE Acceptance NE Acceptance CE Acceptance SE Service Provider CE Service Provider NE Service Provider monitor monitor monitor Web service 1

13. Co-allocation agreement provider Logging & Bookkeeping Co-allocation Agreement Provider - (single reservation) passes the resource ID list to the specific agreement provider - Supports logic for management of co-allocation • Provisioning: in case of concurent allocations • Status: in case of failure of one or more reservations - Provides status information about co-allocations - Returns the co-allocation handle Co-allocation Agreement Provider CE ID list Computing Agreement Provider CE Acceptance CE Service Provider monitor 1

14. Agreement provider Logging & Bookkeeping Agreement Provider • - for a list of Resource IDs, it contacts the corresponding service provider and verifies the actual possibility to reserve the service via CA/NE/SE Acceptance • - identifies the agreements through a handle • - provides information about reservation status • - supports protocols to manage the case of a service that is the composition of services independently administered (such as in the case of a network path crossing multiple network administrative domains) • - translates the high-level agreement terms specified by the user to a quantitative expression that is • understood by the Service Providers Co-allocation Agreement Provider CE ID list Computing Agreement Provider CE Acceptance CE Service Provider monitor 1

15. Acceptance and Service provider Acceptance • - controls the access to a given resource instance • authentication and authorization • checks the agreement context (eg. the type of service requested to • address the right Service Provider if multiple options exist) • Service Provider • - More than one Service Provider per resource instance possible (for some type of resource such as the network) • - It determines if an agreement request can be satisfied (by checking the a slot table DB) • If so, it returns an agreement handle • the monitor provides information about the status of a given active agreement Co-allocation Agreement Provider CE ID list Network Agreement Provider NE Acceptance_1 NE Acceptance_n NE Acceptance_2 NE Service Provider_1 NE Service Provider_2 NE Service Provider_n NE Service Provider_3 monitor monitor monitor monitor

16. JC MON CE Service Provider LRMS Monitoring Two types of Status information needed: • the agreement status -> provided by the Agreement provider • the amount of reserved resources actually used at a given time -> provided by MON Examples of consumers of monitoringinformation: • the end-user: information directly from the LB • different solution: direct query of the Agreement • Jobs: they need to be informed when an Agreement status changes from INACTIVE to ACTIVE. In this case, a daemon should run on the WN to periodically check the status of a given Agreement. Logging & Bookkeeping CE Acceptance

17. Matchmaking • The EDG library has to be extended in order to: • for each job submission, make use of existing related reservation handles • support reservation and co-allocation resource discovery • optimize the resource discovery phase with specific policies in case of co-allocation: Examples: • CE and network reservation, outputSE known • Find CEs close to outputSE • For each couple (outputSE, CE_i), find network path • CE, SE and network reservation • Find SEs that support reservation • For each SE_i, find suitable CEs that support reservation • For each couple (SE_i, CE_j), find network path

18. Use case 1: job with reserved CE • User needs to specify the agreement identifier (ID) associated to the submitted job • Once the job is passed to WM, before proceeding with the execution the job, WM needs to verify the status of the Agreement by querying the the LB: WM gets Agreement ID status from LB; If (not OBSERVED) WM returns an error; else * PUSH mode If (ACTIVE and SATISFIED) WM gets the CE ID associated to the agreement ID from the LB; WM submits the job on CE ID; else hold job in task queue until Agreement status = ACTIVE; * PULL mode WM puts job in TQ; when Agreement status = ACTIVE CE ID gets job associated to relevant agreement ID from TQ • NOTE: condor_glidein could be used at the CE Service Provider Level

19. Use case 2: Job with reserved SE • Submission of job with reserved SE: • USER specifies agreement ID in the JDL • WM queries LB to determine the corresponding SE_ID • MM selects the CE Ids close to SE_ID • Case 1: user wants to replicate some output files automatically • JDL contains OutputData • A deamon in job wrapper (WN) checks the agreement status • when ACTIVE • job wrapper transfers output to SE_ID • Case 2: user wants to replicate some output files • JDL contains OutputSE • After the production of the output files, the job waits until the agreement ID status is ACTIVE • then the job transfers files