Resource Brokering in the PROGRESS Project

1. Resource Brokering in the PROGRESS Project Juliusz Pukacki Grid Resource Management Workshop, 21-22 October 2003

2. Progress Project • Duration: December 2001 – May 2003 • Deployment: June 2003-December 2003 • Project Partners • SUN Microsystems Poland • PSNC IBCh Poznań • Cyfronet AMM, Kraków • Technical University Łódź • Co-funded by The State Committee for Scientific Research (KBN) and SUN Microsystems Poland

3. PROGRESS Goals • Provide computational portal for scientists • Provide easy to use user interface • Provide Grid middleware infrastructure • Make use of distributed cluster of SUN SF6800 is a pilot testbed installation • Make use of PIONIER network infrastructure

4. PROGRESS Testbed • Cluster of 80 processors • Networked Storage of 1,3 TB • Software: ORACLE, HPC Cluster Tools, Sun ONE, Sun Grid Engine

5. PORTAL PORTAL GRID SERVICE PROVIDER GRID MANAGEMENT SYSTEM GRID MANAGEMENT SYSTEM HPC RESOURCES HPC RESOURCES 4-tier new grid-portal environment 3-tier classical grid-portal environment Architecture - overview

6. Grid Broker - goals • Provide universal mechanism of resource management for Grid environment • Make use of common lower level Grid infrastructure (Globus) • Service oriented approach • Well-defined universal interface, implemented in common technology (Web Services) • Modular architecture

7. Grid Broker - functionality • Ability to choose the best resource for the job execution, according to Job Description and chosen mapping algorithm; • Ability to submit the GRMS Task according to provided Job Description; • Ability to submit a set of tasks with precedence constraints (sequence/parallel structure) • Provides information about the Task status; • Provides other information about Tasks (name of host where the Task is/was running, start time, finish time);

8. Grid Broker – functionality (2) • Provides list of resources that fullfills requirements provided in Job Description); • Provides a list of Tasks submitted by given user; • Ability to transfer input and output files (GridFTP, GASS, Data Management System); • Ability to cancel the Task; • Provides notification mechanism

9. Grid Broker - overview Progress Grid Middleware User Access Layer Grid Infrastructure Information Service • MDS • GRAM • GridFTP • GASS Broker PROGRESS Portal Data ManagementSystem

10. Grid Broker - details DB Resource Discovery PROGRESS Services Web Service Interface Workflow Mngmt. PROGRESS Portal Broker System Layer Services Task Registry Task Manager

11. Grid Broker - modules • Web Service Interface • Provides GSI enabled web service interface for Clients (PROGRESS Portal) • Supports callback communication to Client (notification service) • Workflow Management Module • Creates workflow graph of tasks from Job Description • Puts tasks to database • Controls of tasks execution according to precedence constraints

12. Grid Broker – modules (2) • Broker Module • Manages process of task submittion • Chooses the best resources for task execution • Transfers executable to chosen resource • Transfers input and output files • Resource Discovery Module • Finds resources that fullfills requirements described in Job Description • Provides information about resources, required for task scheduling

13. Grid Broker – modules (3) • Task Manager Module • Ability to check current status of task • Ability to cancel running task • Monitors for status changes of running task • Task Registry • Stores information about Task execution (start time, finish time, machine where executed, current status, Job Description)

14. Job Description - XRSL • Provides ability to express precedence constraints of Task • Grid Broker’s Job contains one or more Tasks • Tasks are grouped into sequence or parallel execution

15. Job Description - schema

16. Task2a Task2b Task2a Task2b Task3c Task2c Parallel/sequence structure Job3 parallel Job1 single Job2 sequence Task1 Task1 Task1 Task2 Task3 Task2 Task3

17. Job Description - Task • Task executable • file location • arguments • file argument (files which have to be present in working directory of running executable) • environment variables • standard input • standard output • standard error

18. Job Description - Task • Resource requirements • name of host for job execution • operating system • required local resource management system • minimum memory required • minimum number of cpus required • minimum speed of cpu • other parameter passed directly to Globus GRAM

19. XRSL - example <?xml version="1.0" encoding="UTF-8"?> <job id="Application"> <user dn="dc=pl, dc=poznan, dc=man, uid=pukacki"/> <task id="Task1"> <executable name="computation" dmsid="12345" type="single" count="1"> <arguments> <value>4</value> <file id="772" type="in">mydata.dat</file> </arguments> <environment> <stdin dmsid ="772"></stdin> <stdout dmsid="774"/> <stderr dmsid="775"/> <variable name="CONST">5</variable> </environment> </executable> <resources> <resource type="memory" value="20"/> </resources> </task> </job>

20. XRSL – example (2) <?xml version="1.0" encoding="UTF-8"?> <job id="Application"> <user dn="dc=pl, dc=poznan, dc=man, uid=pukacki"/> <sequence id="seq1"> <task id="Task1"> <executable name="computation" dmsid="12345"> <arguments> <value>4</value> </arguments> <environment> <stdout dmsid="774"/> </environment> </executable> </task> <task id="Task2"> <executable name="computation" dmsid="8888"> <arguments> <value>4</value> </arguments> <environment> <stdout dmsid="774"/> </environment> </executable> </task> </sequence> </job>

21. Thank you