Workflow execution in EGEE VOs, P-GRADE portal for EGEE VOs

1. Workflow execution in EGEE VOs, P-GRADE portal for EGEE VOs Peter Kacsuk MTA SZTAKI, Hungary Kacsuk@sztaki.hu

2. Technology concerns of Grid systems • Fast evolution of Grid systems and middleware: • GT1, GT2, OGSA, OGSI, GT3, WSRF, GT4, … • Many Grid systems are built based on these different technologies • EGEE (LCG-2, g-Lite), NorduGrid, UK NGS, Grid2003, etc.

3. User concerns of Grid systems • How to cope with the variety of these Grid systems? • How to develop/create new Grid applications? • How to execute Grid applications? • How to observe the application execution in the Grid? • How to tackle performance issues? • How to port legacy applications • to Grid systems • between Grid systems? • How to execute Grid applications over several Grids in a transparent way?

4. What does the user want? • Fast development of Grid workflow application • Various workflow components: • Sequential code • MPI code • Legacy code • Parallel execution of workflow components in the Grid • Simultaneous execution of workflow components in different Grids • Monitored execution of a workflow in the Grid • To port applications among Grids Portal technology could be an answer for the users

5. Properties of the P-GRADE Portal • General purpose, graphical, workflow-oriented Grid portal to support development and execution of workflow-oriented Grid applications • Supported services – functionalities • MyProxy – proxy credential management • GT2/GT3 GRAM – job execution • Mercury – job/resource monitoring • PROVE – workflow/job execution visualization • BDII and MDS-2 - information system access • Multi-Grid portal • GridSphere based • Easy to expand with new portlets • Easy to tailor to end-user needs

6. Workflow support in the P-GRADE Portal • Contains • built-in workflow editor and • workflow execution manager • Components of the workflow can be • Job to be executed as a Condor job in GT2/GT3 Grids (like LCG-2, GridLab, etc.) • GEMLCA service to be executed in a GT-3 and GT-4 Grids (UK OGSA test-bed project) • Portal release, tutorial and demo are available at: http://www.lpds.sztaki.hu/pgportal

7. What is a workflow? • The workflow is a graph where • Nodes are jobs (or services) • Arcs represent file transfer between the jobs (services) • Semantics of the workflow: • Job can be executed if all the necessary file transfers represented by the arcs are completed

8. Two level parallelism by a workflow • The workflow concept enables the efficient solution of complex problems in a distributed environment like Grid • Semantics of the workflow enables two levels of parallelism: • Parallel execution inside a workflow node • Parallel execution among workflow nodes Jobs can be parallel

9. Ultra-short range weather forecast (Hungarian Meteorology Service) Forecasting dangerous weather situations (storms, fog, etc.),crucial task in the protection of life and property 25 x Processed information: surface level measurements, high-altitude measurements, radar, satellite, lightning, results of previous computed models 10 x 5 x 25 x • Requirements: • Execution time < 10 min • High resolution (1km)

10. Workflows developed in P-GRADE can be used in the P-GRADE portal • They can be • Submitted to the Grid by the portal • Modified by the workflow editor of the portal • P-GRADE is the development environment and the portal is the Grid execution environment Why is it called P-GRADE Portal? • P-GRADE is a Parallel Grid Run-time and Application Development Environment • Containing workflow editor and execution support • The P-GRADE workflow editor is compatible with the workflow editor of the portal

11. Principles of the P-GRADE portal Certificate server CERTIFICATE (download) Remote Clusters to be controlled WORKFLOW MANAGER (submit) CERTIFICATE (upload) EDITOR (save|upload) Portal server Workflow (result) EDITOR (open) WORKFLOW MANAGER (output)

12. P-GRADE Portal  Login • Current release is 2.1 • A username and a password are required for login • Apply them at the service provider: • SZTAKI for HunGrid and SEE-Grid • Univ. of Westminster for UK NGS

13. Principles of the P-GRADE portal Certificate server CERTIFICATE (download) Remote Clusters to be controlled WORKFLOW MANAGER (submit) CERTIFICATE (upload) EDITOR (save|upload) Portal server Workflow (result) EDITOR (open) WORKFLOW MANAGER (output)

14. Certificate Manager(certificates portlet) • The certificates could be obtained from the MyProxy server using the “Download” button after entering the certificate and server details • Login and proxy downloading require different usernames and passwords • To submit jobs users should have valid certificates • The Certificate Manager handles users’ certificates uploading, downloading, displaying & allocating them • Downloaded certificates have to be allocated to a Grid using the “Set for Grid” button • This operation displays the details of the certificate and the list of available Grids • Different Grids (eg. SEEGRID and HunGrid) require different user certificates • The Certificate Manager handles different certificates of different Grids • There is a possibility to displaydetails of certificates using the “Details” button

15. Resource Manager(settings portlet) • Resource Manager helps to add & delete resources of different Grids • To add a new resource to a Grid, users have to select it using the “Resources”button • The portal displays the list of available resources of the selected Grid • Users can add resources from the list using the “Load default” button or edit them manually

16. User concerns of Grid systems • How to cope with the variety of these Grid systems? • How to develop/create new Grid applications? • How to execute Grid applications? • How to observe the application execution in the Grid? • How to tackle performance issues? • How to port legacy applications • to Grid systems • between Grid systems? • How to execute Grid applications over several Grids in a transparent way?

17. Main interactions 2. Certificate server Remote Clusters to be controlled WORKFLOW MANAGER (submit) CERTIFICATE (download) CERTIFICATE (upload) EDITOR (save / upload) Portal server Workflow (result) EDITOR (open) WORKFLOW MANAGER (output)

18. Workflow Editor • Workflow is a directed graph • Every node is a job • The arrows represent the file transfers among the jobs (and sites)

19. - All jobs have properties window • This window contains the most important information about the job eg. Type, name, required process number, Grid and resource name for execution Workflow Editor

20. Monitoring System(information system portlet) • The portal uses MDS-2 and LCG-2 information systems • Users can select a Grid and a Virtual Organization to be displayed

21. Monitoring System(information system portlet) • Detailed view of the selected Grid

22. Every job has ports • The box in Green color represents the input port • Every port has a properties window to define the special settings Workflow Editor

23. Job i Job j Job k Remote input files are copied upon submitof Job i (Stepi2) Executing Resources (CE) Locations of remote files (SE) Remote output files are copied upon terminationof job i(Stepi3) The eventual communication between the user desktop and the locations of remote file(s) is outside of the scope of the Portal. (Step 0,Step 4) Local output files are copied upon terminationof job i(Stepi3) Local output files may be downloaded upon the termination of the wholeworkflow (Step 4) The executable and the local input files are copied upon submit of job i (Step i2) P-Grade Portal Server Local input files, job executables, and the workflow graph uploaded by the Workflow Editor (Step 1) User Desktop

24. Job i Job j Job k Remote input files are copied upon submitof Job i (Stepi2) Executing Resources (CE) Locations of remote files (SE) Remote output files are copied upon terminationof job i(Stepi3) The eventual communication between the user desktop and the locations of remote file(s) is outside of the scope of the Portal. (Step 0,Step 4) Local output files are copied upon terminationof job i(Stepi3) Local output files may be downloaded upon the termination of the wholeworkflow (Step 4) The executable and the local input files are copied upon submit of job i (Step i2) P-Grade Portal Server Local input files, job executables, and the workflow graph uploaded by the Workflow Editor (Step 1) User Desktop

25. Job i Job j Job k Remote input files are copied upon submitof Job i (Stepi2) Executing Resources (CE) Locations of remote files (SE) Remote output files are copied upon terminationof job i(Stepi3) The eventual communication between the user desktop and the locations of remote file(s) is outside of the scope of the Portal. (Step 0,Step 4) Local output files are copied upon terminationof job i(Stepi3) Local output files may be downloaded upon the termination of the wholeworkflow (Step 4) The executable and the local input files are copied upon submit of job i (Step i2) P-Grade Portal Server Local input files, job executables, and the workflow graph uploaded by the Workflow Editor (Step 1) User Desktop

26. - The box in Grey color represents the output port • Every port has a properties window to define the special settings Workflow Editor

27. Workflow editor The workflow pull-down menu enables saving and later opening the prepared workflows

28. User concerns of Grid systems • How to cope with the variety of these Grid systems? • How to develop new Grid applications? • How to execute Grid applications? • How to observe the application execution in the Grid? • How to tackle performance issues? • How to port legacy applications • to Grid systems • between Grid systems? • How to execute Grid applications over several Grids in a transparent way?

29. Main interactions 3. Certificate server Remote Clusters to be controlled WORKFLOW MANAGER (submit) CERTIFICATE (download) CERTIFICATE (upload) EDITOR (save|upload) Portal server Workflow (result) EDITOR (open) WORKFLOW MANAGER (output)

30. Workflow Management(workflow portlet) • The portlet presents the status, size and output of the available workflow in the “Workflow” list • It has a Quota manager to control the users’ storage space on the server • The portlet also contains the “Abort”, “Attach”, “Details”, “Delete” and “Delete all” buttons to handle execution of workflows • The “Attach” button opens the workflow in the Workflow Editor • The “Details” button gives an overview about the selected workflow

31. Workflow Execution(workflow portlet) - The jobs are executed on the selected Grids and resources • -The arrows represent the file transfers among the sites • The blocks correspond jobs of the workflow -When the workflow has been completed the results are downloadable in zip format • - After the submission the status of the workflow’s jobs will be initialized • White color represents the INIT status -Red color represents the RUNNING status - Green color depicts the FINISHED status -Portal does on-line workflow animation according to the status of each job

32. Workflow Execution(workflow portlet) • The portal displays the list of jobs and their status • The current status of the jobs are represented by colors • It also provides access to their logs and outputs, and visualizes them White/Red/Green color means the job is initialised/running/finished

33. Main interactions 4. Certificate server Remote Clusters to be controlled WORKFLOW MANAGER (submit) CERTIFICATE (download) CERTIFICATE (upload) EDITOR (save|upload) Portal server Workflow (result) EDITOR (open) WORKFLOW MANAGER (output)

34. Workflow Execution(workflow portlet) • When the jobs are finished the results could be downloaded in zip format • The “Output” button displaysthe output of each job White/Red/Green color means the job is initialised/running/finished

35. Downloading the results…

36. User concerns of Grid systems • How to cope with the variety of these Grid systems? • How to develop new Grid applications? • How to execute Grid applications? • How to observe the application execution in the Grid? • How to tackle performance issues? • How to port legacy applications • to Grid systems • between Grid systems? • How to execute Grid applications over several Grids in a transparent way?

37. On-Line Monitoring(workflow portlet) • The portal also monitors and visualizes jobs and processes of workflows • The portal monitors and displays workflows • The portal also generates a statistic view

38. Job execution visualization

39. User concerns of Grid systems • How to cope with the variety of these Grid systems? • How to develop new Grid applications? • How to execute Grid applications? • How to observe the application execution in the Grid? • How to tackle performance issues? • How to port legacy applications • to Grid systems • between Grid systems? • How to execute Grid applications over several Grids in a transparent way?

40. Today’s reality:Technology specific Grid portals NorduGrid GS GS GS GS Grid portal GT-3 GS GS LCG-2 GS GS GS GS GS GS GS Web browser Grid portal HCG GS HCG GT3 GS GS GS GS GS Grid portal LCG-2 GT3 GS GS GS GS GS GS GS GS GS GS LCG-2

41. Future Grid portals: Functionality specific portals GS GS GS GS Medical Grid portal GS GS GS GS GS GS GS GS GS Web browser Computa-tional Grid portal GS GS GS GS GS GS Physics Grid portal GS GS GS GS GS GS GS GS GS GS

42. Future Grid portals: Functionality specific portals This is the P-GRADE Portal! GS GS GS GS Medical Grid portal GS GS GS GS GS GS GS GS GS Web browser Computa-tional Grid portal GS GS GS GS GS GS Physics Grid portal GS GS GS GS GS GS GS GS GS GS

43. Future multi-Grid portals GS GS GS GS GS GS WS GS GS GS GS GS GS GS GS GT3 Jini GS GS GS GS GS GT3 GS GS GS GS GS GS GS GS WS GS GS

44. The problem of current portals • They tightly connected and tailored to only one particular Grid • If the user wants to move to another Grid she has to learn the new environment • They do not support the simultaneous access of several Grids • P-GRADE portal release 2.1 solves these problems

45. Multi-Grid portals Portal classification

46. MISI (current) Portals LCG-2 P-GRADE-Portal Rome London Athens

47. MIMI PortalP-GRADE portal (2.1) GridLab P-GRADE-Portal User can choose where to execute the workflow SEE-Grid Rome London Athens

48. MIMC PortalP-GRADE portal (2.1) GridLab P-GRADE-Portal Different jobs can be executed in different grids SEE-Grid London Rome Athens

49. - All jobs have properties window • This window contains the most important information about the job eg. Type, name, required process number, Grid and resource name for execution Workflow Editor

50. Workflow Execution(workflow portlet) White/Red/Green color means the job is initialised/running/finished