1 / 24

Defining and Running Parametric Study Workflow Applications by the P-GRADE Portal

Defining and Running Parametric Study Workflow Applications by the P-GRADE Portal. P. Kacsuk, G. Sipos, A. Toth, Z. Farkas, G. Kecskemeti and G. Hermann MTA SZTAKI www.lpds.sztaki.hu/pgportal pgportal@lpds.sztaki.hu. Outline of the talk. Introduction to P-GRADE portal PS-extension motivations

ziarre
Download Presentation

Defining and Running Parametric Study Workflow Applications by the P-GRADE Portal

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Defining and Running Parametric Study Workflow Applications by the P-GRADE Portal P. Kacsuk, G. Sipos, A. Toth, Z. Farkas, G. Kecskemeti and G. HermannMTA SZTAKIwww.lpds.sztaki.hu/pgportalpgportal@lpds.sztaki.hu

  2. Outline of the talk • Introduction to P-GRADE portal • PS-extension motivations • Principles of the PS-portal • Usage of the PS-portal • Distributing workflows among different Grid resources • Conclusions European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  3. Proxy management Definition of Grid resources Workflow creation Job mapping to Grid resources Workflow management and execution visualization P-GRADE portal in a nutshell European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  4. References • P-GRADE Portal service is available for • Central European VO of EGEE • GILDA: Training VO of EGEE • HUNGRID VO of EGEE • SEE-GRID infrastructure • GIN VO • National Grids (UK National Grid Service, HunGrid, etc.) • Under preparation for • US Open Science Grid, TeraGrid • Economy-Grid, Swiss BioGrid, Bio and Biomed EGEE VOs, BioInfoGrid, BalticGrid • Many national Grids European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  5. User’s feedback • Users want parameter study (PS) support at workflow level • It means: • If the user has an existing workflow in a repository, he would like to run • the same workflow (without any change) • with many different parameters European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  6. Related research • PS at job level – main subject is scheduling • PS at workflow level – The workflow description supports the creation of PS runs • What our users want: executing workflows stored in workflow repositories with many inputs as PS European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  7. Principles of the PS-portal Storage elements 4 input files • Take the existing workflow as a black box • Modify only the external input ports to accept many input files • Create the cross-product of the input files and run the workflow with all combinations • Because of the large number of files both PS inputs and outputs are stored in the Grid! 3 input files Storage elements 12 outputfiles Storage elements European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  8. Generating cross products and executable workflows (e-WFs) 4 x 3 normal workflow execution 1 PS workflow execution PS port: 4 instances of the input file PS port: 3 instances of the input file = European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  9. Steps of creating a PS-workflow1. Start from a tested Workflow European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  10. 2. Define the PS Port(s) Any external input port can be defined as PS-port European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  11. 3.Define Subdirectory of Inputs PS port is represented by a directory and not by a single file European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  12. 4. Define target Grid and subdirectory for the results European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  13. 5. PS Workflow ready to Submit European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  14. Progress of submissions in PS detailed view Total = Init + Submitted + Rescue + Error + Finished European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  15. How to distribute workflows among different Grid resources? Goal: Assign the 24 jobs to 24 different Grid resources 1 PS workflow execution PS port: 4 instances of the input file PS port: 3 instances of the input file = European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  16. Single-Grid, Static Assignment Assigns the 24 jobs to 2 Grid Resources within 1 Grid 1 PS workflow execution PS port: 4 instances of the input file A A A A B B B B PS port: 3 instances of the input file = A A A A B B B B Assign resource A in Grid X Assign resource B in Grid X A A A A B B B B European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  17. Single-Grid, Dynamic Assignment via Broker Assigns the 24 jobs to 24 Grid Resources within 1 Grid 1 PS workflow execution PS port: 4 instances of the input file C D A B G H E F PS port: 3 instances of the input file = I K L J O P M N Assign to broker of Grid X Assign to broker of Grid X V S T R Y Z U X European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  18. P-GRADE portal is a multi-Grid portal TeraGrid (GT2/GT4) P-GRADE-Portal SEE-Grid (glite) London Rome Athens European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  19. Static Multi-Grid Assignment Assigns the 24 jobs to 24 Grid Resources within 2 Grids 1 PS workflow execution PS port: 4 instances of the input file XC XD XA XB YG YH YE YF PS port: 3 instances of the input file = XI XK XL XJ YO YP YM YN Assign to broker of Grid X Assign to broker of Grid Y XV XS XT XR YY YZ YU YX European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  20. Dynamic Multi-Grid Assignment Assigns the 24 jobs to 24 Grid Resources within 2 Grids 1 PS workflow execution PS port: 4 instances of the input file YC XD XA XB YG XH YE XF PS port: 3 instances of the input file = YI XK XL XJ YO YP YM XN Assign to meta-broker Assign to meta-broker YV XS XT XR YY XZ YU YX European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  21. Conclusions • Our solution: • Keeps both the workflow GUI and the parameter study support concept as simple as possible. • Enables run any existing workflow with different parameter sets without modifying the structure of the workflow. • Manages the execution of the workflows on as many Grid resources as possible. • Enables the collection of Grid resources from several Grids even if they are based on different Grid technology. European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  22. Thanks for your attention ? Further information: www.lpds.sztaki.hu European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  23. Distributing workflows among different Grid resources • Single-Grid, Static Assignment: Statically assigning each node of the workflow to different resources in the same Grid (done) • Single-Grid, Dynamic Assignment: Dynamically assigning each node of the workflow to different resources in the same Grid using the broker of the Grid (done) • Static multi-Grid Assignment: Statically assigning different Grids to different nodes (done) • Dynamic multi-Grid Assignment: Dynamically assigning the nodes to different Grids using a meta-broker (under investigation) European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

  24. Meta- Broker Multi-Grid portals with meta-broker EGEE Grid Provides the largest achievable parallelism for PS workflows Broker UK NGS Broker Selects Grid for each node of the PS workflow European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

More Related