1 / 21

Introduction to the SHIWA Simulation Platform

This article provides an overview of the SHIWA Simulation Platform, its architecture, and its various components such as the SHIWA Portal, Repository, and Science Gateway. It also highlights the functionalities and benefits of using the platform for e-scientists.

sbrock
Download Presentation

Introduction to the SHIWA Simulation Platform

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Introduction to the SHIWA Simulation PlatformEGI User Forum, 21.04. 2011. VilniusTamas Kiss – Gabor Terstyanszky – University of Westminster

  2. SHIWA Simulation Platform at a glance • SHIWA Simulation Platform v1 (SSP v1): • deployed at production level since March 2011 • supports coarse-grained workflow interoperability • native workflow system is the P-GRADE workflow system • enables either embedding or invoking ASKALON, GWES, Kepler, MOTEUR, Taverna and Triana workflows • allows e-scientists to search workflows uploaded to the SHIWA Workflow Repository and use them in their experiments

  3. Overall architecture of SSP v1 SHIWA Science Gateway SHIWA VO SHIWA Portal SHIWA Repository local cluster Globus DCI gLite DCI WF1 WFn GEMLCA admin Triana WE Taverna WE WF1 WFm WE1 WEp ASKALON WE MOTEUR WE GEMLCA Repository PGRADE Workflow editor Kepler WE MOTEUR WE GWES WE PGRADEWorkflow engine ASKALON WE pre-deployed-WEs GEMLCA with GIB GEMLCA Service

  4. SHIWA Science Gateway • SHIWA Portal: • P-GRADE 2.4 portal technology • certificate/proxy and DCI resource management • access to different DCI information systems • integrated with the P-GRADE Workflow System (used as native workflow engine) • administration of GEMLCA services • GEMLCA Service: • converts legacy applications such as workflows and workflow engines into Grid services • invokes locally or remotely pre-deployed workflow engines or submits workflow engines to local or remote resources to execute workflows • GEMLCA Repository • manage workflow engine (WE) and workflow (WF) data to support execution • SHIWA Repository: • create, add, edit and delete workflow metadata • upload and download workflows with their implementations and configurations. SHIWA Science Gateway SHIWA Portal SHIWA Repository WF1 WFn GEMLCA admin WF1 WFm WE1 WEp GEMLCA Repository PGRADE Workflow editor PGRADEWorkflow engine GEMLCA with GIB GEMLCA Service

  5. SHIWA VO • VO services: • Logical File Catalogue: • lfc-egee.in2p3.fr • Computing elements: • karwendel.dps.uibk.ac.at – Austrian NGI (Globus) • othello.zih.tu-dresden.de – German NGI (Globus) • ngs.wmin.ac.uk – UK NGI (Globus) • phoebe.deimos.htc.biggrid.nl – Dutch NGI (g-Lite) • Storage resources: • carme.htc.biggrid.nl (9.9TB) • Workflow engines: • Moteur engine to submit to g-Lite resources • Askalon and GWES engines to submit to Globus resources • Triana, Taverna, Moteur, Kepler and Askalon engines deployed locally at UoW SHIWA VO local cluster Globus DCI gLite DCI Triana WE Taverna WE ASKALON WE MOTEUR WE Kepler WE MOTEUR WE GWES WE ASKALON WE pre-deployed-WEs

  6. Family of P-GRADE Portal products P-GRADE portal Creating (basic) workflows and parameter sweeps for clusters, service grids, desktop grids www.portal.p-grade.hu P-GRADE/GEMLCA portal (University of Westminster) To wrap legacy applications into Grid Services To add legacy code services to P-GRADE Portal workflows http://www.cpc.wmin.ac.uk/cpcsite/gemlca WS-PGRADE Creating complex workflow and parameter sweeps for clusters, service grids, desktop grids, databases Creating complex applications using embedded workflows, legacy codes and community components from workflow repository www.wspgrade.hu SHIWA Portal – based on P-GRADE portal

  7. P-GRADE/GEMLCA Portal in a nutshell • General purpose, workflow-oriented computational Grid portal. Supports the development and execution of workflow-based Grid applications – a tool for Grid orchestration • Based on GridSphere-2 • Easy to expand with new portlets (e.g. application-specific portlets) • Easy to tailor to end-user needs • Developed by SZTAKI and the P-GRADE portal alliance • Grid services supported by the portal:

  8. GEMLCA Repository • legacy code repository which stores workflows and workflow engines with their Legacy Code Interface Descriptions (LCID) • Entities in the repository: • workflows - LCID + data (implementation + configuration) • workflow engines - LCID + data (binary + configuration) • Actors: • e-scientists: browse the repository and simply select the application to include in their workflows • workflow developers: create, upload and manage workflow’s LCID and data into the repository • workflow engine developers: create, upload and manage workflow engine’s LCID and data into the repository.

  9. GEMLCA Repository • ports are defined automatically • command line parameters are displayed automatically

  10. SHIWA Workflow Repository • a workflow repository with extended metadata support • Entities in the repository: • workflows - describe the workflow behaviour • implementations - everything that is necessary for the workflow engine to be able to run the workflow • configurations - input files and parameters of workflows • Actors: • e-scientists: search/browse the repository to find and download workflows • workflow developers: enable e-scientists to run workflows - define workflows with their implementations and configurations, and upload and publish them • repository administrators: manage the repository as system administrators.

  11. SHIWA Repository Data Structure • Workflow description • plain text describing inputs and outputs and explains what the workflow does • Workflow implementations • all implementation files or references to these files • data/metadata necessary to run the workflow on its associated workflow engine. • Workflow configurations • pre-defined input files and parameters or references to these, and other data/metadata of a workflow. • Workflow engines • files and other data/metadata necessary to execute a workflow engine on a grid site or references to them

  12. SHIWA Workflow Repository http://dev17-portal.cpc.wmin.ac.uk:8080/shiwa-repo/

  13. SSP v1 – User scenarios Workflow engine upload SHIWA Science Gateway SHIWA Portal SHIWA Repository Actor: WE developer Frequency: once for a particular version of a WE Process: step 1- create and upload workflow engine’s legacy code description (LCID) using the GEMLCA admin portlet. step 2 - upload the workflow engine’s binary to the GEMLCA Repository (WE1, …WEp) WF1 WFn s1 create LCID GEMLCA admin WE+ LCID upload WE s2 WF1 WFm WE developer WE1 WEp GEMLCA Repository PGRADE Workflow editor PGRADEWorkflow engine GEMLCA with GIB GEMLCA Service

  14. SSP v1 – User scenarios Workflow upload SHIWA Science Gateway Actor: WF developer Frequency: once for a particular version of a WF Process: step 1 - create and upload the workflow’s LCID description using the GEMLCA admin portlet step 2 - upload the workflow descriptor file to the GEMLCA Repository (WF1, …WFm). step 3 - define the workflow metadata using the SHIWA Repository GUI step 4 - upload the workflow descriptor file to SHIWA Repository (WF1, …WFn). SHIWA Portal SHIWA Repository create WF metadata s3 WF1 WFn upload WF s4 s1 create LCID GEMLCA admin WF + LCID s2 upload WF WF1 WFm WF developer WE1 WEp GEMLCA Repository PGRADE Workflow editor PGRADEWorkflow engine GEMLCA with GIB GEMLCA Service

  15. SHIWA VO SHIWA Science Gateway local cluster Globus DCI gLite DCI SHIWA Portal SHIWA Repository WF1 WFn Triana WE search WF GEMLCA admin Taverna WE s1 MOTEUR WE WF1 WFm s2 WE1 WEp WF list ASKALON WE Kepler WE edit WF GEMLCA Repository s2 PGRADE Workflow editor ASKALON WE MOTEUR WE GWES WE s4 WE + WF e-scientist s3 PGRADEWorkflow engine GEMLCA with GIB pre-deployed-WEs s5 invoke WE GEMLCA Service SSP v1 – User scenarios Workflow creation and execution – invoking pre-deployed WF engines

  16. SSP v1 – User scenarios Workflow creation and execution – invoking pre-deployed WF engines Actor: e-scientist or workflow developer Goal: to run the embedded or non-native workflows on locally or remotely pre-deployed workflow engines step 1 - browse the SHIWA repository, select workflows to execute step 2 - create embedded workflows by adding non-native workflows selected in step 1 using their workflow IDs step 3 - P-GRADE workflow engine submits the embedded workflow through GEMLCA Service step 4 - the GEMLCA Service combines the LCIDs of the workflow and its corresponding workflow engine. step 5 - the GEMLCA Service invokes the pre-deployed workflow engine to execute the workflow on the target Distributed Computing Infrastructure

  17. SHIWA VO SHIWA Science Gateway local cluster Globus DCI gLite DCI SHIWA Portal SHIWA Repository WF1 WFn Triana WE search WF Taverna WE GEMLCA admin s1 MOTEUR WE WF1 WFm s2 WE1 WEp WF list ASKALON WE Kepler WE edit WF GEMLCA Repository PGRADE Workflow editor s2 ASKALON WE MOTEUR WE GWES WE WE + WF s4 e-scientist s3 PGRADEWorkflow engine pre-deployed-WEs GEMLCA with GIB s5 submit WE GEMLCA Service SSP v1 – User scenarios Workflow creation and execution – submitting WF engines to target DCI

  18. SSP v1 – User scenarios Workflow creation and execution – submitting WF engines to target DCI Actor: e-scientist or workflow developer Goal: to submit the workflow engines to local or remote resources to execute the embedded or non-native workflows step 1 - browse the SHIWA repository, select workflows to execute step 2 - create embedded workflows by adding non-native workflows selected in step 1 using their workflow IDs step 3 - P-GRADE workflow engine submits the embedded workflow through GEMLCA Service step 4 - the GEMLCA Service combines the LCIDs of the workflow and its corresponding workflow engine. step 5 - the GEMLCA Service submits the workflow engine with the selected workflow to the target DCI (in this case local cluster)

  19. SSP v1 – How to get access? • Register with the SHIWA VO: • aim: to get authorisation to access GEMLCA resource, WF engines, compute and data resources • https://cclcgvomsli01.in2p3.fr:8443/voms/shiwa-workflow.eu • Get an account to the SHIWA portal • aim: to get access to the portal • Register with the SHIWA repository • aim: to be authorised to upload and modify workflows • repository can be browsed and workflows can be downloaded without account • Detailed how-to-get-access guide: • http://www.cpc.wmin.ac.uk/cpcsite/index.php/Other_projects:SHIWA_SSPv1

  20. SHIWA Simulation Platform - the future • SHIWA Simulation Platform v2 (SSP v2): • to be operational by March 2012 • supports both coarse- and fine-grained workflow interoperability • not only the P-GRADE but the ASKALON and MOTEUR workflow systems could also be native workflow systems • supports the SHIWA Desktop solution • converts workflows from one workflow representation to another using the IWIR Interpreter

  21. Thank you.Questions? • SHIWA Portal: • https://grid2-portal.cpc.wmin.ac.uk:8080/gridsphere/gridsphere • SHIWA Workflow Repository • http://dev17-portal.cpc.wmin.ac.uk:8080/shiwa-repo

More Related