1 / 17

WS-PGRADE/gUSE Supporting e-Science communities in Europe

WS-PGRADE/gUSE Supporting e-Science communities in Europe. Zoltan Farkas MTA SZTAKI LPDS, Hungary zoltan.farkas@sztaki.mta.hu. Outline. WS-PGRADE/gUSE in a nutshell Flexibility of: Parallelism Compute infrastructure access Data storage access Customization possibilities

grant
Download Presentation

WS-PGRADE/gUSE Supporting e-Science communities in Europe

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. WS-PGRADE/gUSESupporting e-Science communities in Europe Zoltan Farkas MTA SZTAKI LPDS, Hungary zoltan.farkas@sztaki.mta.hu

  2. Outline • WS-PGRADE/gUSE in a nutshell • Flexibility of: • Parallelism • Compute infrastructure access • Data storage access • Customization possibilities • Customized gateway examples • Interested in the details?

  3. WS-PGRADE/gUSE Generic-purpose gateway framework • Based on Liferay • General purpose • Workflow-oriented gateway framework • Supports the development and execution of workflow-based applications • Supports the fast development of domain-specific gateways by a customization technology • Most important design aspect is flexibility

  4. Parallel execution inside a workflow node • Parallel execution among workflow nodes Multiple jobs run parallel Each job can be a parallel program Flexibility in exploiting parallelism Multiple instances of the same workflow with different data files • Parameter study execution of the workflow

  5. Flexibility of using compute infrastructures • Flexible management of Security: • Individual users’ certificate (X.509, SAML, pubkey, …) • Robot certificates • Flexible access to various types of DCIs: • Clusters (PBS, LSF, MOAB, SGE) • Cluster grids (ARC, gLite, GT2, GT4, GT5, UNICORE) • Supercomputers (e.g. via UNICORE) • Desktop grids (BOINC) • Clouds (Direct Access, EGI FedCloud, CloudBroker Platform) • XSEDE

  6. Flexibility in data storage access • Use Data Avenue Blacktop service • To access data storages in different DCIs • To transfer files among the storages of different DCIs • To upload/download files to/from the storages of different DCIs • Data Avenue Liferay portlet to access the data transfer services of Data Avenue Blacktop • See details: https://data-avenue.eu/ • Currently supported protocols: • HTTP(s), SFTP, GSIFTP, SRM, iRODS, S3

  7. FS1 FS2 FS3 FSn Data Avenue services Data Avenue @ XY Data Avenue Portlet DataAvenue @ SZTAKI WS-PGRADE gateway Data Avenue Blacktop service Amazon Openstack gLite GT5

  8. FS2 FS5 FS1 FS3 Generic data transfer among WS-PGRADE workflow nodes DCI1 DCI2 WS-PGRADE Workflow J1 J3 J2 J4 J5 The Data Avenue Blacktop services are available not only by the Data Avenue portlet but also by the nodes of a WS-PGRADE workflow J: Job FS: File storage system, e.g. gsiftp, iRODS, SRM

  9. Data Avenue in WS-PGRADE/gUSE • Data sources and destinations of jobs can be selected • gUSE automatically manages data transfers using Blacktop • Actual transfer delegated up to the worker node wherever possible, bypassing the Blacktop service if the middleware is capable of handling the protocol

  10. Flexibility of gateway types and user views • Generic purpose gateways (workflow view) • Core WS-PGRADE/gUSE (e.g. Greek NGI) • Generic purpose gateway for specific technologies (workflow view) • SHIWA gateway for workflow sharing and interoperation • Domain-specific science gateway instance • Autodock gateway (end-user view) • Swiss proteomics portal (customized GUI using ASM API) • VisIVO Mobile (use of Remote API)

  11. WS-PGRADE/gUSE customization possibilities • Application Specific Module: ASM API • Remote API • End User View • Job submission

  12. Some examples of SCI-BUS domain-specific gateways

  13. VisIVO Astrophysics Science Gateway Architecture

  14. VisIVO Mobile

  15. gUSE-based gateways • Close to 130 deployments world-wide • More than 17.000 downloads from SourceForge

  16. Summary: SCI-BUS solution to build science gateways by communities Portlet Developers Gateway Developers Data management SCI-BUS Portlet Repository Job submission Scientists Workflow Template A …. Download from sourceforge Domain specific Gateway SCI-BUS Generic Framework based on WS-PGRADE and Liferay Portlet Portlet Portlet Portlet Data transfer SCI-BUS Generic Framework Profile Gateway Operators Monitoring Function Workflow Application Function Function Application Sequencing SHIWAWorkflow repository fMRI DCI1 MRI simulator NonLinSyst simulator DCIn Application Developers DCI2 ….

  17. Additional info • SCI-BUS: http://sci-bus.eu/ • WS-PGRADE/gUSE: http://guse.hu, http://guse.sf.net/ • Data Avenue: https://data-avenue.eu/ • More in-depth talk: • When: 2:30-4 p.m. Friday, Oct. 3 • Where: Purdue University, LWSN, Room 1106 • Thank you for your attention!

More Related