SHIWA is supported by the FP7 Capacities Programme under contract No. RI-261585

1. Running Workflowson Clouds and Grids Gabor Terstyanszky, University of Westminster T. Fahringer, P. Kacsuk, J. Montagnat, I. Taylor e-Science Workshop, Oxford, 15 March 2012 SHIWA is supported by the FP7 Capacities Programme under contract No. RI-261585

2. Workflow Interoperability Challenge

3. SHIWA Project Partners: Duration: July 2010 – June 2012 (extended to September 2012) 3

4. Project Objectives and Services Objectives: • To create an environment which enables seamless execution of workflows of different workflow systems through workflow interoperability Services: • To support the whole workflow lifecycle: editing, uploading, browsing downloading and executing workflows • To provide coarse- and fine-grained workflow interoperability solutions • To offer Distributed Computing Infrastructure interoperability solution • To provide desktop computer and portal interfaces to manage workflows Key actors: • researchers workflow engine developers workflow developers

5. SHIWA Data Structure • Workflow description abstract workflow • plain text which describes the inputs and outputs and explains what the workflow does • Workflow implementations concrete workflow • It contains all implementation files or references to these files (via eg. URLs) and also holds other data/metadata necessary to run the workflow on its associated workflow engine. • Workflow configurations • It contains pre-defined input files and parameters or references to these files and parameters and other data/metadata of a workflow. • Workflow engines • It contains files and other data/metadata necessary to execute a workflow engine on a grid site or references to them

6. Coarse-Grained Interoperability Workflows of Workflow Engine A Workflow of Workflow Engine B J1 GEMLCA client GEMLCA Service Workflow Engine B J2 WF4 Workflow Engine A J3 DCI • native jobs (or workflows): J1, J2, J3 • non-native workflows: WF4 • - black boxes which are managed as legacy code applications

7. CGI Infrastructure SHIWA Science Gateway SHIWA VO SHIWA Portal SHIWA Repository local cluster Globus DCI gLite DCI WF1 WFn Legacy Code GUI GEMLCA Repository Triana WE WS-PGRADE Workflow editor WF1 WFm Taverna WE WE1 WEp MOTEUR WE WS-PGRADE Workflow engine GEMLCA with GIB ProActive WE ASKALON WE Kepler WE GEMLCA Service ASKALON WE MOTEUR WE GWES WE Proxy Server pre-deployed-WEs SHIWA Proxy Server • SHIWA Science Gateway Resources native WE WS-PGRADE local resources: invocation of locally deployed WEs portal WS-PGRADE v3.3.2 WE submission to local cluster repository GEMLCA + SHIWA repo remote resources: through remotely pre-deployed submitter GEMLCA with GIB WEs to gLite and Globus DCIs proxy management SHIWA Proxy Server

8. SHIWA Portal: Editing Workflows

9. SHIWA Portal: Configuring Workflows

10. SHIWA Portal: Executing Workflows

11. SHIWA Workflow Repository Browse Workflows Browse Implementations Search Sort

12. SHIWA Workflow Repository

13. CGI User Scenario: Native WE SHIWA VO SHIWA Science Gateway local cluster Globus DCI gLite DCI submit WE SHIWA Portal SHIWA Repository s7 WF1 WFn Triana WE Legacy Code GUI search WF Taverna WE s1 s3 MOTEUR WE WF1 WFm WE1 WEp ProActive WE ASKALON WE Kepler WE edit WF GEMLCA Repository WS-PGRADE Workflow editor s2 ASKALON WE MOTEUR WE GWES WE WE + WF s5 WF list s4 user WS-PGRADE Workflow engine GEMLCA with GIB invoke WE s7 pre-deployed-WEs GEMLCA Service s6 Proxy Server SHIWA Proxy Server

14. CGI User Scenario: Non-native WE SHIWA VO SHIWA Science Gateway local cluster Globus DCI gLite DCI SHIWA Portal SHIWA Repository WF1 WFn Triana WE Legacy Code GUI search WF Taverna WE s1 GEMLCA Repository MOTEUR WE WF1 WFm WE1 WEp ASKALON WE ProActive WE Kepler WE WS-PGRADE Workflow Editor user ASKALON WE MOTEUR WE GWES WE WE + WF s5 s2 invoke WE s7 GEMLCA with GIB WS-PGRADE Workflow Engine s4 MOTEUR Workflow editor GEMLCA Service s6 s3 MOTEURWorkflow Engine Proxy Server GEMLCA Client GEMLCA UI SHIWA Proxy Server 14 14

15. workflow for DCI B DCI Interoperability jobs in non-JSDL J1 jobs in JSDL J2 J3 J1 J4 Workflow Engine JSDL Translator J2 J3 J4 DCI Bridge DCI Metabroker Proxy Server 15

16. DCI Bridge • Features: • web service based app which provides standard access to different DCIs: • clouds - EC2, OpenNebula • clusters - LSF, PSB • desktop grids - BOINC, OurGrid, XtremWeb • service grids - ARC, gLite, Globus, Unicore • workflow jobs are submitted to the BES Factory Service of the DCI Bridge where jobs are described in JSDL • Components: • resource registry - configuration interface of DCIs • application manager - BES management • runtime system - job execution management via BES Factory • monitoring - logs and messages handling and visualisation

17. DCI Bridge inputs & executables Portal proxy server WS-PGRADE Workflow Editor DCI Bridge PluginA clouds s7 WS-PGRADE Workflow Engine BES Factory Service s5 Input Queue s1 s6 s4 Plug-in Manager PluginB desktop grids s11 s8 s2 s3 Upload Marnager Job Registry PluginC s10 service grids s9 outputs

18. Fine-Grained Interoperability Workflow of Workflow Engine B IWIR workflow Workflow of Workflow Engine A ASKALON MOTEUR PGRADE Triana J1 J1 J1 J2 J3 J2 J3 J2 J3 J4 J4 J4 WFA ->IWIR frontend plug-in WFA ->IWIR backend plug-in Workflow Engine B DCI • front-end plug-in: converts workflow of workflow engine A into IWIR (Intermediate Workflow Interpreter Representation) • back-end plug-in: converts from IWIR into workflow of workflow engine B 18 18

19. SSP DCI & FGI Infrastructure DCIs SHIWA Science Gateway Unicore DCI ARC DCI SHIWA Portal SHIWA Repository WF1 WFn gLite DCI Globus DCI GEMLCA Repository WF1 WFm GWES WE Kepler WE WS-PGRADE Workflow editor WE1 WFp PGRADE WE Triana WE WS-PGRADE Workflow engine GEMLCA submitter DCI Bridge Taverna WE ProActive WE ASKALON WE MOTEUR WE JSDL Translator Meta Broker Proxy Server pre-deployed-WEs • SHIWA Science Gateway Resources native WE ASKALON, MOTEUR, WS-PGRADE local WEs: Kepler, MOTEUR , Taverna & Triana WEs portal WS-PGRADE v3.4.1 submit to the local cluster repository GEMLCA + SHIWA remote WEs: ASKALON, GWES & MOTEUR WEs GEMLCA ARC, gLite, Globus and Unicore DCIs proxy server

20. DCI Interoperability Scenario SHIWA Science Gateway SHIWA Portal SHIWA Repository WF1 WFn search WF GEMLCA Repository s3 s1 WF1 WFm WE1 WFp WS-PGRADE Workflow editor edit workflow DCIs s2 s4 Unicore DCI s3 ARC DCI submit workflow WS-PGRADE Workflow engine DCI Bridge s7 gLite DCI Globus DCI s5 JSDL Translator Meta Broker s6 Proxy Server 20

21. FGI Interoperability Scenario SHIWA Science Gateway search WF SHIWA Portal SHIWA Repository s1 WF1 WFn s3 GEMLCA Repository edit WF s2 WF1 WFm Workflow Editor WE1 WFp c WS-PGRADE Workflow Editor s4 FP1 BP1 s6 Workflow Engine FP2 BP2 c WS-PGRADE Workflow Engine DCI Bridge JSDL Translator s5 FPn BPm s7 submit WF Meta Broker Unicore DCI ARC DCI Proxy Server gLite DCI Globus DCI DCIs 21

22. SHIWA Simulation Platform: Access • SHIWA Science Gateway (SHIWA Portal + SHIWA Repository) • joint portal and repository account given by the science gateway administrator • DCI resources • certificate given by the relevant DCI administrator • Access to the SHIWA Portal • shiwa-portal.cpc.wmin.ac.uk/liferay-portal-6.05 • Access to the SHIWA Repository • shiwa-repo.cpc.wmin.ac.uk 22

23. Conclusions • the simulation platform supports the whole lifecycle of workflows, i.e. creating, testing, uploading, browsing, downloading and running workflows • users can use workflows of their own and of other workflow systems through the same user interface, i.e. they can re-cycle or share workflows created by other research teams or even other research communities • workflow developers (or e-scientists) can create, run, upload workflows through the simulation platform • researchers (astro physicists, earth scientists, bio scientists, etc.) can browse the workflow repository, select and run workflows through the simulation platform • the simulation platform provides access to major European e-infrastructure, the European Grid Initiative (EGI) infrastructure which offers large pool of compute and storage resources