SHIWA is supported by the FP7 Capacities Programme under contract No. RI-261585

1. SHIWA andCoarse-grained Workflow InteroperabilityGabor Terstyanszky, University of WestminsterSummer SchoolBudapest02-06 July 2012 SHIWA is supported by the FP7 Capacities Programme under contract No. RI-261585

2. Workflow Interoperability Challenge

3. SSP Objectives and Services Objectives: • To create an environment which enables seamless execution of workflows of different workflow systems through workflow interoperability Services: • To support the whole workflow lifecycle: editing, uploading, browsing downloading and executing workflows • To provide coarse- and fine-grained workflow interoperability solution • To offer Distributed Computing Infrastructure interoperability solution • To provide desktop computer and portal interfaces to manage workflows Key actors: • researchers workflow engine developers workflow developers

4. SHIWA Data Structure • Workflow description abstract workflow • plain text which describes the inputs and outputs and explains what the workflow does • Workflow implementations concrete workflow • It contains all implementation files or references to these files (via eg. URLs) and also holds other data/metadata necessary to run the workflow on its associated workflow engine. • Workflow configurations • It contains pre-defined input files and parameters or references to these files and parameters and other data/metadata of a workflow. • Workflow engines • It contains files and other data/metadata necessary to execute a workflow engine on a grid site or references to them

5. Coarse-Grained Interoperability: Single Non-native Workflow Workflow of Workflow Engine B Submission client Workflow Engine A Submission Service Workflow Engine B WF DCI

6. Coarse-Grained Interoperability: Meta-Workflow Workflows of Workflow Engine A Workflow of Workflow Engine B J1 Submission client Submission Service Workflow Engine B J2 WF4 Workflow Engine A J3 DCI • native workflows: J1, J2, J3 • non-native workflows: WF4 • - black boxes which are managed as legacy code applications

7. SSP: CGI Infrastructure DCIs SHIWA Science Gateway SHIWA Repository Unicore DCI ARC DCI SHIWA Portal WF1 WFn GEMLCA admin gLite DCI Globus DCI GEMLCA Repository WS-PGRADE Workflow editor WF1 WFm ASKALON WE Galaxy WE Triana WE WE1 WEp Taverna WE WS-PGRADE Workflow engine GWES WE Kepler WE GEMLCA with GIB Pegasus WE MOTEUR WE GEMLCA Service Proxy Server ProActive WE PGRADE WE SHIWA Proxy Server Workflow Engines • SHIWA Science Gateway Resources native WE WS-PGRADE local resources: invocation of locally deployed WEs portal WS-PGRADE v3.4.1 WE submission to local cluster repository GEMLCA + SHIWA repo remote resources: through remotely pre-deployed submitter GEMLCA with GIB WEs to ARC, gLite, GlobusUnicore DCIs proxy management SHIWA Proxy Server

8. SHIWA Portal: Abstract Workflow Jobs Ports 8

9. SHIWA Portal: Configuring Workflow

10. SHIWA Portal: Executing Workflow

11. SHIWA Repository: Browse View search domain

12. SHIWA Repository: Browse View inputs outputs data sets implementation

13. SHIWA Repository: Table View Browse Workflows Browse Implementations Advanced Search Search Sort 13

14. SHIWA Repository: Execution Support Execution data 14

15. SHIWA Portal: Legacy Code Explorer Taverna workflow

16. SHIWA Portal: Repository Portlet Workflow Repository

17. CGI Developer Scenario: Specifying WE

18. CGI Developer Scenario: Specifying WF through portal

19. CGI Developer Scenario: Specifying WF through desktop

20. CGI User Scenario: Native WE DCIs Unicore DCI gLite DCI ARC DCI Globus DCI SHIWA Science Gateway SHIWA Repository ASKALON WE Galaxy WE WF1 WFn Triana WE Taverna WE GWES WE step 1search WF Kepler WE step 3retrieve WF data Pegasus WE WF1 WFm MOTEUR WE SHIWA Portal WE1 WEp step 2 edit WF ProActive WE PGRADE WE GEMLCA Repository WS-PGRADE Workflow editor step 5retrieve WF WE + WF Workflow Engines WF list user WS-PGRADE Workflow engine GEMLCA with GIB step 7 run WF step 6retrieve proxy GEMLCA Service step 4submit WF Proxy Server SHIWA Proxy Server 20

21. CGI User Scenario: Non-native WE DCIs Unicore DCI gLite DCI ARC DCI Globus DCI SHIWA Science Gateway SHIWA Repository ASKALON WE Galaxy WE WF1 WFn Triana WE Taverna WE GWES WE Kepler WE step 1 search WF GEMLCA Repository Pegasus WE WF1 WFm MOTEUR WE SHIWA Portal WE1 WEp ProActive WE step 5 retrieve WF PGRADE WE WS-PGRADE Workflow Editor user step 2 edit WF WE + WF Workflow Engines GEMLCA with GIB WS-PGRADE Workflow Engine step 4 forward WF step 7 run WF MOTEUR Workflow editor step 3 submit WF GEMLCA Service step 6 retrieve proxy MOTEURWorkflow Engine Proxy Server GEMLCA Client GEMLCA UI SHIWA Proxy Server

22. CGI User Scenario: WF Execution via Desktop step 1 search WF step 4edit WF SHIWA Science Gateway SHIWA Repository step 2 retrieve WF user WF1 WFn SHUWA Desktop step 5ask run WF GEMLCA Repository step 3forward WF bundle WF1 WFm SHIWA Portal WE WE1 WEp step 6submit WF WS-PGRADE Workflow Editor DCIs GEMLCA with GIB WS-PGRADE Workflow Engine Unicore DCI ARC DCI GEMLCA Service gLite DCI Globus DCI Proxy Server SHIWA Proxy Server

23. SSP DCI & FGI Infrastructure DCIs SHIWA Science Gateway Unicore DCI ARC DCI SHIWA Portal SHIWA Repository gLite DCI Globus DCI WF1 WFn GEMLCA admin GEMLCA Repository WF1 WFm ASKALON WE Galaxy WE Triana WE WS-PGRADE Workflow editor WE1 WFp Taverna WE GWES WE Kepler WE WS-PGRADE Workflow engine GEMLCA submitter DCI Bridge Pegasus WE MOTEUR WE ProActive WE JSDL Translator Meta Broker PGRADE WE Proxy Server Workflow Engines • SHIWA Science Gateway Resources native WE ASKALON, MOTEUR, WS-PGRADE local WEs: Kepler, MOTEUR , Taverna & Triana WEs portal WS-PGRADE v3.4.1 submit to the local cluster repository GEMLCA + SHIWA remote WEs: ASKALON, GWES & MOTEUR WEs GEMLCA ARC, gLite, Globus and Unicore DCIs proxy server

24. DCI Interoperability Scenario SHIWA Science Gateway SHIWA Portal SHIWA Repository WF1 WFn step 1 search WF GEMLCA Repository step 3retrieve WF data WF1 WFm step 2 edit WF WE1 WFp WS-PGRADE Workflow editor DCIs step 5forward WF Unicore DCI ARC DCI WS-PGRADE Workflow engine DCI Bridge JSDL Translator gLite DCI Globus DCI step 8submit WF step 6select DCI step 4translate WF Meta Broker step 7retrieve certificate Proxy Server 24

25. SHIWA Simulation Platform: Access • SHIWA Science Gateway (SHIWA Portal + SHIWA Repository) • joint portal and repository account given by the science gateway administrator • DCI resources • certificate given by the relevant DCI administrator, for example the VO sysadmin • Access to the SHIWA Portal • ssp.shiwa-workflow.eu • Access to the SHIWA Repository • repo.shiwa-workflow.eu 25

26. Conclusions • the simulation platform supports the whole lifecycle of workflows, i.e. creating, testing, uploading, browsing, downloading and running workflows • users can use workflows of their own and of other workflow systems through the same user interface, i.e. they can re-cycle or share workflows created by other research teams or even other research communities • workflow developers (or e-scientists) can create, run, upload workflows through the simulation platform • researchers (astro physicists, earth scientists, bio scientists, etc.) can browse the workflow repository, select and run workflows through the simulation platform • the simulation platform provides access to major European e-infrastructure, the European Grid Initiative (EGI) infrastructure which offers large pool of compute and storage resources