CloudBroker Platform integration into WS-PGRADE/gUSE

1. CloudBroker Platform integration into WS-PGRADE/gUSE Zoltán Farkas MTA SZTAKI LPDS zoltan.farkas@sztaki.mta.hu

2. Outline • Motivation • WS-PGRADE/gUSE • CloudBroker Platform • Integration: • Architecture, features • DCI Bridge modifications • WS-PGRADE modifications • WFI modifications • SaaS vs IaaS execution modes

3. Motivation There are many user communities who would like to access compute resources in a transparent way through science gateways Cloud resources are getting more and more popular Clouds are more reliable than grids Additionally, cloud resource can provide a vast amount of resources (flexible resource allocation) A customizable science gateway with cloud support is capable of satisfying compute needs of complex scientific computations Aim: provide access to cloud resources in a transparent way

4. WS-PGRADE/gUSE in SCI-BUS

5. The CloudBroker Platform Easy, scalable, secure, integrable and pay-per-use access to scientific and technical applications in the cloud High performance computing application store and marketplace with accounting and billing Deployment and execution of applications in the cloud with the same parameters and files as for local execution Accessible through the internet via a web browser UI Integration possibilities into third party tools via web service-based APIs Public version available under https://platform.cloudbroker.com Using infrastructure as a service (IaaS) from cloud providers Offering platform as a service (PaaS) for software vendors Providing software as a service (SaaS) to end users Available compute resources: Amazon, IBM, OpenStack, OpenNebula Available storage resources: Amazon, Walrus, Rados S3

6. CloudBroker Platform Architecture R&D End Users and Software Vendors CLI WebBrowserUI GenericWorkbenches Domain-Specific Gateways Web Service API CloudBroker Platform CloudBroker Integration ChemistryAppli-cations …Appli-cations BiologyAppli-cations MedicineAppli-cations Engineering Appli-cations …Cloud AmazonCloud IBMCloud

7. Integrated architecture

8. Integration features I. Slight modification in the fully featured UI Also available in every customization level, completely hiding the fact that cloud is being used: ASM Remote API End-user view DCI Bridge

9. Integration features II. Support for commercial clouds with costs (prices configured in CloudBroker Platform): Estimated job cost before submission Actual job and workflow cost after execution

10. Integration features III.

11. DCI Bridge modifications I. New Configuration tab for CloudBroker

12. DCI Bridge modifications II. New CloudBroker plugin to manage CloudBroker jobs Using slightly extended JSDL<JobDefinition xmlns="http://schemas.ggf.org/jsdl/2005/11/jsdl" xmlns:ns2="http://schemas.ggf.org/jsdl/2005/11/jsdl-posix" xmlns:ns3="uri:MBSchedulingDescriptionLanguage" xmlns:ns4="extension.dci" xmlns:ns5="http://cloud.extension.jsdl" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://schemas.ggf.org/jsdl/2005/11/jsdl"> <JobDescription> <JobIdentification> <JobName>Workflow Test Applications 1.0</JobName> <JobAnnotation>1f803ac0-8cfd-47de-9b9a-718f2a287f20</JobAnnotation> </JobIdentification> <Application> <ApplicationName>Workflow Test Applications 1.0</ApplicationName> <ApplicationVersion>Workflow Test Applications 1.0 ARInew.sh</ApplicationVersion> <ns2:POSIXApplication_Type> <ns2:Executable>Workflow Test Applications 1.0 ARInew.sh</ns2:Executable> <ns2:Argument>A</ns2:Argument> <ns2:Output>job.out</ns2:Output> <ns2:Error>job.err</ns2:Error> <ns2:UserName>10196</ns2:UserName> <ns2:GroupName>http://192.168.152.73:8080/wspgrade</ns2:GroupName> </ns2:POSIXApplication_Type> </Application> <Resources> <ns5:CloudResources_Type> <ns5:CloudResource> <ns5:ResourceName>OpenNebula MTA SZTAKI SZTAKI</ns5:ResourceName> <ns5:RegionName>OpenNebula MTA SZTAKI SZTAKI SZTAKI OpenNebula</ns5:RegionName> <ns5:InstanceTypeName>OpenNebula MTA SZTAKI SZTAKI SZTAKI OpenNebula x86_64</ns5:InstanceTypeName> </ns5:CloudResource> </ns5:CloudResources_Type> </Resources>

13. WS-PGRADE modifications I. CloudBroker Authentication Portlet

14. WS-PGRADE modifications II. IaaS and SaaS-based configuration

15. WFI modifications Produce JSDL document for DCI Bridge based on workflow XML containing CloudBroker-specific settings

16. Accessible Cloud Resources Access provided by the CloudBroker Platform Commercial: Amazon EC2 IBM OpenSource/Free: OpenStack OpenNebula Currently, within SCI-BUS accessible: MTA SZTAKI OpenNebula (400 cores) BIFI OpenStack (50 cores)

17. Example applications - IaaS University-level cloud resource access Run any application

18. Example applications - SaaS Predeployed application execution Docking portal: AutoGrid, AutoDock, AutoDock Vina (should be deployed)

19. Summary WS-PGRADE/gUSE offers production-level cloud access through the CloudBroker Platform Mainstream cloud resources can be used (both IaaS and SaaS model is supported) Slight UI changes (can be completely hidden for end-users) Commercial exploitation available

20. Questions? ?