1 / 17

VLDATA WP3: Compute Management

VLDATA WP3: Compute Management. Peter Kacsuk. Objectives.

meriel
Download Presentation

VLDATA WP3: Compute Management

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. VLDATAWP3: Compute Management Peter Kacsuk

  2. Objectives • To deal with high level computing tasks for data processing, like user driven workflows and data driven pipelines, as well as all types of simulation workflows, and any other stand-alone computing activities and the necessary components, tools and interfaces built on top of the framework to handle these activities. • Development and usage of concepts and tools for dynamic, flexible and intelligent management of compute resources for scientific data processing. • To create programmable, configurable, reliable, usable, and traceable platforms/services to facilitate this management in the back-end. • Create a 3-layer cyberspace where • users can create and integrate workflows via gateways and desktops • These workflows can access any kind of major DCIs VLDATA

  3. Concept to realize the objectives • The WP will build on the results of several previous EU FP7 projects as well as the Requirements analysis and Gap analysis reports of WP2. • It also considers future directions taking into account the possibilities of new computing platforms like clouds. • WP3 further develops the results of the SHIWA, ER-Flow and SCI-BUS projects in order to create a flexible, interoperable workflow development and execution environment that can be incorporated into science gateways and desktop solutions. • Science gateways will be created and maintained either in a static or a very dynamic way according to the demands of the user or user community using virtualization and cloud technologies. VLDATA

  4. Cyberspace infrastructure Cyberspace er Gateways Bio1 ChemistN Physics2 Kepler Taverna Galaxy WF systems Cloud EMI Grids Globus DCIs Combining SCI-BUS and SHIWA/ER-Flow technologies users can access and use many WFs and many infrastructures in an interoperable way no matter which is their home WF system

  5. Cyberspace infrastructure to access cloud resources via gateways, desktops and mobiles Cloud 2 Amazon Cloud 1 OpenNebula Cloud 3 OpenStack Cloud N IBM

  6. An example for cloud usage: PaaS service for building SaaS services SaaS service Developers Data management SCI-BUS Portlet Repository Job submission Workflow Template A DCI1 …. PaaS Gateway Portlet Portlet Portlet Portlet PaaS gen gateway SCI-BUS Generic Framework SCI-BUS marketplaces SaaS Services GaaS gateways Function Function Function Application Sequencing SHIWAWF Repository fMRI DCI1 MRI simulator NonLinSyst simulator DCIn DCI2 ….

  7. Example for a SaaS service generated by the PaaS service • User can start from the marketplace an autodock application stored as SaaS service in the marketplace • It is realized in the cloud as a customized gateway with the required BOINC infrastructure • It is scalable since new BOINC clients can be dynamically added to the BOINC server if it is required University Cloud BOINC Client (VirtualBox) Autodock application 3G Bridge (GBAC) BOINC Server (GBAC) BOINC Client (VirtualBox)

  8. Marketplace containing • SaaSs • CITS Generic gUSE workflow containing SaaS nodes and Cloud InfrastructureTemplates (CITs) nodes • SHIWA Repository containing • WFs • WF SaaSs WF SaaS CIT 1 MPI SEQ SEQ CIT 2 Kepler WF WS Call

  9. Major tasks Mobile access Workflow systems Gateways Cloud solutions Marketplace WS-PGRADE PaaS SaaS WS-PGRADE SHIWA InSilicoLab GaaS VLDATA

  10. Major tasks Workflow systems Task 1 WS-PGRADE/gUSE SHIWA technology Task 1.2 Task 1.1 Improved SHIWA Repo Optimized DIRAC execution SHIWA Repo with Cloud support Workflow provenance Better integration of SHIWA Repo and gUSE WF Editor Dashboard Improved WF interoperability WF extension for SaaS and CIT VLDATA

  11. Major tasks Gateways Task 2 WS-PGRADE/gUSE InSilicoLab Task 2.1 Integration Task 2.2 Task 2.3 Close integration with SCI-BUS Portlet Repo To be defined by Cyfronet Close integration with SHIWA WF Repo To be defined by Cyfronet Close integration with marketpalce To be defined by Cyfronet Improved DIRAC execution, logs and statistics To be defined by Cyfronet VLDATA

  12. Major tasks Cloud solutions Task 3 PaaS SaaS GaaS Task 3.2 Task 3.1 Task 3.3 Automatic generation of SaaS and put in marketpalce Automatic generation of SaaS and put in marketpalce Automatic scaling Automatic scaling Automatic scaling Integration with physical DCIs Integration with physical DCIs Integration with physical DCIs Integration with virtual DCIs Integration with virtual DCIs Integration with virtual DCIs VLDATA

  13. Major tasks Task 4 Marketplace Task 4.1 Automatic deployment Automatic scaling Integration with physical DCIs Integration with virtual DCIs Generating Cloud Infrastructure Templates Task 4.2 VLDATA

  14. Major tasks Task 5 Mobile access Workflow systems Gateways Cloud solutions Marketplace VLDATA

  15. Budget VLDATA

  16. Deliverables • D3.1: Improved WS-PGRADE workflow system, M12 • D3.2: Improved WS-PGRADE workflow system, M36 • D3.3: Improved SHIWA technology and SHIWA Repository, M24 • D3.4: SHIWA Repository/marketplace with cloud support, M48 • D3.5: WS-PGRADE/gUSE science gateway framework with improved DIRAC execution, logs and statistics, M18 • D3.6: WS-PGRADE/gUSE science gateway framework integrated with repositories, M30 • D3.7: WS-PGRADE/gUSE science gateway framework integrated with marketplace, M42 VLDATA

  17. Deliverables • D3.8: GaaS solution, M12 • D3.9: PaaS solution, M24 • D3.10: SaaS solution, M36 • D3.11: One-stop-shop Marketplace, M18 • D3.14: Cloud infrastructure templates in the marketplace, M24 • D3.15: Mobile access to WP3 technologies, M42 VLDATA

More Related