1 / 27

Agile Computing Federation and Slovenian plans for EINFRA-1

This presentation discusses the Slovenian Initiative for National Grid and their plans for integrating hybrid grid/cloud computing, addressing the challenges of resource management, user access, and architectural alternatives. It also explores the goals, necessary steps, and hard problems associated with this integration, as well as current involvements and future projections.

cfeinberg
Download Presentation

Agile Computing Federation and Slovenian plans for EINFRA-1

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Agile Computing Federationand Slovenian plans for EINFRA-1 Jan Jona Javoršek Jožef Stefan Institute jona.javorsek@ijs.siSLING – Slovenian Initiative for National Grid Jožef Stefan Institute http://www.ijs.si/ http://www.sling.si/

  2. Slovenian Point of View • Strong HEP basesome large international projects • NREN support, NGI, EGI membership, official gvt. support, equipemnt funding promises • Increasingly integrated ARC-usingHTC users and centres • ARC-using users from other domains(KT, biomedical, mathematics, statistis, applied linguistics)

  3. Slovenian Point of View • Blocked PRACE, politicking Cloud.eu • Centre integration / funding pressures(no central management) • NREN: public cloud expectationsEGU Gain integration • FUD regarding lack of EGI planning, envisioned grid future, CERN standing, distributing computing future • Huge demands for future projects (Belle II) • Expectations of uninterrupted functionality

  4. Agile Computing Federation • Goals: • Reuse of existing technologies • Reuse of existing infrastructure:institutional, national, international • No interruption of existing services • Unified user access regardless of tech • Coordination with ongoing efforts

  5. Similar infrastructures • Available resourcesfor researchers: • National andinternational grids • HPC centres • Private and public clouds

  6. Computing Centre Dilemma • Different situations vs. size: • Multiple independant infrastructures • Multiple centres in the same instututions, with different networks • Multiple infastructures in the same centres and network • Resource reuse / reallocation / pain

  7. Confusing access paths • HPC centres: application policies and local comitees • Grid: PKI, certificates and VOs • Cloud: different APIs, PR confusion, vendor/site isolation

  8. Parallel efforts • NREN / large institute can have: • Custom services • Internal cloud • HPC centre • 1 or more grid sites • Public/private cloud One agile setup required!

  9. Required components • Provisioning system • Local resource management • User management, authorization • Runtime envs / modules / VM images • Information system, brockering • Service mngt, check-pointing, transfer • Data management

  10. CVMFS Choices... Salt KeyStone Ceph CERN Agile model Globus NorduGrid ARC gLite Cinder PKI VOMS dCache Torque OpenMP SLURM OpenStack gFTP Glance OpenNebula oVirt Puppet science portals VRC

  11. Architectural Alternatives • Grid/Cloud integration • Unified Grid/Cloud access • Virtualized Grid Services • Integrated hybrid Grid/Cloud

  12. Goals • No interruption in existing services: grid, HPC, cloud • Soft transitions: • Glite → ARC • oVirt → OpenStack • PBS → SLURM • Unification of stack:for admins (agile stack, monitoring, repos)for users (auth/z, RTE/images, projects)

  13. User Satisfaction • Single point of access:(EduGain + VOMS + Keystone?) • Single interface(Horizon + ARC?) • Flexible runtime environment or image repository(ARC RTE + Glance: VM vs RTE,OpenMP, OpenCL)

  14. Necessary steps • Provisioning, abstraction and birtualization of resources and services • Hybrid resource manager • Customization of infosys:predefined instance flavors,service registration, quotas • Storage and data management abstractions

  15. Hard Problems • Data management (hard) • Workflow managment (harder) • Task management(hardest)

  16. Micro and Macro Climate • H2020 ARC development project(with 2 nordic parners, ATLAS interest, interest for EU-T0) • H2020 Dirac development project(Belle2 involvement) • National projects(ongoing HTC Puppet – HTC enabling grid-compatible provisoning and agile enhancement) • National commitment: equipment, technical stuff, operation costs, int. project cofunding • Continued EGI, HTC, Géant, WLCG • No clear HPC / PRACE / HTC COE / Cloud l

  17. Involvements • CERN: ATLAS • Belle2, Pierre Auger • Support existing groups(KT, biomed, HPC/fluid dynamic) • Interest (NG, Finland, Hungary, SE EU) • Close/Opportunistic: OpenArchive, Clarin, Dariah ...

  18. Projection • 1Y: Experimental setup:large institute (multiple sites) • 2Y: National federation:NREN + NGI + HPC centres • 3Y: International federation:Project, other groups / projects • 4Y: „hard“ problems

  19. Questions? • ? Jan Jona Javoršek Jožef Stefan Institute jona.javorsek@ijs.siSLING – Slovenian Initiative for National Grid http://www.ijs.si/ http://www.sling.si/

  20. Additional Slides

  21. Grid and Cloud integration • VM managers = LRMS • Virtualized WN • Virtualized storage • Example: WnoDes

  22. Unified Grid/Cloud access • One client used to submit both jobs and servies • virtualized WN • Adapted LRMS (grid & cloud) • Shared Cloud Storage • Examples: Swarm, XtremWeb

  23. Virtualized Grid Services • Virtualization of all grid services • Changes in infosys and registration services • Cloud instances part of grid • Example: none

  24. Integrated hybrid G / C • Transparent acces of resources: grid and cloud • LRMS for grid and cloud • Monitoring for both resources • Optionnaly virtual WN

  25. Integrated hybrid G / C

  26. Objective?

  27. EduGAIN VOMS Keystone Authorization: Role Discovery / ProvisioningComputing + Data Native Job:CL / MPI VM Job VM service Slurm Compute Quota / Accounting Possibilities

More Related