1 / 34

A Virtualised European Grid Infrastructure?

A Virtualised European Grid Infrastructure?. Steven Newhouse Project Director, EGI.eu. What is EGI?. European: Focus on meeting the needs of European Researchers Their key need is to collaborate internationally Grid: Federation of shared resources, not a single technology

liona
Download Presentation

A Virtualised European Grid Infrastructure?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Virtualised European Grid Infrastructure? Steven Newhouse Project Director, EGI.eu UK eScience Forum 2010

  2. What is EGI? • European: • Focus on meeting the needs of European Researchers • Their key need is to collaborate internationally • Grid: • Federation of shared resources, not a single technology • Supports a collaborative social network • Infrastructure: • Established physical and organisational structures • Services and facilities for an economy to function UK eScience Forum 2010

  3. European Grid Infrastructure • European Data Grid (EDG) • Explore concepts in a testbed • Enabling Grid for E-sciencE (EGEE) • Moving from prototype to production • European Grid Infrastructure (EGI) • Routine usage of a sustainable e-infrastructure UK eScience Forum 2010

  4. European Grid Infrastructure Status April 2010 (yearly increase) • 10000 users: +5% • 243020 LCPUs (cores): +75% • 40PB disk: +60% • 61PB tape: +56% • 15 million jobs/month: +10% • 317 sites: +18% • 52 countries: +8% • 175 VOs: +8% • 29 active VOs: +32% UK eScience Forum 2010

  5. Are users willing to follow? Sustainability Cost Technology Relevance UK eScience Forum 2010

  6. Sustainability My project will last for 5 years, and I will need to read and analyse my data for 10 years after that…. Will you be around that long? UK eScience Forum 2010

  7. EGI Collaboration Research Community Research Community Research Community N G I Research Community Research Community N G I N G I E I R O N G I E I R O EGI.eu UK eScience Forum 2010

  8. EGI.eu Governance • EGI.eu established as non-profit foundation • Governance & ownership by its participants • Participants: • European NGIs • Associated participants: • Other groups willing to agree with EGI.eu objectives • EGI Council: All participants represented • Votes linked to fees UK eScience Forum 2010

  9. EGI.eu • Coordination for European Grid resources • Established February 8th 2010 • Central policy & services needed to run a grid • Sustainable small coordinating organisation • Based in Amsterdam • Coordinating core (~20 people) in Amsterdam • Technical services from partners (~20 people) • EGI and EGI.eu supported initially by EGI-InSPIRE UK eScience Forum 2010

  10. The EGI-InSPIRE Project Integrated Sustainable Pan-European Infrastructure for Researchers in Europe • A 4 year project with €25M EC contribution • Project cost €72M • Total Effort ~€330M • Effort: 9261PMs Funded Un-Funded • Project Partners (50) • EGI.eu, 40 NGIs, 2 EIROs • Asia Pacific (8 partners) UK eScience Forum 2010

  11. A Virtuous Service Cycle Users represented by Virtual Research Communities in the User Community Board Production quality infrastructure built from technology deployed across Europe USAG: User Services Advisory Group OTAG: Operational Tools Advisory Group Resource providers represented by NGIs/EIROs in the Operations Management Board New technology releases assessed against defined criteria before being deployed Technology Coordination Board prioritised for Technology Providers UK eScience Forum 2010

  12. EGI Supports Innovation • Deploy Technology Innovation • Distributed Computing continues to evolve • To include: Grids, Desktops, Virtualisation, Clouds, … • Enable Software Innovation • Provide reliable persistent technology platform • Tools built on gLite/UNICORE/ARC/Globus • Support Research Innovation • Infrastructure for data driven research • Support for international research (e.g. ESFRI) UK eScience Forum 2010

  13. Relevance OK so you have processes…. How are you going to help me do what’s relevant to me and my community? Human Network Representation in the e-Infrastructure 3) COMMUNITY UK eScience Forum 2010

  14. Scaling the Human Network • Across an NGI Research Community • Across an International Research Community • Across different disciplines & skill levels • From an Individual to a Community • Services that enable the human network Services that support building your Virtual Research Community UK eScience Forum 2010

  15. An Evolving Relationship User Services Advisory Group User Community Support Team (EGI.eu) Defined by MoU UK eScience Forum 2010

  16. Our Interaction with VRCs • VRCs link EGI to a community structure • Defined by MoU, conference, organisation, … • A focus for collecting requirements for: • Human and Technical services (e.g. Training) • Discipline specific support (e.g. Bio Apps) • Operation teams in the resource centres It is the human collaborative side of the grid UK eScience Forum 2010

  17. A Virtuous User Cycle Feedback through VRCs in the User Community Board • Integrated Services • Human • Technical • Infrastructure User Services Advisory Group to drive detailed design UK eScience Forum 2010

  18. Current Technical Services • Training Events Calendar • Training Material Repository • Applications Database • Services for your community • Membership: VOMS, VO registration database • Access: Basic portals • Status: Dashboards UK eScience Forum 2010

  19. Current Human Services • Building a community for all users • Two Annual meetings: Users & Technology • For Heavy User Communities • Expand community specific services • Evolving the technology • Definition and verification of requirements • Liaison with software providers • Communication • With NGIs, VRCs, SSCs and other projects UK eScience Forum 2010

  20. Why become a VRC? • User Community Board • Policy, procedures and technical management • User Services Advisory Group • Technical evolution of the provided services • Share your experiences with others • Integrate your resources into production • Access a European support infrastructure UK eScience Forum 2010

  21. Technology I’ve told you what I want, how are you going to support my exact needs, now and in the future, instead of my competitors? UK eScience Forum 2010

  22. Be a Neutral Infrastructure • Consider IP network providers • Supports traffic from different communities • Customised solutions within a generic framework • Standards drive integrated deployment • And for sustainable e-Infrastructures? • Any application, any domain, any technology • A platform for domain specific innovation & use • Integration of any compliant resource UK eScience Forum 2010

  23. European e-Infrastructure Applications Networking Resources HPC Grids Desktops HTC Grids Petascale HPC EGI EGEE PRACE Virtualised Resources UK eScience Forum 2010

  24. The DCI Projects UK eScience Forum 2010

  25. DCI Taxonomy EGI-InSPIRE Operational Infrastructure StratusLab EDGI Users Venus-C IGE EMI Technology Resources UK eScience Forum 2010

  26. Can we learn from others? • Grids have benefited from commoditisation • Hardware: HTC & HPC affordable to all • Networking: GBs can be moved over WAN • Software: Open source software comes of age • Impacts of commodity virtualisation… • For transactional models  • The ‘Cloud’: A model based on compute not data • For large distributed data-oriented models  • The emergence of true ‘function shipping’? UK eScience Forum 2010

  27. Dealing with Virtualisation • Who makes your Virtual Machine Images? • Who checks that they will run anywhere? • Who checks that they are running? • Who checks that they are secure? New technology requires new expertise UK eScience Forum 2010

  28. A Virtualised Future? Experts (Communicating between users & providers) VO Specific Operations Staff End-Users (National, European & Global Collaborations) End-User Technology Experts (National, European & Global Collaborations) Virtual Research Community Infrastructure Providers Infrastructure Providers (Research & Commerical– National, European & Global) UK eScience Forum 2010

  29. New Research Challenges • Virtualised Resources across Europe • Federated Cloud: partnership agreements • Multi-clouds: co-ordinated access • Consolidation in data centres & commercial integration • Provisioning of experimental infrastructure • Deploy on demand for 1 day  18 months • Record provenance so it can be replayed • Self-healing to lower management burden • Security issues still remain • Site: Trusting the software in the VM • Network: Link VMs across WAN by dedicated networks • Data: Where is my data going & who has access? SSAI Collaboration Meeting 2010

  30. Cost This sounds wonderful! How much is it going to cost me? UK eScience Forum 2010

  31. Current Costs UK eScience Forum 2010

  32. Running EGI.eu (4 yrs) EGI.eu European Commission EGI-InSPIRE (25000K) EGI-InSPIRE Central Budget (597K) 597K 4653K EGI.eu Activity (8010K) This moves EGI.eu towards sustainability 2643K 19750K 6000K NGI Activity Partner Tasks (International & Global Tasks) (65848K) NGIs (49455K) 43455K UK eScience Forum 2010

  33. Summary • EGEE: • Demonstrated a production e-infrastructure • EGI: • Provide a sustainable production e-infrastructure • EGI.eu established in Amsterdam • Supported transition through EGI-InSPIRE • Moving towards sustainable coordination UK eScience Forum 2010

  34. Questions? • Call for papers opens October 2010 • See www.egi.eu/EGIUF2011 for more details • Contact: • Steven Newhouse • director@egi.eu UK eScience Forum 2010

More Related