1 / 35

The EGEE project and the future of European grids

The EGEE project and the future of European grids. Guy Wormser Director of “Institut des Grilles du CNRS” Aspera Workshop. PROGRAMS. OPERATING SYSTEM. Disks, CPU etc. Grid: Resource Sharing. Share more than information Data, computing power, applications. Middleware handles everything.

klaus
Download Presentation

The EGEE project and the future of European grids

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The EGEE project and the future of European grids Guy Wormser Director of “Institut des Grilles du CNRS” Aspera Workshop

  2. PROGRAMS OPERATING SYSTEM Disks, CPU etc Grid: Resource Sharing • Share more than information • Data, computing power, applications • Middleware handles everything Your Program The Grid Single computer MIDDLEWARE User Interface Machine Your Program Word/Excel Games Email/Web Resource Broker Disk Server CPU Cluster CPU Cluster Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  3. Distribution Infrastructure 'Standard Interface' Electricity Grid Analogy with the Electricity Power Grid Power Stations Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  4. EGEE : Enabling Grids for E-sciencE • Goal • create a general European Grid production quality infrastructure on top of present and future EU RN infrastructure • Build on • EU and EU member states major investment in Grid Technology • Several pioneering prototype results • Largest Grid development team in the world • Goal can be achieved for about €100m/4 years on top of the national and regional initiatives • Approach • Leverage current and planned national and regional Grid programmes (e.g. LCG) • Work closely with relevant industrial Grid developers, NRNs and US applications EGEE Geant network Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  5. The Large Hadron Collider Project 4 detectors CMS ATLAS LHCb Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  6. Bat 40 Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  7. New solutions are necessary! Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  8. GRID . INFRASTRUCTURE KNOWLEDGE . INFRASTRUCTURE NETWORK . INFRASTRUCTURE How e-Infrastructrures help e-Science • e-Infrastructures provide easier access for • Small research groups • Scientists from many different fields • Remote and still developing countries • To new technologies • Produce and store massive amounts of data • Transparent access to millions of files across different administrative domains • Low cost access to resources • Mobilise large amounts of CPU & storage on short notice (PC clusters) • High-end facilities (supercomputers) • And help to find new ways to collaborate • Develops applications using distributedcomplex workflows • Eases distributed collaborations • Provides new ways of community building • Gives easier access to higher education Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  9. EGEE Main Objectives Operate a large-scale, production quality grid infrastructure for e-Science Attract new resources and users from industry as wellas sciences • Flagship grid infrastructure project co-funded by the European Commission • Now in 2nd phase with 91 partners in 32 countries Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  10. EGEE – What do we deliver? • Infrastructure operation • Currently includes ~250 sites across 45 countries • Continuous monitoring of grid services & automated site configuration/management • Support many Virtual Organisations from diverse research disciplines • Middleware • Production quality middleware distributed under business friendly open source licence • Implements a service-oriented architecture that virtualises resources • Adheres to recommendations on web service inter-operability and evolving towards emerging standards • User Support - Managed process from first contact through to production usage • Training • Expertise in grid-enabling applications • Online helpdesk • Networking events (User Forum, Conferences etc.) Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  11. 240 sites 45 countries 41,000 CPUs 5 PetaBytes >5000 users >100 VOs >100,000 jobs/day • Archeology • Astronomy • Astrophysics • Civil Protection • Comp. Chemistry • Earth Sciences • Finance • Fusion • Geophysics • High Energy Physics • Life Sciences • Multimedia • Material Sciences • … 32% Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  12. Types of applications • Simulation • LHC Monte Carlo simulations; Fusion; WISDOM • Jobs needing significant processing power; Large number of independent jobs; limited input data; significant output data • Bulk Processing • HEP ; Processing of satellite data • Distributed input data; Large amount of input and output data; Job management (WMS); Metadata services; complex data structures • Parallel Jobs • Climate models, computational chemistry • Large number of independent but communicating jobs; Need for simultaneous access to large number of CPUs; MPI libraries • Short-response delays • Prototyping new applications; grid Monitoring grid; Interactivity • Limited input & output data; processing needs but fast response and quality of service • Workflow • Medical imaging; flood analysis • Complex analysis algorithms; complex dependencies between jobs • Commercial Applications • Non-open source software; Geocluster (seismic platform); FlexX (molecular docking); Matlab, Mathematics; Idl, … • License server associated to an application deployment model Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  13. Lab m Uni x USA Brookhaven Uni a UK USA FermiLab Lab a France Tier 1 Physics Department Uni n CERN Tier2 ………. Italy Desktop Lab b Germany NL Lab c  Uni y Uni b   LHC Computing Model The LHC Computing Centre les.robertson@cern.ch Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  14. SEISMOLOGY[1] Fast Determination of mechanisms of important earthquakes (IPGP: E. Clévédé, G. Patau) Challenge Provide results 24h -48h after its occurrence 5 Seisms already ported: Peru, Guadeloupe, Indonesia (Dec.), Japon, Indonesia (Feb.) • Application to run on alert • Collect data of 30 seismic stations from GEOSCOPE worldwide network • Select stations and data • Definition of a spatial 3D grid +time • Run for example 50-100jobs Peru earthquake, 23/6/2001, Mw=8.3 Data used: 15 Geoscope Stations Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  15. Korba coastal aquifer Management of water resources in Mediterranean area (SWIMED) G. Lecca (CRS4 Italy), P. Renard (Unine, CH), J. Kerrou (INAT, Tunisia), R. Ababou (IMFT, Fr) Tunisia 45 km Cape Bon Peninsula 70km south-east of Tunis Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  16. GEOSCIENCES • Generic seismic platform software, based on Geocluster commercial software developed by CGG • Includes 400 geophysical modules, implemented on EGEE • Used by both academics and private companies. • Free of charge for Academics, with charge for R&D Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  17. GATE GEANT4 Application to Tomography Emission • Scientific objectives Radiotherapy planning for improving the treatment of cancer by ionizing radiations of the tumours. Therapy planning is computed from pre-treatment MR scans by accurately locating tumours in 3D and computing radiation doses applied to the patients. • Method GEANT4 base software to model physics of nuclear medicine. Use Monte Carlo simulation to improve accuracy of computations (as compared to the deterministic classical approach) Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  18. Drug Discovery • WISDOM focuses on in silico drug discovery for neglected and emerging diseases. • Malaria — Summer 2005 • 46 million ligands docked • 1 million selected • 1TB data produced; 80 CPU-years used in 6 weeks • Avian Flu — Spring 2006 • H5N1 neuraminidase • Impact of selected point mutations on eff. of existing drugs • Identification of new potential drugs acting on mutated N1 • Fall 2006 • Extension to other neglected diseases Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  19. High Throughput Virtual Docking Chemical compounds : ZINC Molecular docking : FlexX, Autodock Targets structures : PDB Grid infrastructure : EGEE Chemical compounds : Chembridge – 500,000 Drug like – 500,000 Millions of chemical compounds available in laboratories High Throughput Screening 1-10$/compound, nearly impossible Molecular docking (FlexX, Autodock) ~80 CPU years, 1 TB data Computational data challenge ~6 weeks on ~1000/1600 computers Targets : Plasmepsin II (1lee, 1lf2, 1lf3) Plasmepsin IV (1ls5) Hits screening using assays performed on living cells Leads Clinical testing Drug Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  20. Storage Element Computing Element Computing Element Grid workflow • FlexX license server : • 3000 floating licenses given by BioSolveIT to SCAI • Maximum number of used licenses was 1008 Results Compounds list Software Site1 Statistics Parameter settings Target structures Compounds sublists User interface Site2 Compounds database Storage Element Software Results Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  21. gPTM3D 3D Medical Image Analysis Software • Scientific objectives Interactive volume reconstruction on large radiological data. PTM3D is an interactive tool for performing computer-assisted 3D segmentation and volume reconstruction and measurement (RSNA 2004) Reconstruction of complex organs (e.g. lung) or entire body from modern CT-scans is involved in augmented reality use case e.g. therapy planning. • Method Starting from an hand-made rough Initialization,a snake-basedalgorithm segments each slice of a medical volume. 3D reconstruction is achieved in parallel by triangulating contours from consecutive slices. Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  22. Grids key compettive advantages • Transparent access to distributed data • Exemples Earth sciences, Life sciences • Handling of huge datasets • Physique des particle Physics, astrophysics, human sciences • Large flexibility in computing ressources • Disasters management • Avian flu, malaria challenges • Synergy between the grid network and the human network  Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  23. EGEE User Forum 2007 user forum:~30 sessions100+ presentations20 demos~60 posters Co-located with OGF20: 900+ attendees~50 booths Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  24. Infrastructures geographical or thematic coverage Support Actions key complementary functions Applications improved services for academia, industry and the public Registered Collaborating Projects 25 projects have registered as of September 2007:web page Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  25. Collaborating infrastructures Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  26. EGEE-II to EGEE-III • EGEE-III proposal currently under negotiation with European Commission • Key objectives • Expand/optimise existing EGEE infrastructure, include more resources and user communities • Prepare migration from a project-based model to a sustainable federated infrastructure based on National Grid Initiatives • 2 year period – spring 2008 to spring 2010 • No gap between EGEE-II and EGEE-III • Similar consortium • Now structured on a national basis (National Grid Initiatives/Joint Research Units) Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  27. European Grid Initiative • Need to prepare permanent, common Grid infrastructure • Ensure the long-term sustainability of the European e-Infrastructure independent of short project funding cycles • Coordinate the integration and interaction between National Grid Infrastructures (NGIs) • Operate the production Grid infrastructure on a European level for a wide range of scientific disciplines Must be no gap in the support of the production grid Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  28. http://www.eu-egi.org • EGI Design Study proposal approved to the European Commission (started 1st September’07) • Supported by 30+ National Grid Initiatives (NGIs) • 2 year project to prepare the setup and operation of a new organizational model for a sustainable pan-European grid infrastructure • Federated model bringing together NGIs to build a European organisation • Well defined, complimentary responsibilities between NGIs and EGI Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  29. Characteristics of NGIs Each NGI • recognized national body with a single point-of-contact • mobilise national funding and resources • operate the national e-Infrastructure • support user communities (application independent, and open to new user communities and resource providers) • contribute and adhere to international standards and policies Responsibilities between NGIs and EGI are clearly separated and complementary Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  30. 37 European NGIs + Asia, US, Latin America + PRACE + OGF-Europe + … Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  31. Why a « Grid Institute » • Considerable importance of Grids related activity within CNRS • Federate all activities in CNRS related to research on Grids, grids for research and production grids • Better visibility • Meilleure efficiency • Strengthen the links between these domains • Provide a single well identified Point of contact for national and international collaborations • CNRS representative for all european contracts, for disucssion within french Ministry • central core for the emerging French « National Grid Initiative » • Parternship with all major French reserach organisms, CEA, CNES, INRIA, … • Outreach activities, evangelisation of new scientific communities, training,.. Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  32. Institut des Grilles composition • 30 laboratories: APC, CC_IN2P3, CPPM, CREATIS, LIP, I3S, IBCP,IN2P3_adm, IPGP, IPHC, IPNL, IPNO, IRISA, IRIT, LABRI, LAL, LAPP, LIFL, LIG, LIP6, LLR, LORIA, LPC Clermont, LPNHE, LRI, IPSL, LPSC, LSIT, Subatech, UREC • 13 IN2P3 labs linked to EGEE/LCG • 11 computing science labs • 5 labs linked to various applications • Administrative support • GDR Architecture Systèmes et Réseaux (ASR) • Total of 310 people Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  33. IdG objectives and means • Scientific animation • Organization of the national prospective on the needs of the scientific community related to Grids • Call for proposals • Dialog Forum between Grilles production grids and research grids • Interoperability GRID5000/EGEE • Grids Observatory • Middleware of the future • Training • Communication Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  34. The national prospective working groups • Thematic working groups • Planet and Universe sciences; environmental sciences • Life sciences • Human sciences • Chemistry • Physics • Engineering and computing sciences • Subatomic physics (including astroparticles) • Transverse working groups • Data grids • Grids and supercomputers • Regional grids • Grids and very large research infrastructures • Grids and the user • Relationship with industry Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

  35. Summary • Grids are all about sharing – they are a means of working with groups around the world • Today we have a window of opportunity to move grids from research prototypes to permanent production systems (as networks did a few years ago) • Interoperability is key to providing the level of support required for our user communities • EGEE operates the world’s largest multi-disciplinary grid infrastructure for scientific research • In constant and significant production use • A third phase of EGEE is under preparation • Need to prepare the long-term • EGEE, collaborating projects, national grid initiatives and user communities are working to define a model for a sustainable grid infrastructure that is independent of short project cycles • In France, creation of the CNRS Grid Institute as the nucleus of the French NGI to help implementing this future! • Bright opportunities for grids and astroparticles! www.eu-egee.org Inauguration de l'Institut des Grilles, Paris, 3rd December 2007

More Related