1 / 19

EGEE is a project funded by the European Union under contract INFSO-RI-508833

Middleware for the next Generation Grid Infrastructure Erwin Laure EGEE Deputy Middleware Manager A. Aimar, M. Barroso, S. Beco, P. Buncic, A. Di Meglio, A. Edlund, S. Fisher, D. Groep, L. Guy, F. Hemmer, P. Kunszt, M. Livny, O. Mulmo, F. Pacini, F. Prelz, M. Sgaravatto. CHEP 2004.

akando
Download Presentation

EGEE is a project funded by the European Union under contract INFSO-RI-508833

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Middleware for the next Generation Grid InfrastructureErwin LaureEGEE Deputy Middleware ManagerA. Aimar, M. Barroso, S. Beco, P. Buncic, A. Di Meglio, A. Edlund, S. Fisher, D. Groep, L. Guy, F. Hemmer, P. Kunszt, M. Livny, O. Mulmo, F. Pacini, F. Prelz, M. Sgaravatto CHEP 2004 www.eu-egee.org EGEE is a project funded by the European Union under contract INFSO-RI-508833

  2. Collaborations Global Grid Operations, Support and training Network infrastructure(GÉANT) The EGEE Project • EGEE offers the largest production grid facility in the world open to many applications (HEP, BioMedical, generic) • Existing production service based on LCG • Next generation open source web-services middleware being re-engineered taking into account production/ deployment/ management needs • Well-defined, distributed support structure to provide eInfrastructure that is available to many application domains • Middleware Activity • Re-engineer and harden Grid middleware • Provide production quality middleware www.eu-egee.org Chep 2004 - 2

  3. LCG-1 LCG-2 gLite-1 gLite-2 Globus 2 based Web services based Future EGEE Middleware - gLite • Intended to replace LCG-2 • Starts with existing components • Aims to address LCG-2 shortcoming and advanced needs from applications (in particular feedback from DCs) • Prototyping short development cycles for fast user feedback • Initial web-services based prototypes being tested with representatives from the application groups Application requirements http://egee-na4.ct.infn.it/requirements/ Chep 2004 - 3

  4. Guiding Principles • Lightweight (existing) services • Easily and quickly deployable • Use existing services where possible asbasis for re-engineering • Interoperability • Allow for multiple implementations • Perf/Scale. & Resilience/Fault Tolerance • Large-scale deployment and continuous usage • Portable • Being built on Scientific Linux and Windows • Co-existence with deployed infrastructure • Reduce requirements on participating sites • Flexible service deployment • Multiple services running on the same physical machine (if possible) • Co-existence with LCG-2 and OSG (US) are essential for the EGEE Grid service • Service oriented approach • Follow WSRF standardization • No mature WSRF implementations exist to-date so start with plain WS • WSRF compliance is not an immediate goal, but we follow the WSRF evolution • WS-I compliance is important Chep 2004 - 4

  5. gLite Services Service decomposition based on ARDA blueprint Chep 2004 - 5

  6. WMS Chep 2004 - 6

  7. CE • Works in push and pull mode • Site policy enforcement • Exploit new globus GK and CondorC (close interaction with globus and condor team) CEA … Computing Element Acceptance JC … Job Controller MON … Monitoring LRMS … Local Resource Management System Chep 2004 - 7

  8. Data Management • Scheduled data transfers (like jobs) • Reliable file transfer • Site self-consistency • SRM based storage Chep 2004 - 8

  9. Storage Element Interfaces • SRM interface • Management and control • SRM (with possible evolution) • Posix-like File I/O • File Access • Open, read, write • Not real posix (like rfio) Control SRM interface POSIXAPI File I/O User rfio dcap chirp aio dCache NeST Castor Disk Chep 2004 - 9

  10. Replica Catalog Site A Replica Catalog Site B LFN LFN GUID GUID SURL SURL SURL SURL Catalogs • File Catalog • Filesystem-like view on logical file names • Keeps track of sites where data is stored • Conflict resolution • Replica Catalog • Keeps information at a site • (Meta Data Catalog) • Attributes of files on the logical level • Boundary between generic middleware and application layer Metadata Catalog Metadata File Catalog GUID Site ID LFN Site ID Chep 2004 - 10

  11. Job wrapper Job wrapper Job wrapper MPP MPP MPP DbSP Information and Monitoring • R-GMA for • Information system and system monitoring • Application Monitoring • No major changes in architecture • But re-engineer and harden the system • Co-existence and interoperability with other systems is a goal • E.g. MonaLisa e.g: D0 application monitoring: MPP – Memory Primary Producer DbSP – Database Secondary Producer Chep 2004 - 11

  12. Security CredentialStorage myProxy Obtain Grid (X.509)credentials for Joe PseudonymityService (optional) 1. “Joe → Zyx” 2. tbd 3. AttributeAuthority “Issue Joe’sprivileges to Zyx” 4. Joe VOMS “The Grid” “User=Zyx Issuer=Pseudo CA” GSI LCAS/LCMAPS Chep 2004 - 12

  13. GAS & Package Manager • Grid Access Service (GAS) • Discovers and manages services on behalf of the user • File and metadata catalogs already integrated • Package Manager • Provides application software at execution site • Based upon existing solutions • Details being worked out together with experiments and operations Chep 2004 - 13

  14. VDT EDG . . . AliEn LCG . . . Approach • Exploit experience and components from existing projects • AliEn, VDT, EDG, LCG, and others • Design team works out architecture and design • Architecture: https://edms.cern.ch/document/476451 • Design: https://edms.cern.ch/document/487871/ • Feedback and guidance from EGEE PTF, EGEE NA4, LCG GAG, LCG Operations, LCG ARDA • Components are initially deployed on a prototype infrastructure • Small scale (CERN & Univ. Wisconsin) • Get user feedback on service semantics and interfaces • After internal integration and testing components are delivered to SA1 and deployed on the pre-production service Chep 2004 - 14

  15. From Prototype to Release • Prototype setup at 2 sites (CERN & Wisconsin) • ~45 users registered • 2nd release mid August • Many bugfixes • New functionalities • More nodes (more powerful) being added at CERN • EDG WMS added at CNAF • Being ported to SLC3 • Second VO being set up (core services at Wisconsin) • Will use globus RLS • Cf. following talk by Birger Koblitz Chep 2004 - 15

  16. From Prototype to Release • Continuous integration system set up • First results being put forward to testing team • SLC3 version of prototype middleware provided • SCM convergence being reached • SCM plan: https://edms.cern.ch/document/446241 • Common service configuration being worked out • Deployment scenarios being worked out • Testing sites (RAL, NIKHEF, CERN) up and running • Focus on prototype installation and testing • Plans being made for focused testing of gLite components for pre-production service • Test plan defined • Based upon architecture document, release plan, and NA4 requirements • https://edms.cern.ch/document/473264/ Chep 2004 - 16

  17. LCG LCG-2 (=EGEE-0) 2004 prototyping prototyping product 2005 product LCG-3 EGEE-1 gLite and LCG-2 LCG-2focus on production, large-scale data handling • Theservice for the 2004 data challenges • Provides experience on operating and managing a global grid service • Development programme driven by data challenge experience • Data handling • Strengthening the infrastructure • Operation, VO management • Evolves to LCG-3 as components progressively replaced with new middleware-- target is to minimise the discontinuities of migration to the new generation • Aim for migration plan by end of year gLitefocus on analysis • Developed by EGEE project in collaboration with VDT (US) • LHC applications and users closely involved in prototyping & development (ARDA project) • Short development cycles • Co-existence with LCG-2 • Profit as far as possible from LCG-2 infrastructure, experience  Ease deployment – avoid separate hardware • As far as possible - completed components integrated in LCG-2  improved testing, easier displacement of LCG-2 les robertson - cern-it-17 Chep 2004 - 17

  18. Summary • Next generation middleware being designed and assembled • Prototype first tangible outcome • BUT this is a PROTOTYPE not a release! • Architectural and design work well advanced • Architecture document sent to EU • Design document (draft) exists • Feedback from applications and operations essential! • Detailed release plan worked out • https://edms.cern.ch/document/468699 • First components for pre-production service during autumn • Continuous integration and testing scheme defined and adopted • Technology Risk • Will WS allow for all upcoming requirements? • Divergence from standards Chep 2004 - 18

  19. Links • JRA1 homepage • http://egee-jra1.web.cern.ch/egee-jra1/ • Architecture document • https://edms.cern.ch/document/476451/ • Release plan • https://edms.cern.ch/document/468699 • Prototype installation • http://egee-jra1.web.cern.ch/egee-jra1/Prototype/testbed.htm • Test plan • https://edms.cern.ch/document/473264/ • Design document • https://edms.cern.ch/document/487871/ • gLite homepage: • http://www.glite.org/ Chep 2004 - 19

More Related