1 / 24

Grid Interoperability

Grid Interoperability. Track: Comp_3. Workshop FJPPL'07. Introduction to the Grid model. Nowadays the Grid concept is unavoidable for any application needing significant computing resources Grid allows to share CPU and storage from several sites operating heterogeneous hardware and software

veliak
Download Presentation

Grid Interoperability

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grid Interoperability Track: Comp_3 Workshop FJPPL'07

  2. Introduction to the Grid model • Nowadays the Grid concept is unavoidable for any application needing significant computing resources • Grid allows to share CPU and storage from several sites operating heterogeneous hardware and software • The software layer which allows sites to exchange computing tasks through the grid is called "Middleware" • A grid can be • CPU oriented (SETI / BOINC) • Data oriented (Storage Resource Broker) • Both (EGEE – W-LCG) D. Boutigny

  3. The W-LCG grid model Annecy ICEPP Marseille TRIUMF FZK CC-IN2P3 T3 (many) Nantes T1 (11) RAL ASCC CNAF Île de France T0 Brookhaven PIC T2 (~100) NIKHEF Lyon Fermilab NDGF Strasbourg Clermont D. Boutigny

  4. Many kind of Grids • The Global Grid Forum web site is mentioning ~45 major different grid organizations • W-LCG is federating resources from 3 different Grids: • EGEE (Europe + Asia + Canada) • OSG (US) • NorduGrid (Nordic countries) • NAREGI (Japan - ?) • Having 1 single grid is unrealistic • One has to deal with several Grids and provide interoperability • From the user point of view • From the sites' point of view NAREGI (National Research Grid Initiative) is a Grid being developed in Japan D. Boutigny

  5. Project presentation • The LIA "Comp_3" project is proposing to work toward Grid Interoperability • Was an accepted project for the 2006 call • We propose to extend the project for 2007 • Work will concentrate mainly on • EGEE / NAREGI interoperability • Crucial for ILC • Will become important for LHC • NAREGI is a huge effort in Japan, and will certainly become a piece of the W-LCG organization • SRB / iRODS data grid (see later) • Implementation • Development • Interoperability with EGEE / NAREGI NAREGI: 2003-2007 10 billion Yen Extended up to 2009 D. Boutigny

  6. Japan S. Kawabata T. Sasaki G. Iwai K. Murakami Y. Iida France D. Boutigny S. Reynaud F. Hernandez J.Y. Nief Members D. Boutigny

  7. Activities during the first year • 2 workshops • 3 days in September 2006 in Lyon – 6 people from Japan • 4 days in February 2007 at KEK – 5 people from France D. Boutigny

  8. NAREGI at KEK • Watase-san installed and is maintaining NAREGI middleware at KEK NAREGI is based on Grid standards defined at the OGF (Open Grid Forum) • NAREGI beta-1 released in May 2006 and installed at KEK • With some help from the NAREGI support team for the last stage • Compute part (6 nodes) • Data grid part (3 nodes) Tested at KEK with P152 (heavy ions) and Belle simulation Experience with middleware being built up D. Boutigny

  9. D. Boutigny

  10. Next steps (from Watase-san presentation during last workshop) • Release of NAREGI b2 in May 2007 • Expectations • Easy installation by apt-rpm • Stable and Robust middleware • Interoperation with EGEE/gLite • Job submission, Data exchange, Information • Customization of UI / VO portal with NAREGI Web service interface • Various useful features for application • GridFTP-APT: automatic parallel tuning for GridFTP • GridMPI: MPI jobs linking over sites • Bust job submission • Try to install LHC data processing or simulation on NAREGI • To prove usefulness even in HEP application • Release of NAREGI v. 1.0 in April-May 2008 • Expand NAREGI sites to Asia, EU and US in the collaboration for its complementary features D. Boutigny

  11. Interoperability between EGEE and NAREGI • 2 possible approaches • Implement the GIN (Grid Interoperability Now) layer in NAREGI • Defined by the GIN group from the OGF • Short term solution in order to get the Interoperability Now ! • Pragmatic approach • Work with longer term standards defined within the OGF • Develop a Meta Scheduler compatible with many Grid implementations • Based on SAGA (Simple API for Grid Applications) "Instead of interfacing directly to Grid Services, the applications can so access basic Grid Capabilities with a simple, consistent and stable API" • and JSDL (Job Submission Description Language) D. Boutigny

  12. GIN • Pushed by KEK, NAREGI has done considerable efforts to implement the GIN layer • Confirmed during our visit to NAREGI in February • "Trying to identify islands of interoperation between production grids and grow those islands" Developing an interoperation island with EGEE Common framework for Authentication and VO management Cross job submission between gLite/EGEE and NAREGI Data transfer between gLite and Gfarm Grid resource information service around the world From NAREGI presentation in 02/07 D. Boutigny

  13. NAREGI NAREGI Client SRM Client Gfarm API NAREGI Metadata Server DPM (SRM Server) An example of interoperability GIN-data: Data Management and Movement Interoperability between NAREGI and EGEE at the data level Work also on: GIN-auth GIN-jobs GIN-info GIN-ops EGEE gLite Client SRM Client LFC (Metadata Server) GridFTP Server Gfarm Server Storage Storage D. Boutigny

  14. The SAGA / JSDL approach • This approach is being developed at CC-IN2P3 (Sylvain Reynaud) • This interoperability tool is sharing several modules with a software layer which is being developed at CC-IN2P3 in order to easily interface our local batch system to any Grid Computing Element D. Boutigny

  15. JDL RSL selected resource sandbox planning EGEE RUGBI delegate selection & sandbox management job job job desc. JSAGA gLite plug-ins Globus plug-ins WMS SRM input data GridFTP gLite-CE gLite-CE WS-GRAM WS-GRAM firewall D. Boutigny

  16. Execution environment Execution environment … … (multi-users (multi-users, parametric jobs, failure tolerance) Job submission task Job submission task CoG-Core API SAGA job desc. translator resource selection security security input/output sandbox input/output sandbox Web service task Web service task control control monitor monitor to/from JSDL File operation task File operation task best path, delegated best path, SAGA internal (IS internal (IS, QoS, user preferences, RSS, job-list-match, infosite, Lavoisier) delegated, combined login/pwd PKI GSI (classic, VOMS, MyProxy, X509, (CAS?)) login/pwd PKI GSI (classic gsiftp, file, http gsiftp, file, http, srm (1.1, 2.1.1), lfn, srb, mailto, worker, (gfarm?), tar, gzip) Extensible via plug-ins ELIS@ (Enterprise grid with Local Infrastructure and Services for Applications) D. Boutigny

  17. Next steps on NAREGI / EGEE interoperability • Continue work on both directions: GIN and SAGA / JSDL • Try cross job submission on both Grid middleware • Explore data exchange between NAREGI and EGEE D. Boutigny

  18. The Storage Resource Broker (SRB) • SRB is relatively light data grid system developed at SDSC • Considerable experience has been gained at KEK, SLAC, RAL and CC-IN2P3 • Heavily used for BaBar data transfer since years (up to 5 TB/day) • Very interesting solution to store and share biomedical data (images) • Advantages • Easy and fast development of applications • Extensibility • Reliability • Easiness of administration D. Boutigny

  19. MRI Siemens MAGNETOM Sonata Maestro Class 1.5 T Acquisition Control PC DICOM DICOM DICOM DICOM SRB Export PC (DICOM server, SRB client) Biomedical applications using SRB • push DICOM D. Boutigny

  20. From SRB to iRODS • iRODS (iRule Oriented Data Systems ) is the SRB successor • KEK and CC-IN2P3 are both involved in iRODS developments and tests • Should bring many new functionalities D. Boutigny

  21. From SRB to iRODS iRule Oriented Data Systems Definition of rules and micro-services Allows to fully customize the system in order to adapt it to the application D. Boutigny

  22. SRB-DSI Being developed by Iida-san SRB-DSI is a software layer which allows the Grid world based on Globus to interoperate with SRB Globus world: data transfer based on GridFTP SRB: based on its own protocol Crucial to interoperate the LCG and SRB worlds D. Boutigny

  23. GridFTP SRB Protocol SRB Protocol (Svr-Interconnection) globus-gridftp-server globus-gridftp-server globus-gridftp-server SRB-DSI SRB-DSI SRB-DSI The SRB-DSI architecture Scommand edg-gridftp-ls globus-url-copy rac01 GridFTP rsr01 rls09 rls10 SRB (GSI_AUTH) SRB (GSI_AUTH) SRB (GSI_AUTH) MCAT srbServer srbServer srbServer POSIX I/O (LOCAL) POSIX I/O (LOCAL) HPSS API I/O “rls10-hpss-vfs” “rsr01-ufs” “rls09-hpss-hsi” External Disk Sub-sys hpssfsd HPSS API I/O • Each DSI accesse to the following SRB resource • gsiftp://rsr01 -> rsr01-ufs • gsiftp://rls09 -> rls09-hpss-hsi • gsiftp://rls10 -> rls10-hpss-vfs HPSS D. Boutigny

  24. Next step for SRB / LCG interoperability • In order to have LCG and SRB fully interoperable we need to develop an SRB / SRM interface • This will be a common area of work for KEK and CC-IN2P3 in the near future • Iida-san 8 month stay at CC-IN2P3 • Then we will explore the possibility to make SRB an alternative for LCG storage • SRB / iRODS is probably a good candidate to store user's files Grid wide  An idea to be explored D. Boutigny

More Related