1 / 31

European DataGRID for EO

European DataGRID for EO. luigi.fusco@esa.int - julian.linford@esa.int ESRIN, 6-7 May 200 2 CEOS Workshop on GRID. EO applications and GRID requirements ESA EO participation to European GRID projects – DataGrid Ideas for CEOS. Summary. Earth Observation Community

king
Download Presentation

European DataGRID for EO

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. European DataGRID for EO luigi.fusco@esa.int - julian.linford@esa.int ESRIN, 6-7 May 2002 CEOS Workshop on GRID CEOS GRID Workshop

  2. EO applications and GRID requirements ESA EO participation to European GRID projects – DataGrid Ideas for CEOS Summary CEOS GRID Workshop

  3. Earth Observation Community GRID interactive scenario Common access to EO missions catalogues Acquisition plan, order, delivery On demand high level products generation Parametric data fusion and models integration Collaborative publishing of results CEOS GRID Workshop

  4. EO and Networking Computing – which data models? • Distributed Computing • Integration of data from various instruments and missions • High-Throughput Computing • Interferometry … • On-Demand Computing • Generation of EO user products… • Data-Intensive Computing • Archive data re-processing, climate modeling… • Collaborative Computing • Scientists application interactions, Instrument cal/val … Ian Foster andCarl Kesselman, editors, “The Grid: Blueprint for a New Computing Infrastructure,” Morgan Kaufmann, 1999 CEOS GRID Workshop

  5. High demanding computing Pomona (Cal): subsidence velocity fields 40 ERS1/2 images (92-99), Ambiguity: 28 mm Digital Elevation Model • GRID requirements: • large data files (10+ GB) • stages with intensive processing • science driven value adding CEOS GRID Workshop

  6. Science collaborative environment: El Niño SST November 1997: El Niño January 1999: La Niña SST anomaly CEOS GRID Workshop

  7. Global fire atlas - ATSR: 1997 CEOS GRID Workshop

  8. Global fire atlas - ATSR: 1998 CEOS GRID Workshop

  9. Provide a single access point to space systems to emergency & rescue organisations in case of disasters • Participating Space Agencies: CNES, CSA, ESA, ISRO, NOAA, … • Missions: RADARSAT; ERS, (Envisat); SPOT; IRS; NOAA, … CEOS GRID Workshop

  10. Earthnet Facilities real time Infrastructure SEAWIFS LANDSAT 7 TERRA/MODIS AVHRR SPOT IRS-P3 MATERA (I) HISTORICAL ARCHIVES KIRUNA (S) - ESRANGE TROMSO (N) MATERA (I) STANDARD PRODUCTION CHAINS MASPALOMAS (E) NEUSTREL.ITZ (D) METADATA BROWSE WEB PRODUCTS USERS MULTIMISSION DATABASES FOR REMOTE ACCESS AND USER SERVICES ESRIN USERS CEOS GRID Workshop

  11. ENVISAT FACILITIES ORGANISATION • Decentralised architecture, central co-ordination and supervision. • National facilities put at ESA’s disposal via MOUs and contracts. • Direct dealing with scientific users (outside ESA operational remit) • Co-operation with value added industry in E.O. promotion and in technology transfer from research to applications. LRAC/S-PAC FIN Co-PAC UK-PAC ESOC F-PAF D-PAC F-PAC ESRIN I-PAC E-PAC CEOS GRID Workshop

  12. Countries No Projects 1-25 Projects 26-50 51-100 100+ Projects P.I. geographic distribution Stimulating new researchs AOs: Stimulating scientific research world-wide 3500+ science Users of ESA data 120 New Cat-1 Projects in 2001 700 Envisat AOs to start in 2002 CEOS GRID Workshop

  13. Why GRID in EO? (1) • EO Community: Progressive refinement of data from many sourcess to produce higher quality products • Product generation chain involving distributed organisations and users • Collaborative: distributed users and data – large international cooperation • Discovery: large numbers of products & resources • Interoperabiltiy of catalogue and metadata already in operation • Web based data services CEOS GRID Workshop

  14. Why GRID in EO? (2) • Massive, non-stop data volumes • New instruments, sensors & product types • Distributed archives • Historical dataset reprocessing • Complex numerical processing algorithms • Near real-time turnover CEOS GRID Workshop

  15. The Grid from a Services View Environment Cosmology Space Science Applications ... S/C modelling Space weather EO Environment Distributed Data- Remote Problem Remote Collaborative Computing Intensive Visualization Solving Instrumentation Application Applications Applications Applications Applications Applications Toolkits Toolkit Toolkit Toolkit Toolkit Toolkit Toolkit : E.g., Grid Services Resource-independent and application-independent services authentication, authorization, resource location, resource allocation, events, accounting, remote data access, information, policy, fault detection GRID Middleware : Resource-specific implementations of basic services Grid Fabric E.g., Transport protocols, name servers, differentiated services, CPU schedulers, public key (Resources) infrastructure, site accounting, directory service, OS bypass CEOS GRID Workshop

  16. Needed GRID technologies • Resource-independent and application-independent services (middleware) • authentication, authorization, resource location, resource allocation, remote data access, • accounting, security, quality of services, fault detection, real time services, … • Specialized protocols, procedures, data standards, operational environments, interfaces to EO legacy systems… • EO dedicated portal and user access… CEOS GRID Workshop

  17. Participation to GRID initiatives CEOS GRID Workshop

  18. Participation in European GRID projects EU funded • DataGRID – Earth Observation application • EGSO – Solar radiance • DataTAG – access to Trans Atlantic Connectivity • … ESA funded • SpaceGRID – vision of GRID systems for space • ESA internal GRID initiative • … CEOS GRID Workshop

  19. DataGrid EO application objectives • Specification of EO requirements • Bringing Grid-aware application concepts into the Earth Science environment • Adaptation of existing systems and selected EO applications to use the DataGrid infrastructure • Testbed validation through prototypingactivity • Activities handled in coordinationandsynchronisation with other related and relevant work packages • Key partners: ESA-ESRIN, KNMI (NL), IPSL (F) • Associated partners: ENEA (I), BADC (UK) CEOS GRID Workshop

  20. GOME Instrument (1 day coverage) GOME’s Ground track CEOS GRID Workshop

  21. Application of DataGrid in EO • One Use Case being studied in detail (GOME) • Develop generic components • Feedback to DataGrid developers and Architecture Group • Re-use components to add new applications • Testing in “controlled” GRID environment (ESRIN-ENEA) and in “wide-European” environment CEOS GRID Workshop

  22. L1 4724 files = 66 Gb L2 9,448,000 files = 108 Gb Why Grid in EO? An Example: GOME Use Case Process 1 Year of data ESA ESA / KNMI RAW L1 L1 L2 Science Application End User L2 + L3 IPSL VAL L2 VAL Regulated Access to Grid processing power Secure access to Grid-registered high-volume data storage CEOS GRID Workshop

  23. WP1 WP3 Applications Integration WP2 WP8-9-10 Evaluation & Prototyping WP6 WP4 WP5 WP7 Requirements Site H Sites Installation Site G Site F SE SE Site E Sites SE Site D SE Site C SE Site B Installation Management SE Site A SE 1. Organization CVS Repository DataGrid Overview (1/5) Certificate Authorities Replica Management User Interface Computing Element EDG Rules Resource Broker Storage Element Installation Management EDG Membership Registration Information Index Information & Monitoring Application Environments Network Monitoring Documentation Architecture Group Integration Testing Middleware Developers Middleware Packages CEOS GRID Workshop

  24. Certificate Authorities 1. Obtain certificate Users 2. Join VO VO LDAP Server 5. Submit Jobs User Interface Grid Resource Broker Search Information Index Site H Site G CE Site F SE CE Site E SE CE SE Site D CE SE Site C CE SE Site B CE SE CE SE DataGrid Overview (2/5) 2. VO registration and information publishing 3. Sites subscribe to one or more VOs Site A Grid fabric resources 4. Publish details CEOS GRID Workshop

  25. Submit job Search Myjob Retrieve result JDL script Executable Site H Site G input data CE Site F SE input data CE Site E SE input data CE SE Site D CE SE Site C CE SE Site B CE SE CE SE DataGrid Overview (3/5) 3. Job submission with local data Certificate Authorities Information Index Check certificate User Interface Resource Broker Request status CEOS GRID Workshop

  26. Mydata Replicate input data input data input data input data Site H Site G CE Site F SE CE Site E SE CE Site D SE CE SE Site C CE SE Site B CE SE CE SE DataGrid Overview (4/5) 4. Data replication Submit job User Interface Replica Manager Replica Catalog CEOS GRID Workshop

  27. Information Index Replica Catalog LFN LFN LFN PFN PFN PFN Certificate Authorities :: :: :: Check certificate Submit job User Interface Resource Broker Search Search Request status Myjob Retrieve result input data input data JDL script Executable Site H input data Site G Site F CE SE Site E CE SE LFN CE Site D SE LFN CE SE Site C LFN CE SE LFN Logical filename Site B CE SE PFN Physical filename CE SE DataGrid Overview (5/5) 5. Job submission using replicated data CEOS GRID Workshop

  28. DataGrid Activities • Testbed validation • writing scripts to test and validate Testbed1 services • Develop Use Cases for end-to-end GOME processing and validation demonstration across three sites in Holland, France and Italy) • Develop EO Grid Application Interfacing Components • for generic application interfacing • High-speed connection to ENEA HPC network • Installation of ESRIN DataGrid site • using DataGrid installation tools • installation of 2 CEs: • ESRIN cluster using PBS • ENEA using LSF/AFS • and 1 SE 0.5TB RAID array on ESRIN cluster populated from ESA AMS MSS archive CEOS GRID Workshop

  29. DataGrid Issues • Very-large-scale, complex system with large numbers of participants • Dealing with new concepts and technology • Communication and coordination in large, distributed, multi-cultural, multi-institutional development group • Agressive deployment of middleware releases • Driven by needs of HEP • With EO & Biology contributions • Reliant on HEP making the right choices • Testbed stability, usability, performance and scalability • Application Grid interfacing layer needs to be developed • After CLIs, need APIs • Ongoing rapid prototyping and development • Keeping step with code & documentation • Architecture will evolve according to findings • Will take time to make fair assessment CEOS GRID Workshop

  30. Future Directions • In general • OGSA and integration of Web Services • Wider uptake of Grid computing concepts • In EO • Matrix of common application requirements • Development of Generic Grid platform interface components • Portals-based • Application Frameworks CEOS GRID Workshop

  31. Considerations for CEOS “involvement” in GRID • “gridding” of EO emerging technologies and services • Interoperability • EO data format handling • Web-mapping • Archive management • Demonstrate GRID applications • International project dimension • collaborative environment • relation with IGOS, WGISS Test Facilities … • Support “CEOS standardisation” approach to metadata and data access CEOS GRID Workshop

More Related