1 / 25

Federated Earth Observation ( FedEO )

Federated Earth Observation ( FedEO ). A. Della Vecchia, D. Guerrucci , M. Albani (ESA) Yves Coene ( Spacebel ) CEOS WGISS#47 30 th April 2019. Outline. FedEO Software refactoring IDN metadata population ESA PDGS Collaborative Environment SLT Activities Next Steps. Outline.

tedm
Download Presentation

Federated Earth Observation ( FedEO )

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Federated Earth Observation (FedEO) A. Della Vecchia, D. Guerrucci, M. Albani (ESA) Yves Coene (Spacebel) CEOS WGISS#47 30thApril 2019

  2. Outline • FedEO • Software refactoring • IDN metadata population • ESA PDGS Collaborative Environment • SLT Activities • Next Steps

  3. Outline • FedEO • Software refactoring • IDN metadata population • ESA PDGS Collaborative Environment • SLT Activities • Next Steps

  4. FedEO: Federated Earth Observation Gateway System • FedEO = Federated Earth Observation missions access • The FedEO system provides a unique entry point to a growing number of scientific catalogues and services.

  5. WGISS Connected Data Assets

  6. Outline • FedEO • Software refactoring • IDN metadata population • ESA PDGS Collaborative Environment • SLT Activities • Next Steps

  7. Software Refactoring – Objectives • Porting all FedEO components to Docker and Kubernetes – fast and easy deployment and horizontal scalability • Optimization of the gateway and the catalog - quicker time response (3x up to 10x faster wrt concurrent users) • Optimization of the dataset metadata ingestion job – faster ingestion time (+400% up to 40M entries) • Preserving all functional/interoperability requirements

  8. FedEOArchitecure FedEO TTO by June 2019, after ESA EO Catalogue TTO. They share same SW with different configuration

  9. Outline • FedEO • Software refactoring • IDN metadata population • ESA PDGS Collaborative Environment • SLT Activities • Next Steps

  10. FedEO Metadata Mediator • Automatic Procedure – Available on FedEO reference environment, on Operational environment at ESA by June 2019 Partner Metadata repository Harvester tool Metadata Import Metadata Mediator Metadata Preparation Metadata Preparation IDN guideline for Information Content completeness and consistency (HERE) IDN Complementary Informationgcmd keyword FedEO Collection Catalogue ESA Thesauri Service FedEO Gateway ISO to DIF-10 DIF-10 Validator (HERE) DIF-10 Validation Metadata Export DIF-10 Encoding GCMD keyword update, Multiplicity of DIF-10.2 metadata fields and APIs for metadata validation, provided by IDN NASA team (Michael Morhan) IDN repository

  11. DIF-10 Metadata Export • New FedEO Metric web page provides info about collections, granule, reporting about DIF-10 metadata conversion and DIF-10 metadata ready for ingestion into IDN http://geo.spacebel.be/opensearch/asset/index.html FedEO Metric web page source code is a customization of the CWIC Metric Web Page https://cwic.wgiss.ceos.org/asset/index.html. Source Code kindly made available by CWIC team (Li Lin)

  12. Open Issues / Next Steps • ESA • FedEO shall be connected to ESA EO Catalogue after its TTO, by June 2019. All ~200 collections shall be visible and >35M products shall be accessible (further products shall be integrated all along 2019) • DLR • FedEO currently adopted as translator between DLR and IDN. • DLR is preparing all its collection for being exported into IDN, asking to update ESA and GCMD thesauri. • DLR shall ask NASA to remove all the old/deprecated collections currently on IDN • EUMETSAT • A subset of EUMETSAT collections can be ingested passing through FedEO • Misalignment between EUMETSAT platform values and GCMD thesauri (e.g., “METOP” assuming both “METOP A” and “METOP B”) • VITO/JAXA • All available collection correctly encoded in DIF-10 • ROSCOSMOS • Waiting for formal agreement • CCI / CMEMS • Waiting for feedbacks GCMD Thesauri update, and synchronization with ESA Thesauri continuously required

  13. Open Issues / Next Steps • ECV • Action from the 10th Meeting of the CEOS/CGMS Working Group on Climate 21st March 2019: • To export ESA-ECV metadata from http://climatemonitoring.info/ecvinventory/ to IDN passing through FedEO. ECV collections cab be downloaded from here http://climatemonitoring.info/wp-content/uploads/2019/01/ECV_Inventory_v2.0.xlsx • The ECV 2.0 is currently under review, under migration toward release 3.0 • Cross check DIF-10 metadata cardinality vs ECV 2.0. • Identify mandatory IDN metadata not managed by ECV 2.0 • Reporting the results to WGClimate, asking for their integration within ECV 3.0 release • Verify, and quantify effort accordingly, to ingest ECV metadata into FedEO to prepare DIF-10   • Status: • First prototype version of the “ECV” information available on http://geo.spacebel.be/opensearch/asset/index.html • To do: • Report to ECV team (http://geo.spacebel.be/opensearch/dif10/ingestion-ECV.html) • Manage mappings for platforms/instruments/organization names. Names used in the ECV Excel (e.g., NASA, CPOM, NASA JPL) are not identical or missing in the GCMD thesauri • Manage missing platforms/instrument relations in the thesauri • Manage abstract info, concatenating concatenate some columns to have some content in the “ISO abstract”, and prefix it with the title of the column, as ECV Excel does not contain a column with a proper “abstract” describing the collection.

  14. Outline • FedEO • Software refactoring • IDN metadata population • ESA PDGS Collaborative Environment • SLT Activities • Next Steps

  15. ESA PDGS Collaborative Environment • To provide the ESA PDGS with a set of interoperable and federated services permitting the users to: • Access to missions/platforms information supported by a common ontology • Discovery and, if possible, direct access to EO and in-situ data: • Copernicus Missions (e.g. Sentinels) • Third Party Missions - TPMs (e.g. SPOT, Landsat, …) • Heritage Missions - HMs (e.g. ERS-1/2, ENVISAT, …) • Earth Explorer – EEs (e.g. SMOS, Cryosat, SWARM, …) • International repositories (e.g., NASA CMR, CEOS IDN) • Discovery and access to exploitation services (e.g. datacube): • Browse/visualization tools for time series analysis • EO data extraction, resampling and reprojection • Hosted Processing for authorised users/communities (e.g. CAL/VAL)

  16. Operational interface ESA Query and service requests Conceptual Model External Data access interface Operational Data Access Web User Interfaces Web User Interface for pixel-based access, analysis and processing Web User Interface for granule-based discovery and/or retrieval International EO Information Page: - CEOS IDN • Mission oriented GUI • - SWARM • - SMOS • - Cryosat • - External External catalogue application ………. ………. ESA EO Information Page: - ESA EO Gateway Multi Mission GUI: – ESA EOCAT ESA PDGS Data Cube GUI, VIRES GUI, etc… ESA EO Catalogue OpenSearch (collections and granules) Knowledge-Base Facility Centralised Knowledge-Base Facility Standardised Interoperable Interfaces OpenSearch (collections and granules) Standard services request Interfaces (e.g., restful API) Thesauri Repository Metadata management Protocol for data access (e.g., OGC, Open Datacube) EO Data & Service Metadata Repository Products Access Interface Data Repositories Data Repositories Copernicus Data Repositories External Web Interface ESA Missions Data Repositories External Web Interfaces Hosted Processing Standardised Interoperable Services Interfaces IT Resources provided as services (IaaS, PaaS, SaaS) S-1 S-2 S-3 HMs TPMs EEs S-5 User Applications Temporary ESA User Applications VM – Processors Storage VM – Browse Images Generation VM – DataCube Engine/API

  17. ESA Catalogue TTO – Results • ESA Catalogue shall: • be the centralised metadata repository of ESA Collaborative Environment • reuse same FedEO SW • manage collections Digital Object Identifiers (DOIs) • be part of CEOS WGISS Data Asset via FedEO

  18. Outline • FedEO • Software refactoring • IDN metadata population • ESA PDGS Collaborative Environment • SLT Activities • Next Steps

  19. SLT Activities • Participation in the review and issues of the following CEOS WGISS documents: • WGISS Connected Data Assets Client Partner Guide (OpenSearch), including IDN/CWIC/FedEO Asset • WGISS Connected Data Assets FedEO Data Partner Guide (OpenSearch) • WGISS Connected Data Assets Error Handling Guide • http://ceos.org/ourwork/lessons-learned-best-practices/

  20. WGISS Action • WGISS-46-19: WGISS CDA SLT to discuss way forward to implement a single front-end/portal within the IDN to access the WGISS CDA for data discovery and access. • M2M Interfaces • APIs allows two steps search to external clients aligned to CEOS OpenSearch BP 1.2 • GUI Interface • The user can be redirected from IDN to CWIC and/or to FedEOclient, via DIF-10 metadata links • IDN client able to manage two step search (overlap with https://search.earthdata.nasa.gov/ ??)

  21. Scenario 2 – GUI Interface ESA Collection Landing Page Second Step OSDD FedEO Client ISO 19139-2 Metadata

  22. Outline • FedEO • Software refactoring • IDN metadata population • ESA PDGS Collaborative Environment • SLT Activities • Next Steps

  23. Synergy with OGC initiatives • Requested to OGC to approve that the related document package enters the so-called “public review” phase (3 weeks). Public comments period was closed in August 2018 (OGC call for public review). • All public comments addressed that were received (September 2018-March 2019)and document package resolving comments redelivered to OGC TCC/OAB: • OGC 13-026r9: 6 comments • OGC 17-003: 24 comments (incl. 18 from DLR – related to JSON-LD Linked Data encoding) • OGC 17-047: 6 comments (incl. 4 from VITO). • All commentsreceivedwereprocessedby the SWG and dispositions agreedwithauthors of the comments. • Presentation to OGC TC (Web conference) scheduled for 30 April 2019 (15h CET) to allowstarting final TC Vote for adoption as standard immediatelyafter. • OGC 17-084: Most recent update published on SWG Web pages (6 March 2019). Work to resume shortly.

  24. FedEO Evolution • ESA GSTP Project “ESE-ERGO” (Sep ‘19 – Mar ‘21) aims to: • Adoption of OpenAPI, also encouraged by OGC, as standard interface for its services, in combination with JSON and JSON-LD • Analyse and adopt new international guidelines and standards to improve gateway/catalogue functionalities, performance and interoperability. This includes: • Emerging GeoDCAT-AP specification to publish EO metadata • OGC 13-026r9 to introduce enhanced functionalities (e.g., flexible string search, results ranking, links to EO external services) • OGC 17-047 to provide a JSON based encoding response from OpenSearch interface (e.g., quicker catalogue response, ease User interface development) • Transition to an internal metadata model of which other metadata models are views, to allow more flexibility and to align to similar approaches applied by NASA (e.g., Unified Metadata Model – UMM/-C/-G/-S/-JSON). This includes: • Recent OGC guidelines, in cooperation with CEOS/NASA, about collection and product metadata model based on GEO-JSON(-LD) encoding (e.g., OGC 17-084, OGC 17-003) • Generalization of the concept of catalogue entry in order to consider granules (/products), collections (aggregation of granules/products) and services related to both of them (e.g., remote processing, datacube, etc…) seamlessly (e.g., GeoDCAT-AP, ISO 19115-1 & ISO 19115-3, etc…) • Web-based metadata management tool to allow authorised users to add/modify collection catalogue entries

More Related