ESA PDGS Data Cube

ESA PDGS Data Cube Andrea Della Vecchia, Damiano Guerrucci, Mirko Albani (ESA) Simone Mantovani (MEEO) CEOS WGISS#47 29th April 2019

Outline • ESA PDGS Data Cube • Use Cases • EO Data Management • In-Situ Data Management • CEOS WGCV • System Scalability • ESA PDGS Collaborative Environment • Next Steps

PDGS Data Cube Objectives • Brings the users to the data • Provides standardised and harmonised interfaces to manage EO and in-situ data • Innovative, and complementary, approach to access and visualize data • Permits on-the-fly pixel based processing and bulk asynchronous processing • Integration into ESA Collaborative Environment

UC1: EO Data Discovery, Visualization and Pixel-based Access • UC2: EO Service Discovery and Processing

CLI / REST API Web based GUI Jupyter Notebook VisualizationlayerStandardised data accessinterfacesallowconnecting a wide range of userinterfaces DAS OGC WCS Server Datacube Engine/API VMsThe deployment of DAS in front of each data source enableseffectivepixel-basedaccessservices ESA Third Party Missions ESA Heritage Missions ESA Earth Explorer Data layerData remainattheirown location (multiple data centers) with the original data format

TPMsonline data storage @ESA

HMs online data storage(Temporarystorage@ESA)

EEs online data storage @ESA

UC1: EO Data Discovery, Visualization and Pixel-based Access • The users: • connect to the GUI (https://eodatacube.eu/) • select the collection(s) • define AOI/TOI • connect to the centraljupyterhub • use/edit/create notebooks to manage the full data cycle (discover, access, process, visualize) • Open his/herown terminal • use API / CLI to code and send the access / processing query The GUI: connects to datacube @ESA sends the access / processing query to the DAS component The DAS: retrieves the data from the online storage sends back the GUI

UC2: EO Service Discovery and Processing • The users: • connect to the GUI (https://eodatacube.eu/) • select the collection(s) • define AOI/TOI • connect to the centraljupyterhub • use/edit/create notebooks to manage the full data cycle (discover, access, process, visualize) • Open his/herown terminal • use API / CLI to code and send the access / processing query The jupyterhubprovides (https://jupyter.eodataservice.org/user/demo/tree) : notebooks with examples of access / processing query to the DAS component @ESA interactiveinterface for live coding to discover, access, process, visualize the datacube(s)

CLI / REST API F E D E R A T I O N Web based GUI Jupyter Notebook VisualizationlayerStandardised data accessinterfacesallowconnecting a wide range of userinterfaces DAS DAS DAS DAS DAS DAS DAS Datacube Engine/API VMsThe deployment of DAS in front of each data source enableseffectiveaccessservices Data layerData remainattheirown location (multiple data centers) with the original data format Thematic data Other geospatial data Mission-specific data

Federated Infrastructure Available Collections (~ overall storage +5PB) • High Resolution optical satellite data (Level 1) • Sentinel 2 (10m, global, since 2015) • Landsat 8 (30m, Europe, since 2013) • Land Products (Level 2+) • Vegetation Indexes (250m - 10m, 2000) • Land Surface Temperature (1Km, 2000) • Soil Moisture (100Km - 10km, 1978) • Land Cover type (1km, 2000) • Atmospheric Products (Level 2+ Numerical Model) • Air Temperature (12.5 - 1km, 1979) • Precipitation (100km - 4.5km, 1979) • Aerosol Optical Thickness (1km, 2000) • Ocean products (Level 2+ Numerical Model) • Sea Surface TEMPERATURE (5km, Global, 2017) • OCEAN Salinity (10Km, Global, 2010) • Chlorophyll data (5km, Europe, 2010 – 2013) • Sea Surface Height (12km, Europe, 1993 – 2017) • Sea Surface Velocity (12km, Europe, 1993 – 2017) • ESA TPMs/ HMs/ EEs • Landsat 4/7 (30m, Europe, since 1984) • SPOT (1986 – 2015) • ERS-1/-2 (1991 – 2011, on demand) • ENVISAT (2002 – 2012, on demand) • SMOS (10km, since 2010) In BLUE external repositories

ESA DC / ODC Federation – Next Step WGISS-46-28: WGISS-Exec (Robert Woodcock, Andrea Della Vecchia) to prepare one to two slides for the SEO on what WGISS is doing for Data Cubes and propose a way forward for cooperation between WGISS and SEO on the FDA Data Cube topic. ESA “Data Cube Service” Project will address international interoperability issues … WCS 3.0 Evolution … Standardised OGC WS Interface OGC WCS Server Proprietary APIs ODC ESA Third Party Missions ESA Heritage Missions ESA Earth Explorer

UC2.1: In-situ/ EO Data Discovery, Visualization and Pixel-based Access • - Atmospheric Correction Inter-Comparison Exercise (ACIX) • - Radiometric Calibration Network (RadCalNet) • UC3.1: In-situ / EO Service Discovery and Processing (on-going) • - Atmospheric Correction Inter-Comparison Exercise (ACIX) • - Customized UI to support AOT/WV/SR inter-comparison • - Jupyter notebook to support live coding / sharing

EO and In-situ Data Management Preferred - Under development Preferred Optional SQL ingestion In-situ On-the-flyTranslator rasterization EO WCS • Pre-processing (e.g., ARD): • Re-projection • Re-sampling • Format conversion OpenSearch

Atmospheric Correction Inter-Comparison Exercise (ACIX)http://calvalportal.ceos.org/projects/acix • ACIX aimed to bring together the developers of Atmospheric Correction (AC) processors, to generate who were invited to generate the corresponding Bottom-Of-Atmosphere (BOA) products. The European Space Agency and NASA organise this exercise on AC processors inter-comparison using Landsat-8 and Sentinel-2 imagery of various sites, i.e. agricultural, snow/artic areas, deserts and coastal. A common and harmonised inter-comparison procedure was agreed and followed by all the participants. • For more information about the ACIX workshops, please follow the links to the SPPA web pages: ACIX I 1st WS , ACIX I 2nd WS • Objectives • To elaborate concepts, protocols and guidelines for the inter-comparison and validation of BOA products • To better understand BOA reflectance uncertainty contributors by comparing the outputs of different AC schemes • To identify and review the different uncertainty contributors • To propose further improvements of the available AC schemes • Expected Outcomes • Description of concept, protocols and procedures for inter-comparing and validating products • Assessment of the relative differences among the inter-compared AC processors results • Definition of key regions and key periods for validation and quality assessment • Description of a coordinated plan for inter-comparison and validation activities

ACIX Inter-Comparison Exercise • Inter-comparison of BOA radiometric calibration processors • AERONET stations over 19 sites to perform in-situ measurements of AOT, WV and SR • Landsat-8 and Sentinel-2 acquired over sites and converted to BOA with selected processors, to generate AOT, WV and SR • Inter-comparisons performed for all the products on image subset of 9km x 9km centred on every AERONET sites

Radiometric Calibration Network (RadCalNet)https://www.radcalnet.org/#!/ • RadCalNet is an initiative of the Working Group on Calibration and Validation of the Committee on Earth Observation Satellites. The RadCalNet service provides satellite operators with SI-traceable Top-of-Atmosphere (TOA) spectrally-resolved reflectancesto aid in the post-launch radiometric calibration and validation of optical imaging sensor data. The free and open access service provides a continuously updated archive of TOA reflectances derived over a network of sites, with associated uncertainties, at a 10 nm spectral sampling interval, in the spectral range from 380 nm to 2500 nm and at 30 minute intervals. Each individual site is equipped with automated ground instrumentation in order to provide continuous measurements of both surface reflectance and local environmental/atmospheric conditions needed for the derivation of TOA reflectance values. TOA reflectances provided on this portal are derived from the individual sites surface and atmosphere measurements using a common method through a central processing system. Each member site takes responsibility for the quality assurance of the surface/atmosphere measurements provided and is subject to peer review and rigorous comparison to ensure site-to-site consistency and SI traceability.

RadCalNet Exercise • Support post-launch TOA radiometric calibration and validation of optical EO data • Ground nadir-view reflectance measurements (converted to TOA), and atmospheric measurements such as surface pressure, columnar water vapour, columnar ozone, aerosol optical depth and the Angstrom coefficient, over 4 sites • Nadir-view top-of-atmosphere reflectance at 30 minute intervals from 9am to 3pm local standard time at 10 nm intervals from 400 nm to 2500 nm

ACIX/RadCalNet Inter-Comparison Exercise Web based GUI Jupyter Notebook • Data Preparation: in-situ and EO Level 2 data are indexed and registered in the data cube, preserving both radiometric and geometric properties • Data Access: DAS shall offer access to both EO and In-situ data via unified interface (i.e. WCS) • Data Processing:Jupyter allows the users to perform inter-comparison scripts, sharing the code with authorise users, and accessing always the same data via WCS server. • PROs • Centralised data repository, replacing scattered ftp servers • Harmonised and standardised online data access interface, permitting data download • Reproducibility of inter-comparison exercises (via Jupyter notebook), restricted to authorised users • Easy sharing of algorithms and results DAS OGC WCS Server Landsat-8/Sentinel-2 Data In-Situ: ACIX and RadCalNet

ESA PDGS Data Cube SW Refactoring • Software refactoring currently on-going • Porting all PDGS data cube components to Docker and Kubernetes – fast and easy deployment and horizontal scalability • Faster indexing time • Quicker time response • To support concurrent users • Preserving all functional/interoperability requirements • Operational Environment ready by June 2019 • http(s)://datacube.pdgs.eo.esa.int/ • http(s)://jupyter.pdgs.eo.esa.int/

Interfaces • Adherence to standards / Evolution of standards • Discovery – OGC/CEOS OpenSearch BP – to discover collections and products • Access – OGC Web Coverage Service (WCS 2.0) – to provide pixel based access (including sub-setting in any dimension) • Processing • OGC Web Processing Service (WPS 2.0) – to execute processing on data cubes • Jupyther HUB – “The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text” (see https://jupyter.org)

Collaborative Environment • EO collaborative environment is a virtual working environment providing discovery and access services to EO data, analysis tools, processors, Information and Communication Technology resources required to work with them, through one coherent interface. • A collaborative environment aims to change the paradigm from • “data to user“ to “user to data”

ESA PDGS Collaborative Environment • To provide the ESA PDGS with a set of interoperable and federated services permitting the users to: • Access to missions/platforms information supported by a common ontology • Discovery and, if possible, direct access to EO and in-situ data: • Copernicus Missions (e.g. Sentinels) • Third Party Missions - TPMs (e.g. SPOT, Landsat, …) • Heritage Missions - HMs (e.g. ERS-1/2, ENVISAT, …) • Earth Explorer – EEs (e.g. SMOS, Cryosat, SWARM, …) • International repositories (e.g., NASA CMR, CEOS IDN) • Discovery and access to exploitation services (e.g. datacube): • Browse/visualization tools for time series analysis • EO data extraction, resampling and reprojection • Hosted Processing for authorised users/communities (e.g. CAL/VAL)

Operational interface ESA Query and service requests Conceptual Model External Data access interface Operational Data Access Web User Interfaces Web User Interface for pixel-based access, analysis and processing Web User Interface for granule-based discovery and/or retrieval International EO Information Page: - CEOS IDN • Mission oriented GUI • - SWARM • - SMOS • - Cryosat • - External External catalogue application ………. ………. ESA EO Information Page: - ESA EO Gateway Multi Mission GUI: – ESA EOCAT ESA PDGS Data Cube GUI, VIRES GUI, etc… OpenSearch (collections and granules) Centralised Knowledge-Base Facility Knowledge-Base Facility Standardised Interoperable Interfaces OpenSearch (collections and granules) Standard services request Interfaces (e.g., restful API) Thesauri Repository Metadata management Protocol for data access (e.g., OGC, Open Datacube) EO Data & Service Metadata Repository Products Access Interface Data Repositories Data Repositories Copernicus Data Repositories External Web Interface ESA Missions Data Repositories External Web Interfaces Hosted Processing Standardised Interoperable Services Interfaces IT Resources provided as services (IaaS, PaaS, SaaS) S-1 S-2 S-3 HMs TPMs EEs S-5 User Applications Temporary ESA User Applications VM – Processors Storage VM – Browse Images Generation VM – DataCube Engine/API

PDGS DataCube into Collaborative Environment GUI @ MEEO DAS & Jupyter @ ESA Data @ ESA (NFS read-only)

Next Step – 2019 • ESA PDGS Data Cube TTO by Q2 2019, including: • Software refactoring, based on Kubernetes, to boost performance and manage concurrent users • Internal OpenSearch interface aligned with CEOS OS BP 1.2 • All ESA TPM/HM/EE collections available • WCS & Jupyter interfaces for ESA Cal/Val initiatives (ACIX, RadCalNet) • Update of ESA by Dec ’19, including: • Cooperation with CEOS WGISS / WGCV / FDA teams • WPS interface available • Interoperability enhancement (e.g., connection with Open Data Cube instances in synergy with ESA Datacube Service Project) • Web User Interface for ESA Cal/Val initiatives (ACIX, RadCalNet) • New ESA missions available

ESA PDGS Data Cube