RSS support to scientific exploitation of EO data

1. RSS support to scientific exploitation of EO data Giancarlo Rivolta giancarlo.rivolta@esa.int ESRIN, 11 October 2012

2. RSS Overview RSS G-POD process RSS flexible resources for on-demand processing Success stories Future directions GPOD demo Questions & Answers Summary

3. RSS Overview • Research and Service Support • The ESA Research and Service Support (RSS) service provides resources for supporting Earth Observation (EO) data exploitation • RSS users are EO Principal Investigators and EO Service Providers • The environments made available to RSS users are: • Grid Processing On-Demand (G-POD) • Service Support Environment (SSE) • E-collaboration environment (Join&Share) • OGC services (WMS, WFS, WCS) • Virtual Reality Theatre (VRT) • Knowledge-based Earth Observation (KEO) • Reference Data Sets (RDS) • EO Ontology search engine • More than 3500 people are registered as RSS users • On the industry side Logica delivers the RSS service as part of the ESRIN O&M Framecontract • More information available from the RSS Portal (rssportal.esa.int ) Let’s focus on G-POD

4. Research Process

5. RSS G-POD process steps • Principal Investigators • EO algorithms delivery • Data type and range indication • Output validation • On-demand EO data processing • Use of produced data (projects delivery, publications, etc) • RSS • Data are made available in the RSS catalogue • Algorithm porting • Integration design • Implementation • Test and validation (involving the PI) • On-demand EO data processing • Delivery

6. RSS Flexible Resources EO Scientists Principal Investigators On-demand processing service: Process delivery EO data Application Technology Infrastructure ESRIN - 172 cores - 400 TB Flexible Infrastructure - 10-200 cores - 1-10 TB UK-PAC - 88 cores - 300 TB • Flexible Infrastructure satisfies: • HW requirements • Connectivity requirements • SLA (HA, help desk, ticketing systems, etc.)

7. RSS G-POD Team • On-demand Processing: team composition and responsibility • The G-POD team is composed by 4 selected highly skilled resources • Key areas of responsibility are: • Interaction with Principal Investigators • Algorithm Understanding and proposal evaluation • Resources planning (data, storage, integration, capacity, etc) • Algorithm integration (porting, design, implementation, test) • Data management • Processing environment set-up • User support (for on-demand processing) • Bulk Processing management • Key skills are: • Remote sensing background / EO data exploitation expertise • Software development /software integration • Virtualization technology • Grid / Cloud technology • Operations and Maintenance • Scientific goal orientation

8. Success Story: MGVI • MERIS Global Vegetation Index Cat-1 project • Objective: Development of MERIS derived products based on the Fraction of Absorbed Photosynthetically Active Radiation (FAPAR) algorithm • IPR: MGVI algorithm developed and owned by the Joint Research Centre • Evolution and time span: Several GPOD-MGVI versions integrated since 2007, bulk re-processings over the entire ENVISAT mission (2002-2012) • Outcomes/Benefits: • Access to online data for bulk re-processing • Systematic global products published on ESA Earthnet Portal • Regional products distributed via RSS SSE and RSS OGC service as well • On-demand MGVI regional available • MGVI Europe / Africa distributed to JRC for creating products for desertification monitoring • Several scientific papers published • MGVI delivered as complementary information to Fire detection products over Senegal • From JRC contribution to the paper “A Model for the Scientific Exploitation of Earth Observation Missions”: • RSS supported the process of algorithm development, test and deployment over a general purpose processing on demand environment dedicated to the scientific exploitation of EO data generated by ESA missions, as well as the provision of tools and environments in support to that process.

9. Success Story: KLIMA-IASI • The KLIMA-IASI ESA Cat-1 project was initially structured along 3 phases • 1- The KLIMA-IASI application is integrated in G-POD • The data set used consists of: IASI data levels L1B, L1C and L2, in EPS format; for the month of July 2008; • The results of the application on this data set are used to determine if results produced from Level L1B data are better than the results produced from Level L1C; • The phase is concluded with the decision to use Level L1B or Level L1C data; • Also, during this phase, processing performance is measured and the processing time for the qualification campaign is estimated • 2- IASI Level L1 (B or C) and L2 are ingested for the whole year 2009, in EPS format, up to 5Tb • As a "qualification campaign", the KLIMA-IASI application is run upon the year 2009 data and results are used to validate the quality of the algorithm in relation to Gosat measurements; • Also, during this phase processing performance is measured and the exploitation scenario defined • 3- Full IASI Mission data and EumetCast NRT are systematically ingested in G-POD • The KLIMA-IASI application performance is optimized in line with NRT requirements; • The KLIMA-IASI application is enhanced to accept BUFR data format; • KLIMA-IASI application is provided as a service to the User Community • IPR: KLIMA-IASI algorithm developed and owned by the IFAC-CNR, Florence • Evolution and time span: Several GPOD-MGVI versions integrated since 2011, bulk processing on dataset from 1 year IASI data (Mar 2010 – Feb 2011) • Outcomes/Benefits: • On-demand KLIMA-IASI service available to PI for pre-analysis • Access to cloud resources for flexible capacity to meet the deadline • Access to online data for bulk re-processing (11 out of 12 months processed) • Several scientific papers published and presentations at conferences • Process speeded-up by 2 orders of magnitude (flexible)

10. Success Story: SMOS • SMOS testbed • Objective: Provide a flexible test environment to support ESA GQ users for L1 calibration and developers for L2 OS • Evolution and time span: Several SMOS L1 and L2 OS re-processing campaigns with custom configuration (recently processed sample SMOS L1 products over 25 months (from June 2010 to June 2012, 1728 L1A/B + 6576 CRS products), SMOS L2 OS 1 month) • Outcomes/Benefits: • Fast integration of new versions (if ICD unchanged) • Access to online data for bulk re-processing • Access to flexible cloud resources for meeting deadlines • On-demand SMOS L1 available for GQ • On-demand SMOS L2 OS available for developer • SMOS L0 NRT ingestion chain set-up for L1 NRT custom re-processing • Testbeds required for other sensors as well • Open call for proposal published on EOPI

11. Future directions • In the light of the RSS success story the following should be considered • RSS model as basis for the definition of a harmonized model of research and service support • Federation of payload data ground segment services to the scientific community

12. Contacts & further information Giancarlo Rivolta giancarlo.rivolta@esa.int RSS Team rss_team@esa.int GPOD Team eo-gpod@esa.int G-POD Portal gpod.eo.esa.int Join&Share wiki.services.eoportal.org

13. GPOD demo Now the GPOD Team will give a 15-minute GPOD demo GPOD Portal: gpod.eo.esa.int

14. Questions and Answers Q&A?

15. Thank you Giancarlo Rivolta Logica xxx Visitor address Street Postcode City Country www.logica.com Contact: Name Function T: +XX (0) 000 00 000 000 F: +XX (0) 000 00 000 000 E: address@logica.com