WP9 Earth Observation Applications

1. WP9 Earth Observation Applications luigi.fusco@esa.int

2. Outline (3 min) • EO and GRID requirements • EO, WEB Services and Application layered architecture in EDG • Work performed and achievements in year 2 • Plans for year 3 • Summary and Questions (3 min) (15 min) (3 min) (6 min)

3. The Earth Observation Community • Operational community (people in charge of facilities) • Facilities are widely distributed (data acquisition, storage, processing, dissemination) • Each facility deals with multiple missions to serve science and commercial users • Strongly dependent on “available” Information Technologies • Often in place to meet national institutional requirements • Science community (wide user communities) • Multi disciplines: ocean, land, atmosphere, … • Data access requirements for long term series + real time • Complex data processing and integration in modeling requirements • International cooperation is a must in global environmental issues

4. Altitude 0 to 100 km: GOMOS, MIPAS and SCIAMACHY are building a three-dimensional profile of ozone concentrations in the atmosphere. Altitude 0 to 20 km: MIPAS and SCIAMACHY are detecting low levels of gases from industry, power generation and agriculture. Altitude 0 to 10 km: MERIS obtains an image in which the clouds you see are but a part of a complex map of the concentration of water vapour. Altitude 0 to 4 km : ASAR and RA-2 create an accurate digital map of your surroundings, with height contours as accurate as 10 m. Ground level: ASAR, AATSR and MERIS map the vegetation and land use around you. Sea level: AATSR measures sea surface temperature to 0.3 °C accuracy. MERIS precisely maps ocean colour, plankton and chlorophyll distributions. ASAR and RA-2 measure ocean currents, average wave-heights and wind velocities. Underwater: RA-2 and DORIS combine to produce a detailed map of local gravitational strength, detecting the distribution of denser and less dense rock in the Earth crust beneath the oceans. ENVISAT Applications and Data …400+ TB/year…

5. Envisat science community No Projects 1-25 Projects 26-50 51-100 100+ Projects AOs: Stimulating scientific research world-wide 3500+Envisat science Users 700+ Envisat AOs started in 2002 P.I. geographic distribution

6. The reference EDG EO application • GOME Ozone monitoring by satellite and ground-based observations • multi-year global coverage dataset • Generation of total Ozone and Ozone profiles from GOME data • complex set of processing steps to transform raw satellite measurements into high-quality data products • Two alternative Ozone profile processing algorithms are validated against ground-based Lidar measurements • Collaboration among different institutes • France (IPSL), Italy (ESA, ENEA, UTV), Holland (KNMI) involved in the GRID demonstration

7. EDG EO challenge: Processing / validation of 1y of GOME data Raw satellite data from the GOME instrument (~75 GB - ~5000 orbits/y) Level 1 ESA(IT) – KNMI(NL) Processing of raw GOME data to ozone profiles. 2 alternative algorithms ~28000 profiles/day (example of 1 day total O3) LIDAR data (7 stations, 2.5MB per month) IPSL(FR) Validate some of the GOME ozone profiles (~106/y) Coincident in space and time with Ground-Based measurements DataGrid environment Level 2 Visualization & Analyze

8. Outline • EO and GRID requirements • EO, WEB Services and Application Layered Architecture in EDG • Work performed and achievements in year 2 • Plans for year 3 • Summary and Questions (3 min)

9. Emerging technologies of EO interests related to GRID • Standards for access to metadata and data • Interoperability (syntax and semantics) protocols • Services for inventory, browse, order, online data access • Data Archive Models • Community web services world wide accepted (OpenGIS consortium) • web mapping (processing at server) • web coverages (data set location and access)

10. URL Webmap Service Environmental Data USER Location Based Services Catalogue Systems Satellite Images Geographical Data Web Mapping services

11. EO Grid Engine GRID Surfer (Desktop application) Web Portal Application Web Mapping Technology Application Web ServiceInterface Earth ObservationGRID Engine Web Services Technology Interface Metadata Management Archive Management JDL Composition Job Execution Data Transfer & Replication GRID + other Technology Data Grid Services Fabric & Resources GRID

12. Desktop Application (GRID Surfer) EO Web Portal Application Open GIS Web Services MUIS Catalog Processing Algorithms Validation Algorithms Data Packaging EO GRID ENGINE Web Interface Services JDL Composition Job Execution Data Transfer & Replication Metadata Management Archive Management Interface User Interface Information Index Replica Catalogue VO Directory Grid Security Middle ware Resource Broker Replica Manager Storage Elements Computing Elements GRID Computing Cluster Disk Pools Archive Storage Grid Gateway Metadata Catalogue Network Service EDG EO Architecture Layers Problem SolvingEnvironments & Frameworks EO ApplicationsTools & Services Grid ApplicationInterfacing Data GridMiddleware Services LocalResources

13. Outline • EO and GRID requirements (updated view) • EO, WEB Services and Application Layered Architecture in EDG • Work performed and achievements in year 2 • EO Application environment and components • EO infrastructure deployment • Increasing GRID awareness in EO • Participation to ATF • Y2 Deliverables • EDG TestBed evaluation • D9.3 Overview • Plans for year 3 • Summary and Questions (15 min)

14. Objectives and achievements in Y2 • EO Application environment and components • Adaptation of processing (OPERA and NNO) and validation algorithms • WebMap Portal, EO GRID Engine, GRID Surfer GUI • EDG application interfacing components and utilities • Integration of EO Level-1 product catalogue & archive (MUIS & AMS), integration of LIDAR data • Partitioning of Level-2 products by latitude and longitude • Metadata catalogue for GOME Level-2 products using Spitfire • Metadata catalogue for Lidar data using Spitfire • EO Infrastructure deployment • Installation of EDG components UI, CE, SE • EDG interface to ENEA GRID • Local GLOBUS installations • Integration of GLOBUS and AFS/LSF

15. Grid-enabled EO WEBMAP Portal http://giserver.esrin.esa.it/grid-demo

16. EO Application Grid Interfacing Components • Examples of Components : • getInfo : generic resource discivery, e.g. • listObjects, listSites, listCEs, ListSEs, listCloseSEs, etc. • se_check : SE resource discovery • rm_copy : data replication tool • run : automated job submission, execution monitoring, performance statistics gathering, results retrieval • runstat : status & performance analysis reports for large number of jobs • getPF : obtains physical files

17. EO Grid Surfer Select resources Browse Replica Catalogue Proxy initialisation Replicate data Select map area Transfer files

18. Objectives and achievements in Y2 • Increasing GRID awareness in EO • CEOS GRIDS Task Team created • Support to ESA Grid initiatives (ESTEC meeting) • New applications for year 3 • EDG Installations planned at ESA sites • Build-up of EO GRID resources, knowledge & expertise • Participation to ATF • Feedback on performance of 1.1, 1.2 and follow-on releases • EO Requirements and WP9 user cases discussion • Discussion of Common Application Layer Requirements is proposed • Y2 Deliverables • EO User Requirements and Scaling Study (D 9.6) • TB 1.4 performance and evaluation report (D 9.3)

19. Objectives and achievements in Y2 (full details in D9.3 report) • EDG TestBed evaluation • Data Replication • Level-1 orbit data distributed in 5 SEs (CERN, CNAF, LYON, NIKHEF, RAL) • 4,700 15 Mb files (70.5 Gb) replicated – over 10,000 entries in EO RC • Data Processing • Tests were made submitting jobs to process Level-1 data in batches of 1..10..20..50 and 100 orbits at a time • Results strongly depends on Testbed stability – thousands of jobs submitted • Sometimes the TB performed very well (ALL jobs completed successfully) • Other times varying success rates obtained (e.g. 10..30..80% jobs successful) • Level-2 Products • Resulting products stored in CloseSEs - 4, 205 orbits successfully processed • Physical locations on SEs registered in Replica Catalog using WP2 middleware • Product Metadata stored in EO Spitfire catalogues • Level-2 Product Retrieval and Validation • Integration of Testbed and EO components to produce an “End-to-end” GOME Processing and Validation chain has been successfully demonstrated • Significant feedback to Developers, ITEAM & ATF

20. EO EDG Testbed evaluation Tools for automated job submission, monitoring, statistics gathering & analysis, for large numbers (1000s) of jobs (*) each job processes 1 GOME orbit using NNO algorithm

21. Overview of D9.3 Deliverable • Evaluation of Application Testbed 1.4 releases during December 2002 and January 2003 • Test cases and results for : • Data transfer and replication of Level-1 data to GRID Storage Elements • including analysis of ESA Archive System performance • Running the OPERA and NNO algorithms on the Grid • Storing the output results on the GRID Storage Elements • Registering the results meta-data in EO database using spitfire • Loading Lidar data on a GRID SE • Running the Validation algorithm using the meta-data databases • Running the Validation algorithm using file partitioning • Due for delivery in March 2003

22. Outline • EO and GRID requirements (updated view) • EO, WEB Services and Application Layered Architecture in EDG • Work performed and achievements in year 2 • Plans for year 3 • Summary and Questions (3 min)

23. EO Plans for Y3 • Complete, optimize and extend GOME validation in EDG • Process all 5 years (testbed permitting) • EDG Installations and EO dedicated infrastructures • Continue EO components development • New EO apps & users for EDG • CEOS GRID: validation of the EO application in EDG GRID environment across Atlantic GRID systems (EDG, NASA, NOAA…) • GREASE: simulation of OMI space instrument in EDG environment. Integration of a proprietary (Dutch Space) workflow tool in EDG • Preparation of GOMOS “operational” validation • Interest in Ozone profile science community to extend GOME validation experience to GOMOS (and possibly to other ENVISAT instruments) • Discussed at ENVISAT validation conference in Dec 02. • Feasibility study started. Go-ahead decision to be taken in coming weeks. • Partners: ESA, ACRI (France), IPSL/SA, IPSL/DataGrid, FMI • Final deliverables • D9.4 (July), D9.5 (December)

24. GOMOS “Operational” Validation in EDG GOMOS • One profile/spectrum every 0.5s from 10 to 120 km height • New potential partners involved (F - CNES, Service d’Aeronomie, ACRI; FIN – FMI; I – Intecs, …) • Example of e-collaboration in calibration/validation Example of 1 day GOMOS Star Occultation

25. Summary and questions Summary EO and GRID requirements EO, WEB Services and Application layered architecture in EDG Work performed and achievements in year 2 Plans for year 3 Questions?