Earth Observation and Climate Applications for EGEE

1. Earth Observation and ClimateApplications for EGEE IPSL: M. Petitdidier, C. Boonne, O. Marti, M.A. Foujols IPGP: JP Vilotte, G. Molguiny, DKRZ: J. Biercamp KNMI: W. Som de Cerff, ESA-ESRIN: L. Fusco

2. Objectives in DataGrid Lessons learned in DataGrid Achievements EGEE Description of the applications Conclusion Outline

3. Introduction Earth observation: Geoide, atmosphere and Ocean Interface between them Data acquisition, quality control, validation Studies space time evolution processus parametrization for modelling Prediction short term: meteorology, seismic, long term: climate

4. Objectives in DataGrid • Demonstrate how Grid infrastructure can respond to the complexity and constraints imposed by applications in the EO domain • By deploying EO applications on the EDG middleware (test and evaluate) • Development of EO Grid application interfaces and tools • Dissemination & promotion of this new technology to other scientific teams, satellite operational services and private companies • Participation to non grid oriented workshops, demonstrating Grid • Publication of results obtained on Grid in scientific papers

5. Earth Observation Application in EDG • Use case Ozone profile processing and validation • Common EO problem: measurement validation • Applies to (almost) all instruments and data products, not only GOME, not only ozone profiles • Validation consists of finding, for example, less than 10 profiles out of 28,000 in coincidence with one lidar profile for a given day

6. Level 1 ESA(IT) – KNMI(NL) Processing of raw GOME data to ozone profiles. 2 alternative algorithms ~28000 profiles/day (example of 1 day total O3) IPSL(FR) Validate some of the GOME ozone profiles (~106/y) Coincident in space and time with Ground-Based measurements DataGrid environment Level 2 Visualization & Analyze GOME Processing and Validation Usecase Raw satellite data from the GOME instrument (~75 GB - ~5000 orbits/y)

7. Requirements - Achievements • Handling of large number and large volume of files from different satellite instruments (Gome, Gomos) and ground measurements • Production of about 70,000,000 of Ozone profiles on the Grid stored by orbit (2000 profiles/orbit) and a smaller part by individual files. Not all of them are registered. • Processing with complex algorithms (e.g. Neural approach, inversion approach) using IDL(Interactive Data Language)-runtime • Creation and secure access to metadata catalogues and data • Satellite and Lidar metadata catalogues • Access secure and restricted to data and metadata • Interface Grid MW with the already developed operational satellite tools and infrastructure • MUIS catalogue (ESA proprietary product catalogue) • AMS Archive system (ESA proprietary data archive)

8. GOME Use Case file numbers 1 Year of GOME data : Just one year out of 8 years of data

9. ACHIEVEMENTS- Metadata Creation and secure access to metadata catalogues and data • EO Virtual Organization (VO) • Virtual Organization Management Service (VOMS) : permits restricted role to subgroups of a VO • Spitfire, EDG module, transparent and secure access to database (e.g. MySQL) used for Ozone profile files (~60000) and Lidar(<300) metadata • Query the database --> relevant logical filename --> EDG RC : Physical filename --> GridFTP for data retrieval • Efficient to query or insert tuples • Drawback: need to create the database

10. Grid metadata service integrated with the EDG RC : EO RMC: 25 attributes (for level1, level2, Lidar metadata), 32,000 tuples inserted as a test caseExample: The Metadata Catalogue is searched to locate the intersecting Profiles :User chooses a LIDAR station, and a time interval (start date, end date)The resulting LFNs are then used to build the JDL scripts for the Validation jobsImprovements needed:Better support for Application Metadata (e.g. DateTime type, Polygon)Restricted access for the different partners of one VO Use of Replica Metadata Catalogue for application

11. OGC Web Services Client Applications SOAP Part of present operational ESA EO environment WCS / WFS / WMS AMS ESA Data Archive SOAP Catalogue (CSS) MUIS ESA Catalogue GRID Engine Web Portal geant DATAGRID European Grid Storage Elements Client Applications (JAVA / Linux / Windows) ComputingElements INTERFACE WITH OPS TOOLS: MUIS (Multimission User Information Service) and AMS (Archive Management System)

12. Reference EO Applications in EGEE • Extend the Ozone experience: • Partners: KNMI, IPSL, SRON, RIVM, Univ. TorVergata, ESA, ..... • Goals: Use of “operational Grid” by the Ozone community • To produce and/or store the retrieved Ozone profiles or columns on the Grid • To store on the Grid the Ozone databases, old and new ones • To extend the processing capabilities (assimilation, new parameters, short term prediction, climatology, process studies...), and obtained new scientific results • To facilitate the collaboration and communication among the community • Important for validation, to exchange data in the emerging large scale European projects (GMES)

13. Ozone Validation: Gome Validation using several Lidar stations in a short time Comparison of Gome (red line) and Lidar data at OHP 1997 validation Green line: mean of [O3(Go)-O3(Li)]/O3(Li) Red line: rms Table Mountain Haute Provence Mauna Loa

14. Ozone profile retrieval from GOME (1997-2003) S. Casadio – ESA ESRIN (GOME 3D Ozone volume over Antarctica - Sept 02, NNO Level 2 products generate in EDG)

15. Additional Ozone data sets • Maintain/extend data availability on the Grid • all the level1/2 data and reference data (Lidar, Radiosounding) • Extend GOME experience to other datasets • Envisat Sciamachy, Gomos…. • Reprocessing with new algorithm and comparison • Generate level 3 data on the Grid • Assimilation of Ozone data into models (UK) • Study of Ozone holes

16. Potential EO Applications in EGEE: seismology • Model and Tomography (IPGP, France) • Simulation: propagation of waves in the Geoide • Numerous data and computations, access to databases • Test started in DataGrid by accessing a cluster • Orpheus Database (in discussion) • General seismology database located in Netherlands • Test to port a subset of the database on the Grid and to access it

17. Potential EO Applications in EGEE: Climate • Model: Atmosphere, Ocean, Hydrology, Atmospheric and Marine chemistry…. • European effort: PRISM, ENES, GEMS • Goal:Post treatment: Comparison of model outputs from different runs and/or institutes • Characteristics of model outputs : • Large volume of data (TB) from different model outputs, and experimental data • Run made on vector computer • Post-treatments: visualisation, statistics, comparison….

18. Climate : Strategy • Secure access to the mass storages where the data are stored (like the ESA portal) (prototype by IPSL+CEA is starting soon) • Difficulty: authorization to implement this access • Metadata Catalogues implemented on the GRID (Spitfire or RMC) • metadata catalogues : one or several? • access to models and data • Demonstration between two institutes (IPSL, DKRZ) • extend to the other institutes • Extension to other European laboratories linked via European projects(PRISM, ENES, GEMS) to develop coupled model, and to compare them.

19. Possible other extensions • In the Space community (scientific and operational), ESA has played an important role to facilitate the access to satellite data via the Grid, to convince the international and operational community of the interest of Grid and to develop some tools to make easier the use of satellite data via the Grid. • Other Satellite data: Envisat ASAR, MERIS, … • build on ongoing ESA funded projects • Integration of science tools and applications • following community interests

20. Material Means Starting point: Node at ESA: cluster, SE, UI Node at IPSL+IPGP at Paris University Computer Center : 6PC, SE(500Gb), UI Other at KNMI: 4 PC

21. Requirements • Continuation of EO Virtual Organization + VOMS • New EO Applications proposed in EGEE • NIKHEF and KNMI to maintain the EO-VO • ISPL/CNRS could lead participation to EGEE • ESA-ESRIN willness to provide data and infrastructure • Improvements: • data access (restriction) • the replica metadata catalogue • Important questions addressed by the owner of large datasets : • need on the grid secure, controlled and restricted accesses • Porting and query of their metadata catalogue (compatibility) • Update and mirroring

22. CONCLUSION • Earth Observation is ready to port new applications today due to the dissemination work towards our colleagues during DataGrid Project • Deployment of EO observations on the Grid will be driven by goals to be attained in the EU projects, Grid is not explicitly mentioned then no funded. • Interaction with the CrossGrid partners about meteorology, pollution, flood prediction…then possibility of new projects • CEOS (Committee of Earth Observation by Satellite) Meeting on June 22 and 23 in London with an invitation to EU representative Man power effort needed to help this deployment for EO