Use Case 1: Sciamachy Data Center

1. SciaGrid project results Use Case 1: Sciamachy Data Center Wim Som de Cerff John van de Vegte Richard van Hees David Groep Jan Just Keijser Maurice Bouwhuis Pieter van Beek

2. Content • What is the NL-SCIA-DC? • Why Grid? • Implementation • Results and outlook

3. What is Sciamachy? • SCIAMACHY is an passive imaging spectrometerSCanning Imaging Absorption spectroMeter for Atmospheric CartograpHY • Satellite instrument on the ESA ENVISAT satellite • Objective is to perform global measurements of trace gases (e.g. ozone, NO2, CH4, aerosols) in the troposphere and in the stratosphere. • The solar radiation transmitted, backscattered and reflected from the atmosphere is recorded at relatively high resolution (0.2 nm to 0.5 nm) over the range 240 nm to 1700 nm, and in selected regions between 2000 nm and 2400 nm. • SCIAMACHY has three different viewing geometries: nadir, limb, and sun/moon occultations which yield total column values as well as distribution profiles in the stratosphere and (in some cases) the troposphere for trace gases and aerosols.

4. Sciamachy product examples • Ozone hole Southern Hemisphere (October 2008)

6. Why is NL-SCIA-DC needed? • Complementing ESA’s distribution facilities • User need for fast and complete access to GOME and SCIAMACHY data • Supporting the development of Dutch algorithms • Distribution of Dutch data products • Domain specific search/query capabilities

7. Goals for NL-SCIA-DC Provide to the users: • Access to Sciamachy, GOME, MERIS and AATSR data • Selection methods, for easy selection of data • Downloading of selected datasets and products • Deployment of Dutch data products • Test environment for new data processors • (fast) dataset (re)processing capabilities

8. Overview of NL-SCIA-DC Tape Archive

9. Data • GOME level 1b, 2: from 1996 up to now • 1.5 Terabyte of data • metadata and data products databases • All pixels can be queried and browsed • Sciamachy: level 0, 1 and 2: from 2002 up to now • 40 Terabyte of data, and growing • metadata and data products database • Accessible through catalogue, including extracted metadata • All pixels can be queried and browsed • Archive and metadata database are automatically updated (satellite dish, ftp, DVD) All data can be accessed online Via browser or application

10. Data in the product databases • PostgreSQL 8.3 with PostGIS extension used • Database is now 112 Gbyte and growing

11. Users • The NL-SCIA-DC has 120 registered users from 22 countries, from 71 different organizations. • Bulk data users. Data is delivered directly to them by sftp. Current bulk data users (standing order) are KNMI, SRON, BIRA (Belgium), University of Heidelberg (Germany) and ISAO (Italy). • TEMIS (ESA)Tropospheric Emission Monitoring Internet Site (TEMIS) aims to compute and deliver global concentrations of tropospheric trace gases, and aerosol and UV products derived from observations of nadir-viewing satellite instruments such as GOME, SCIAMACHY and (A)ATSR. TEMIS is part of the Data User Programme (DUP) of the European Space Agency (ESA). The service of TEMIS centres around four themes: Air pollution monitoring, UV radiation monitoring, Support to Protocol monitoring, Support to Aviation control. • PROMOTE (GMES)To deliver the Atmosphere GMES Service Element a sustainable and reliable operational service to support informed decisions on the atmospheric policy issues of stratospheric ozone depletion, surface UV exposure, air quality and climate change

12. User interface ‘classic’ client – server Java Applet Search, process, download

13. Why Grid? • Datasets are large and not easily downloaded to a workstation • Users want to run their algorithm on a larger set of Sciamachy data • Running an algorithm on a large set takes too long on a single workstation • Algorithms are mostly embarrassingly parallel  very much suited to run in a Grid environment! • Also very interesting for reprocessing of data Sciamachy chains for metadata extraction, CH4 level 2, level 2 daily average, level 3 daily and plot processing

14. SciaGrid Project • Together with NIKHEF and SARA • NIVR GO financed project • Aim: ‘Griddify’ the NL-SCIA-DC • Share archives and databases at KNMI and SRON • Make data accessible for resources at NIKHEF and SARA (Grid) • Run NL-SCIA-DC jobs on Grid infrastructure, using the NL-SCIA-DC GUI In the project: • Experiments with Storage Resource Broker (SRB) • Robot certificate • Pilot job engine

15. Overview

16. Results SciaGrid • SRB did not solve our problem; Drawbacks: • Adding an existing archive is not easy • Licensing of SRB • Future? • Solved the metadata part in an other way, Grid FTP selected for data access • Certificates: NL-SCIA-DC has (first issued) robot cert! • Users can use their own login from NL-SCIA-DC to submit jobs • Pilot Job framework used • Gain better successful submission ratios • Minimize Grid component installations at KNMI/SRON

17. SciaGrid setup

18. Available Status NL-SCIA-DC Debug…

19. Summary and outlook • Grid experiment was successful  connection to the Grid established • Data is accessible at Grid resources • Jobs can be submitted using the NL-SCIA-DC GUI • Release of User interface asap so users can actually use the new functionality • NL-SCIA-DC operations in SciaVisie project • Grid component expanded in Big Grid (?)