Data processing and quality control activities in support to the SCIAMACHY mission

1. Data processing and quality control activities in support to the SCIAMACHY mission • Dehn(1), F. Niro(1), A. von Bargen(2), T. Fehr(3) • (1) Serco SpA, ADehn@serco.it, FNiro@serco.it (2) DLR, Albrecht.von-Bargen@dlr.de (3) ESA/ESRIN, Thorsten.Fehr@esa.int DPQC activities Abstract The DPQC is a consortium of specialized companies collecting engineering and scientific expertise all over Europe. The DPQC role is to provide on behalf of ESA the service of product monitoring and processor implementation for the ERS and ENVISAT mission as for the future ESA missions. While performing these tasks the DPQC interacts with ESA as well as with the instrument provider (AOIP) and the science user community (QWG). Whenever an anomaly is detected during the quality check an investigation is started by involving all the necessary expertise. The outcome of the investigation is a remedy that could imply a workaround solution or a re-processing instance; in some cases the solution could drive the evolution of the processor baseline and/or the review of the calibration and validation plan. The final outputs of the DPQC service are the daily and monthly reports that are delivered to the science user community. Furthermore the DPQC contributes to workshop and meetings in support to the mission. A schema showing the activities performed by the DPQC and the main entities involved in this process is shown in Fig. 1. • The objective of this paper is to highlight the importance of the satellite instrument and products monitoring and to show the implication of this task for the scientific user community. The quality of the satellite data can be variable due to several sources of instrument and products degradation due to: • The Space Segment (e.g.: anomaly in the satellite communication, instrument ice contamination) • The Ground Segment (e.g.: anomaly in activation of ADF, processor implementation errors) The quick detection of all instrument and product anomaly is crucial for instrument safety, as well as for optimizing the quality of the products before delivery to the users. Furthermore the investigation on the detected anomaly could drive the development of the operational processor and could help to optimize the instrument operations and the calibration plan. In fact the quality control is a dynamic process that evolves in order to satisfy the new requirements coming out during the mission. In this paper we will address the quality control measures focusing on the SCIAMACHY mission. The monitoring baseline and evolution and the processor upgrades will be outlined; furthermore the implications of these activities for the scientific user community will be underlined. Fig. 1 - Flowchart showing the DPQC anomaly handling process Mission and processing status Validation data set Valid. Groups • The SCIAMACHY instrument performs nominally. No particular anomalies have been detected since begin of mission, instrument unavailabilities are mainly due to single event upsets (SEU). • The following operational SCIAMACHY products are provided to the user community: • Level 1b (SCI_NL__1P) NRT and off-line: Geo-located and engineering calibrated Scientific measurements • Level 2 Off-line (SCI_OL__2P) : Vertical column amounts of O3, NO2… retrieved from the Nadir measurements and stratospheric profiles of O3, NO2 derived from Limb scans • Additionally • Level 0 data (SCI_NL__0P), consisting in the instrument source packets and telemetry data are available on the ground segment for monitoring and calibration purposes. • Table 1 summarizes the actual processing configuration. Before a whole mission re-processing is initiated with an upgraded processor version, a scientific validation is performed based on independent geophysical correlative measurements in order to review the quality of the Level 1b and Level 2 data products. The SCIAMACHY validation is organized by the SCIAVALIG team, who also defines the Validation Data Set corresponding to validation campaigns from ground based, balloon, aircraft or satellite inter-comparison measurements. Tab. 1 – SCIAMACHY Processing configuration. D-PAC FTP Server • The generation of the Validation Data Set (VDS) is performed within the DPQC as well as the required full mission re-calibration (generation of ADFs). Fig 2. demonstrates the process in order to retrieve a new VDS. • For SCIAMACHY a VDS of 3885 NADIR/LIMB states and 78 full orbits were processed with • Level 1b IPF 6.02 • Level 2 Off-line processor 3.00 • The results of the Validation analysis were presented during the ACVE-3 in December 2006 in ESRIN and were used to give reccomendations on processor improvements especially in view to full mission re-processing. Level 1b Level 2 4 ADF types (LK1, SU1, PE1, SP1) L0 extrac-ted child Product L0 full Product L2 OL Prod. VDS L1 Prod. L0 extrac-ted child Product (VDS) Extract. Scripts SCI-CAL The current Product Specification for level 1 and 2 products is 3k IPF L0-L1 v6.02 L2 OL Proc.3.00 Fig. 2 – SCIAMACHY Validation Data set generation steps Monitoring activities An example of monitoring results of the leakage current variations in all eight SCIAMACHY channels are shown in Fig. 5. In this figure the leakage constant part FPN (fixed pattern noise) of the leakage current calibration measurements are analyzed by determining the ratios of the FPN of each month with a time distance of one orbit, one day, one week, two weeks, three weeks and a month. We can observe from this figure that the leakage variations are sometimes higher than the noise; furthermore a large noise in the IR channels is visible. Monitoring configuration Tab. 2 – Main quality indicators for SCIAMACHY Products SCIAMACHY is an Announcement of Opportunity Instrument. Therefore both, DPQC and the Instrument Provider, have responsibilities with respect to the instrument and products performances. In particular SOST-DLR and SOST-IFE are responsible for the instrument configuration and performances, while the DPQC, ESA and DLR contribute to the level 1 and level 2 performance assessments. This paper will address only the monitoring activities performed by the DPQC consortium. The quality check provided by the DPQC is extended to all the product levels (L0, L1 and L2). The QUADAS tool is used to extract the quality information parameters from the products and to generate a wide variety of plots and table to be reported in daily and monthly report. Table 2 lists the most important quality indicators in the products, however many more parameters are inspected using QUADAS tool. Fig. 5 – SCIAMACHY Long Term Trend analysis on leakage current, FPN Daily monitoring Another example of level 1 monitoring results is shown in Figure 6, where the leakage variation for selected pixels in channels 6-8 as a function of the orbit phase (12 values between 0 and 1) are shown. The orbital variation is due to the thermal background that varies along the orbit. Especially for infrared products retrieved from detector channel 8, the leakage variable correction improves the quality of these products. The orbital variation follows a sinusoidal curve. Outliers for individual pixel are due to the bad or dead pixels. QUADAS reports are checked daily in order to detect anomalies in the Flight or Ground Segment. The level 0 daily monitoring provides a first check of the operational processing performance, since it allows detection of anomalies in the acquisition or transmission of the instrument source packets. Furthermore the level 0 monitoring is important to monitor the instrument behavior, especially house keeping data like detector temperatures. The monitoring of L1b products allows the quality check of the spectra and to verify the correctness of the spectral and radiometric calibration as well as the leakage current calibration. With the level 2 Off-line monitoring the scientific validity of the LIMB and NADIR data can be checked, in particular concerning any problem in the operational retrieval. Fig. 6 – SCIAMACHY leakage variable analysis channels 6-8 In the last part of the SCIAMACHY BMR the level 2 processing configuration is reported and the results of the level 2 Off-line data monitoring are shown. The level 2 products monitoring is focused on the consistency check of the Nadir NO2 vertical columns. An example of NO2 world map plots is shown in Figure 7 and Figure 8, where the NADIR NO2 vertical column density (VCD) values averaged over one month (Feb 2007) are shown together with their averaged errors. A high concentration of NO2 over industrial regions, as over North America, central Europe, China and South Africa are visible regularly. Anomalous high values of VCD errors can be also observed in the region of Scandinavia, this anomaly is currently under investigation. Examples of daily report plots are presented in Fig. 3 and Fig. 4, where the NO2 vertical column in a global map and the O3 stratospheric profiles are shown respectively. These plots demonstrate that the quality monitoring process involves scientific expertise in the verification of the data. It is interesting to see in these plots the maxima of NO2 concentration over China and the typical ozone profile shape in the stratosphere. Fig. 8 – SCIAMACHY NO2 VCD error, mean value Februray 2007 Fig. 7 – SCIAMACHY NO2 mean VCD February 2007 Anomaly investigations and monitoring evolution In addition to the routine monitoring activities, ad-hoc analysis are carried out by the DPQC in order to solve instrument or processing anomalies and to improve the monitoring efficiency. Fig. 3 – Level 2 off-line monitoring: NADIR NO2 data Fig. 4 – Level 2 off-line monitoring: LIMB O3 data Source of information Long term analysis The SCIAMACHY long term trend analysis is documented on a bi-monthly basis and a report is made available to the users. The Bi-Monthly Report (BMR) is composed of analysis carried out by the DPQC in collaboration with DLR, ESA and SOST. An overview of the content of this report is hereafter presented. The first part of the report is compiled by the SOST-team and describes the planned operations and the instrument status. Further instrument and platform anomalies are reported, and long term analysis on the light paths and PMD monitoring results are shown. In the second part of the report inputs from the DPQC are included. Initially the products availability statistics in the processing centers are reported. In a second chapter the level 1 processing configuration and products quality is described. In particular the sun mean reference spectra and the leakage current variations over the reporting period are analyzed for every detector channels. Data quality disclaimers provide information on known deficiencies in processing, and on transient degradations not yet compensated http://envisat.esa.int/dataproducts/availability/disclaimers/ SCIAMACHY instrument monitoring reports generated by the SOST team inform about instrument operations and performance http://atmos.caf.dlr.de/projects/scops/ Daily/monthly reports, inform about calibration and processing configuration, anomalies, degradation, and performance, they can be accessed at: http://earth.esa.int/pcs/envisat/sciamachy/reports/ The unavailability intervals of the SCIAMACHY instrument are available on line http://envisat.esa.int/instruments/availability/ For any questions on SCIAMACHY, please contact ESA’s SO Helpdesk eohelp@esa.int