Envisat Validation Workshop Atmospheric Chemistry Validation Team Ground-Based Measurements and Campaign Database Subgro

1. Envisat Validation Workshop Atmospheric Chemistry Validation Team Ground-Based Measurements and Campaign Database Subgroup Campaign Database Terje.Krognes, tk@nilu.no Aasmund.Fahre.Vik, afv@nilu.no Norwegian Institute for Air Research (NILU)

2. PresentationOutline • Data Centre Status • Tools • Local end-user tools • WWW search tools • Commissioning Phase Activities

3. nadir web DOCS DATA ORIGINATOR DATA table.dat ASC2HDF HDF2ASC FILES http or ftp DATABASE Error email OK? http SEARCH DATA USER ftp to zardoz Data flow • Signed protocol required for access to tools, docs and files • ECMWF data requires additional protocol

4. spare Internet nadir http Storage Firewall zardoz ftp,telnet Data Centre Status (1/3): Hardware • The machines have been swapped between production and spare servers • The hardware has not been changed

5. Data Centre Status (2/3): Software • Web-server (nadir) runs Linux Redhat 7.2 with up2date ~weekly (6.2 at 2nd rehearsal) • File server (zardoz) runs Solaris 7, as before • Nusphere MySQL • Apache 1.3.22 (1.3.14) • MySQL 3.23.44 (3.23.32) • PHP 4.0.6 (4.0.4pl1) • IDL 5.4, as before • HDF 4.1r3, as before • No re-boot on nadir since April 2002 • Uptime on zardoz since April 2002 is approximately 99,98% (2 half-hour maintenance + re-boot)

6. Data Centre Status (3/3): Metadata • Metadata Guidelines updated by metadata board on 24 April 2002 • During commissioning phase, updates of various metadata table entries, without affecting structure: • Project-related information • Actual descriptors • Guidelines still applicable, but more clarifications to be added.

7. New user tools • Major re-write of the ASC2HDF file generation and checking tool around launch • 6 smaller upgrades of the ASC2HDF package after launch • Entire ASC2HDF source code (FORTRAN) published to users. HDF output routines may be combined with user’s own input and processing modules • IDL-tool to generate HDF data files (without error checking): IDL2HDF • IDL tool for reading and plotting HDF data files (HDF2IDL), to be downloaded and used locally • IDL subroutine to read HDF Cal/Val data, for integration into user-developed software (nilu_hdf_read_user.pro)

8. New web functionality - ECMWF • ECMWF plots (NH 2,5º*2,5º and T106) • ECMWF 10-day back trajectory plots • T106 extractions to HDF files

9. New web functionality - search and plot • Use of defaults • Facility to save and retrieve a search • Provision for downloading multiple files in tarball via web interface • Improved on-line plotting (also for multidimensional data) • Corrected display of comments to variables

10. File searching (1/4) • Defaults are marked with *** • Page initially opens without defaults • Page may be reset or set to saved defaults bys selecting a previously saved query • Files may be filtered by a large number of metadata parameters

11. File searching (2/4) • The default criteria page lists all criteria sets that this user has previously saved • Entries may easily be removed • The main search page will be loaded with the selected defaults when launched from this page • Defaults may include 4D space-time parameters and variable name wild-cards

12. File searching (3/4) • The NUMERICAL SIMULATION entry is now marked with *** since we selected it as default through the saved query page • Variable name / mode / descriptor have wildcard entries automatically added to the drop down lists. • Variable name and variable mode accept multiple selections • 4D space-time criteria may be entered as a single point, or as a max/min box • All files that include or touch the 4D criteria are listed as matches

13. File searching (4/4) • Files may be selected by posting date • Files updated after a given date may be selected • The file list may be sorted by file name, by submitter or PI, by posting date, or by AOID • If no criteria are selected, the entire file list will be requested

14. File List • The file list displays the current selection criteria • A single file may be downloaded immediately • Files may be marked and downloaded in a compressed tarball • Search criteria may be saved for later re-use (type a meaningful description before saving) • The Variables link brings up a variable list and other documentation contained in the file

15. Variable List • The variable list page displays all parameters contained in a file • Variable name, mode and descriptor are displayed • The measurement unit and corresponding SI unit are displayed • All documentation fields attached to the file are displayed • Additional documentation fields attached to a single variable may be displayed through the Comments link • Variables may be selected to generate an on-line plot (next slide) • Observe the dependencies listed for each variable – not all variable combinations will produce meaningful plots

16. Variable plotting (on-line browse)

17. Administrative tools • Web tools to report uploads/downloads • Web tools to archive/retrieve outdated data files • Web tools to check and maintain index database consistency and integrity

18. User activities • Users have sent approximately 200 emails with requests for accounts or updates to metadata, or with questions to HDF file generation • After launch focus has slowly shifted to requests for help with HDF file formatting issues and removal of outdated files • From May to November 2002 users have submitted 5880 HDF data files (570 of these were rejected due to formatting problems, and 238 have later been archived). • From May to November 2002 users have downloaded 4000 HDF data files (totalling 215 MB) (ftp downloads via zardoz not included) • There are currently 4741 ACVT files open to users (22 active AOIDs) • There are currently 462 MAVT files open to users (5 active AOIDs)

19. User activities: Up and Downloads The download statistics only covers WWW downloads from nadir, not FTP downloads from zardoz

20. Nadir-team activities • 300+ interactions with users (of which 50% during Commissioning Phase) • 250+ interactions with ESA and the Metadata Board (10% during Commissioning Phase) • One major re-write and 6 updates of the ASC2HDF and HDF2ASC programs • Maintained and improved shell scripts for automatic file checking and storage • Created additional IDL and FORTRAN tools • Performed extensive web program development • Performed user administration updates • Added metadata entries and updated the table.dat file 23 times since launch • Developed and maintained scripts and data flow to provide pre-processed ECMWF data extractions and plots on the web • Monitored the automatic file reception system (now containing nearly 5000 files totalling 275 MB + index database 20 MB)

21. Metadata issues Although many quality checks are performed on the data upon submission to the database: • The metadata definition is currently not complete enough to perform automatic checks of the validity of reported measurement units • Some components and units may still be inconsistently named or mis-used Several files have indeed been submitted containing such errors due to misinterpretation of the guidelines. The data submittors will be contacted to correct these files.