1 / 33

EDUCE Database Tasks: Installation, Submission, Retrieval, and Data Integrity

This summary highlights the tasks within WP4 of the EDUCE database project, including hardware installation, web data submission tools, Oracle database structures, web page provision, metadata updates, testing, and user guides.

jesusgerald
Download Presentation

EDUCE Database Tasks: Installation, Submission, Retrieval, and Data Integrity

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Tasks within WP4&The EDUCE database - A summaryWP1, WP2, WP4 and WP5 tasksJussi Kaurola, Niilo Siljamo, Antti ArolaOzone and UV-Radiation Research GroupFinnish meteorological InstituteContributions: Tim Martin (UG), Brian Gardiner&Peter Kirsch (BAS), Harry Slaper&Peter den Outer (RIVM), Ola Engelsen (NILU), Martin Stanek (CHMI), Ann Webb&Richard Kift (UMIST), Massimo Ferraguto (SSF), ...

  2. Tasks within WP4 (deliverables): 4.1. Installation of new hardware, permitting... (FMI) 4.2. Installation of WWW data submission tools (FMI) 4.3. Addition of further Oracle structures for non-spectral and ancillary data (FMI, BAS) 4.4. Provision of web pages to provide search access to the Oracle database and guidance to users (FMI) 4.5. Update and maintain metadata pages for the database, including flag for WMO guidelines (UMIST) 4.6. Testing of and report on all database search and access tools, and publishing of user's guide and manual, with copy made available at web site (UG)

  3. The EDUCE database - A summaryWP1, WP2, WP4 and WP5 tasks: • WP1 & WP2 • recent changes in the database • screen shots • submission facilities • retrieval facilities • data integrity • hardware issues • plans for the future • FMI tasklist • summary

  4. FMI task in WP1: "RAF for ozone and albedo" Antti Arola • "The irradiance ratio of snow-covered to snow-free conditions tells how much snow enhances the irradiance on average. • The average albedo RAF in Sodankylä, calculated from 11 years of data, is about 1.19. In other words, snow enhancement is about 19%" • Report 1.4 (11pages):http://www.muk.uni-hannover.de/~martin/results/deliverables/1.4/wp_1_4.pdf

  5. FMI task in WP2: "2.1An evaluation of the impact of atmospheric variability on the detection of trends in UV"Antti Arola • The developed methodology offers a way of separating the effect of a single factor, e.g., ozone, cloudiness, aerosols or albedo, on the short- and long-term variability of UV, using measurements of spectral irradiance • Report 2.1 (13 pages):http://www.muk.uni-hannover.de/~martin/results/deliverables/2.1/wp2.1_finalreport.pdf • Arola A., K. Lakkala, A. Bais, J. Kaurola, C. Meleti, and P. Taalas, "Factors affecting short- and long-term changes of spectral UV irradiance at two European stations", JGR, doi:10.1029/2003JD003447, 2003.

  6. Recent changes in the database • quality flags: • old data flagged "manually" • incoming data flagged automatically • submission: • auxiliary data • retrieval of data: • flags included in the retrieval of spectra • broadband data • auxiliary data: daily values (hourly values not yet) • better information of recent submissions, enhanced user options for output (CHMI)

  7. Data submission facilities • spectral and broadband submission has been in operation since 2001 • the submission of auxiliary data was delayed due to problems with (old) software • now the submission works also for aux data • "short and friendly guides" for spectral and auxiliary data • spectral guide should be OK (after several iterations) • auxiliary guide was revised two weeks ago • planned changes: • speeding up of submission process • make it work more as a background process

  8. Data retrieval facilities • changes made since the last meeting • new oracle table: measurement_type(spectral, broadband, ozone, cloud, dursun, snowcover) • broadband data can be retrieved • daily auxiliary can be retrieved (currently not hourly data) • flags included in headers and can be retrieved separately • some fixes of previously reported problems • planned changes • fix problems with hourly auxiliary data • new user requirements for WPI design, give us feedback! • spectral retrieval somewhat slow and error-prone (queue problems). A completely new way to make SQL query and consequent data handling is planned

  9. flagging of one spectrum • 149263#056 GREEN • GREEN Cloudy 3.639e+02 -1.000e+05 8.380e+01 -1.000e+05 -1.000e+05 • GREEN -1.200e-02 14 • GREEN 1.700e-02 67 • GREEN 1.010e-05 6.600e-06 • GREEN 1.850e+00 0 0 0 • GREEN CLOUDS 2.860e-01 3.207e-01 • SOME_VARIATION_SCAN 9.240e+00 • 9.766e-04 slf00567 • /UVROOT/DB/DBroot/no_ntnu_trh/1997/ol752/11010000.gxa 56

  10. Data integrity: duplicates • what is a duplicate? • a measured spectrum which has several copies in the database • several different spectra have the same measurement time • what is an overlapping spectrum? • several different spectra have overlapping measurement time • both types are included in the data retrievals • both types need to be removed

  11. Data integrity: duplicates • duplicates caused by errors in submitted data • automatic identification: very complex task, not possible with current software • manual work is needed • current status (August run by the BAS tool):belsk bre064 128bru jytot 148855

  12. spurious zeroes • during the flagging spectra with only zero values in irradiance column were recorded • 41345 spectra found • only 93 cases with sza < 90 degrees • at_boku_sonnblick 2at_boku_vienna 1be_iasb_bru 53fi_fmi_sod 2fr_ustl_vda 5gb_umist_reading 11it_jrc_ispra 27no_nilu_andoya 8

  13. Hardware status • the current hardware is becoming old(purchased in 2000 with $40 000) • disks are almost full, Oracle disk at critical level • recent submissions have been very large • total disk space 106 GB • Oracle data size 56 GB • Flexstor data size 23 GB • CPU power is adequate, flagging might be lagging a bit in the case of large submissions • disk failure on last Thursday, which was handled very well by the FMI IT department

  14. Planned changes • some hardware upgrading is needed: • poor mans solution to disk problem:new disks: 4 x 150 GB (4x600 euro)to current disk box (2 for data, and 2 for mirror) • better solution: new disk array (about 15 keuro)and some new disks. Much more tolerant for errors. • best solution: new server (20 keuro), new disk array (15 keuro) and new disks • a short break in service, a longer break in uploading of new data to Oracle • operating system upgrade in November-December causing 2-3 days break • upgrading of e.g. WPI related software (TOMCAT)

  15. Planned changes... • currently everything is in ozone2.fmi.fi server which is outside FMI firewall ("grey area") • an alternative solution: • server with Flexstor and WPI in "grey area" • Oracle inside firewall • a better integration of EUVDB Oracle database to FMI other Oracle databases. Benefits: • security • maintenance • even performance?

  16. FMI task list • follow the link on "General Information Page" • updated regularly • check the "wish list" if you have ideas for new improvementsSome remarks:-campaign data • manual submission by FMI • retrieval from www pages needs planning and work

  17. Summary • personal opinion: now the functionalities of the database are rather good • some FMI milestones were delayed considerably • the whole project time was more or less development period, which certainly has caused headache to some users • remaining work: hourly auxliary data • planned improvements: • smoother submission of large datasets • faster retrieval of spectral data • improved WPI design • our interest is to continue with the database maintenance. Funding to be discussed later...

More Related