1 / 18

Developments at the US GDAC

Developments at the US GDAC. ADMT #10 – Toulouse, France 30 September - 2 October 2009 Mark Ignaszewski FNMOC. Actions Completed. Automated Greylist submissions Implemented Technical File v2.3 format checking Implemented new “latest data” file handling

hinda
Download Presentation

Developments at the US GDAC

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Developments at the US GDAC ADMT #10 – Toulouse, France 30 September - 2 October 2009 Mark Ignaszewski FNMOC ADMT-10 30 September – 2 October 2009

  2. Actions Completed • Automated Greylist submissions • Implemented Technical File v2.3 format checking • Implemented new “latest data” file handling • Real-time and delayed-mode profiles in separate files • Files directly in “latest-data” (no year/month directories) • Files being limited to roughly 100Mb • Running in parallel with old method until GDACs are consistent and users are informed. ADMT-10 30 September – 2 October 2009

  3. Actions Pending • Automate file removal – Dec ‘09 • MD5 signature – Nov ‘09 • Will start when new format checker goes “live” • New Format Checking (separate agenda item) ADMT-10 30 September – 2 October 2009

  4. Enhanced Profile Checking Meta-data checks Date Checks QC code checks <PARAM> variable checks D-mode file checks ADMT-10 30 September – 2 October 2009

  5. Meta-data Checks • PLATFORM_NUMBER: • 5 or 7 numeric digits (second digit “9” for 7) • DATA_STATE_INDICATOR: • One of the recommended codes from reference table 6 • DIRECTION: ‘A’ or ‘D’ • DATA_CENTRE: Valid for the DAC • DATA_MODE: ‘A’, ‘D’, or ‘R’ • INSTRUMENT_REFERENCE: Set (Warning) • POSITIONING_SYSTEM: Set (Warning) • WMO_INST_TYPE: Valid (ref table 8) (Warning) ADMT-10 30 September – 2 October 2009

  6. Date Checks • All dates are checked for validity and consistency • DATE_UPDATE and DATE_CREATION must be set • String date settings checked for validity • DATE_UPDATE / DATE_CREATION / HISTORY_DATE / CALIBRATION_DATE • 14 digit strings; valid (e.g., seconds must be 0 to 59) • Consistency checks ADMT-10 30 September – 2 October 2009

  7. Consistency of Dates Jan 1, 1997 DATE_CREATION GDAC file time HISTORY_DATE DATE_UPDATE JULD JULD_LOCATION CALIBRATION_DATE ADMT-10 30 September – 2 October 2009 • All dates must be after Jan 1, 1997 and before the file time • JULD and JULD_LOCATION within 1 day • No specific ordering among HISTORY_ and CALIBRATION_DATE • On file submittal by DAC: DATE_UPDATE within 2 days of time submitted. (WARNING) • Needed to allow for time zone “fudge factor”

  8. QC Code Checks JULD_QC and POSITION_QC: Valid values ADMT-10 30 September – 2 October 2009

  9. <PARAM> Checks • STATION_PARAMETER: • Only valid parameter names • No “blank” entries (WARNING) • No duplicate entries (WARNING) • PRES, TEMP, PSAL are required • Check that <PARAM> variables exist for every STATION_PARAMETER • Check that no other <PARAM> variables with data exist in the file. • If mode = ‘A’ or ‘D’: Check that all <PARAM>_ADJUSTED have data • Subject to the rules in the QC manual (i.e. QC = ‘4’) ADMT-10 30 September – 2 October 2009

  10. <PARAM> Checks (continued) • <PARAM>_QC and _ADJUSTED_QC • Only valid QC codes - No “fill values” • Missing data flagged with 0, 4, 9 • Real-time profiles: Only codes 0 through 4 • Required parameters (PRES, TEMP, PSAL): • Cannot be code 0 if data is not missing • PROFILE_<PARAM>_QC: Check value • Check that N_PARAM and N_LEVEL are not larger than necessary ADMT-10 30 September – 2 October 2009

  11. D-mode File Checks • DATA_MODE = “D” • DATA_STATE_INDICATOR = “2C” or “2C+” • Same <PARAM> in PARAMETERS as in STATION_PARAMETERS • If PRES_ADJUSTED_QC = 4, TEMP_ADJUSTED_QC and PSAL_ADJUSTED_QC = 4 • *_ADJUSTED = missing ADMT-10 30 September – 2 October 2009

  12. D-mode File Checks • SCIENTIFIC_CALIB_COMMENT and CALIBRATION_DATE set • for every <PARAM> and N_CALIB • At least one HISTORY record • HISTORY_INSTITUTION and _DATE set • Plus, the <PARAM> and date checks previously discussed ADMT-10 30 September – 2 October 2009

  13. Results: Warnings • 9 DACs have problems with NULLs in strings • KORDI seems to be OK • Some only in few variables, some in many variables • A couple “N_LEVELS too large” • KORDI sets the variable larger than necessary a lot • 1 “INSTRUMENT_REFERENCE” not set (JMA) ADMT-10 30 September – 2 October 2009

  14. Results: Date checks • JULD after DATE_CREATION • Coriolis – only a few files • INCOIS – many files – large time differences • HISTORY_DATE and/or CALIB_DATE after DATE_UPDATE • CSIO – Many files – Big time differences • Invalid dates: • AOML, INCOIS: Bad values • MEDS: Too short (missing seconds) ADMT-10 30 September – 2 October 2009

  15. Results: <PARAM> checks • ‘A’ or ‘D’: *_ADJUSTED not set • Identified some missing *_ADJUSTED data • Also found some issues with the tests • PROFILE_<PARAM>_QC: Check values • AOML, Coriolis, CSIO, JMA, MEDS • Missing variables • CSIRO, INCOIS • <PARAM>_QC and *_ADJUSTED_QC • Numerous inconsistencies reported • Some illegal values ADMT-10 30 September – 2 October 2009

  16. Results: D-file checks • DATA_STATE_INDICATOR: Coriolis and MEDS: • Question about “2C+” • A few MEDS files set to “2B” ADMT-10 30 September – 2 October 2009

  17. Results: D-file checks PARAMETER and SCIENTIFIC_CALIBRATION_* • PARAMETER or CALIB_DATE not set for many files • PRES, TEMP: AOML, CSIO, MEDS • TEMP, CNDC: CSIRO • PRES, TEMP, PSAL: JMA • SCI_CALIB_COMMENT not set for many files • TEMP, CNDC: CSIRO • PRES, TEMP: JMA • N_CALIB larger than necessary in many files • BODC (many files), Coriolis (few files) • No calibration information in some D-files: Coriolis ADMT-10 30 September – 2 October 2009

  18. Plan Implement new checking in “DAC test”: 15 Oct 2009 Implement new checking in routine processing AS ADVISORY: end Oct 2009 Transition to IFREMER: Oct 2009 Implement as operational checker: Nov 2009 ADMT-10 30 September – 2 October 2009

More Related