910 likes | 931 Views
Marine Weather & Oceanographic Services Data Flow Project. Data Management To identify and document the life cycles of marine & oceanographic related data and associated processes including quality control and reporting requirements. Contents. Australian Argo : array of profiling floats
E N D
Marine Weather & Oceanographic Services Data Flow Project Data Management To identify and document the life cycles of marine & oceanographic related data and associated processes including quality control and reporting requirements
Contents • Australian Argo: array of profiling floats • Observation Systems Section: Marine Telemetry • In situ observations data via satellite transfer (Future) • Remote Sensing • Space Based Observations Section • Radar Engineering Group • Other programs/products • Global Sea Level Observing System (GLOSS) • Sea Ice • Ocean colour • Deep Ocean Time-Series Station • Ocean Research Vessel • Great Barrier Reef Sea Temperature • Objective: International Programmes & Local Responsibility • JCOMM and JCOMMOPS • CBoM Marine/Ocean Components • CBoM Marine/Ocean System Components • MARS • DODS • Surface Based Observations Section: Marine Operations Group • Automated Shipboard Aerological Programme (ASAP) • Australian Voluntary Observing Fleet / Voluntary Observing Ships (AVOF/VOS) • XBT Ship Of Opportunity Program (SOOP) • Australian Buoy Program • Drifting/Moored Buoys: Data Buoy Coop Panel (DBCP) • Waverider
Objective • The objective of this presentation is to identify and document the life cycles of marine & oceanographic related data and associated processes including quality control and reporting requirements. • The Commonwealth Bureau of Meteorology has both local and international commitments for the collection stage (incorporating quality control), distribution, usage, archival and retrieval stages of the data’s life. • From an international level there is: Joint WMO-IOC Technical Commission for Oceanography and Marine Meteorology (JCOMM) & World Weather Watch (WWW). • Local responsibilities are shared within the following branches in the Bureau: • Bureau of Meteorology Research Centre (BMRC) • Observations & Engineering Branch (OEB) • Central Operations & Systems Branch (COSB) • National Meteorological & Oceanographic Operations Centre (NMOC) • National Climate Centre (NCC) • Services Policy Branch & Regional Forecasting Centres (RFC)
JCOMM JCOMMOPS Context Diagram • JCOMM is a programme for Joint WMO-IOC Technical Commission for Oceanography and Marine Meteorology. It is an intergovernmental body of experts, which provides the international, intergovernmental coordination, regulation and management mechanism for an operational oceanographic and marine meteorological observing, data management and services system. • JCOMMOPS is a centre to provide support at the international level for those in charge of developing and operating marine and oceanographic in situ observing systems.
JCOMMOPS L1: Context Diagram VOS (AVOF) SOOP DBCP AST (Argo) JCOMMOPS DBCP & SOOP • JCOMM: Joint WMO-IOC Technical Commission for Oceanography and Marine Meteorology • JCOMMOPS: JCOMM in situ Observing Platform Support Centre • VOS: Voluntary Observing Ships • AVOF: Australian Voluntary Observation Fleet • SOOP: Ship of Opportunity Programme • DBCP: Data Buoy Coop Panel • Argo: Global array of profiling floats • AST: Argo Science Team • AIC: Argo Information Centre AIC (Argos) JCOMMOPS Database Monitoring Products and Services
Monitoring and Prediction Weather/Ocean/ Climate Services Outputs & Ext Parties Observations and Engineering Branch: National Meteorological and Oceanographic Operations Centre: Oceanographic Systems Development Sub-section Services Policy Branch: Marine Weather Services National Climate Centre: Climate Analysis Section Regional Forecasting Centres Outputs: Mail/internet, Radio, Media, Data Networks Space Based Observations Central Operations and Systems Branch: Central Computing, Meteorological Systems and Data Management Satellites Observation Systems: Marine Telemetry Group Surface Based Observations: Marine Operations Group External Parties: NODC, MEDS, GTSPP, CMR, AODC, GCC, GDAC, DBCP, PSMSL In Situ platforms: -Ships -Buoys -XBTs -Floats -Radio sondes -Tide gauges Waverider Agencies GTS MTN JAFOOS / CSIRO Marine Research National Tidal Facility / National Tidal Centre Bureau of Meteorology Research Centre: Ocean & Marine Forecasting Research & Development Marine/Oceanographic Components System • The following sections within the Bureau have an involvement in Marine / Oceanographic data. • External parties also assist in the input, processing, archive and output of the data.
CADAM (NCC) CMSS GTS MTN SitesDB WMC RSMC ClimateDB Marine/Oceanographic System Components Real Time Systems Climate and Metadata Systems AIFS QADAM Further QC Receiving obs data: BUFR, BUOY, SHIP, TESAC, BATHY, WAVEOB, SATOB, McIDAS, etc. 10 days of real time data Obs & NWP Decoder NEONS RTDB RTH SHIP / BUOY Data convert to sfc_ship_obs format and MQC CMSS QC DADAM ADAM EVE TCZ
QADAM CADAM Quality control procedures operated on data copied from CADAM Corporate database directly capturing data from RTDB. Updates only made through applications from RTDB or QADAM DADAM Development database Logical ADAM Database Structure ADAM: Australian Data Archive for Meteorology
ADAM QC for SHIP/BUOY data • Due to the high volume of data, quality control is performed automatically at the point of ingestion. These checks were programmed in 1995 to meet WMO reporting requirements (e.g. International Maritime Meteorological Tape-IMMT format). Checks were divided into two types (detailed list): • Syntactic checks: To ensure the structure meets reporting requirements. For example: • Invalid callsign • Bad temperature or wind-speed • Substantive checks: For the reasonableness of the content within each observation. For example: • Constraint Checks • Consistency check for the same element for example the minimum temperature must be less than or equal to the maximum temperature. • Consistency Checks for related weather elements.
Syntactic checks: -Data sorted chronologically within callsign -Data from one country only -Invalid callsign -Duplicate date-time -Ill-formed date-time-position, inappropriate 29th Feb etc. -Bad temperature indicator -Bad wind-speed indicator -Bad air temperature sign -Bad quadrant -Missing latitude -Missing longitude -Bad latitude -Bad longitude -Bad cloud height / visibility indicator -Badly-defined wind direction -Badly defined wind speed -Wind direction non-null and wind speed 0 -Invalid air temperature sign -Badly-formed temperature -Badly formed dew point -Invalid wet bulb sign -Badly-formed pressure -Badly-formed present weather -Invalid SST sign -Swell direction badly formed -Ice accretion badly formed -Country of origin badly formed -Overall QC indicator badly formed -Rainfall group badly formed -Second swell direction badly formed -Sea ice report badly formed -Badly formed past weather Substantive checks specified by GCC: -Positional change -Cloud data -Wind speed -Temperature -Pressure -Present and past weather -Sea Surface Temperatures -Wind waves period -Wind waves height -First swell height -Second swell height -Rainfall -Ship’s speed and ship’s direction Substantive checks specified by NCC: Detailed analysis of cloud observations, including a range of internal consistency checks on the heights of the low / medium / high clouds and the relationship between them. If the QC process finds anything amiss, QC flags are adjusted, and, depending on the nature of the problem, values may be adjusted. To my considerable surprise, and I suspect to my colleagues' surprise as well, it appears that around 25% (sic) of cloud observations are adjusted or flagged in this processing. Wind direction and wind speed mutual consistency checks. About 1% of relevant records are flagged to indicate that the original observations did not look completely OK. Checks of mutual consistency involving wave period, wave height, swell heights, swell periods, and wind speed. ADAM QC checks at Ingestion
Computer Message Switching System (CMSS) QC process • The following QC checks are performed at CMSS recognition: • format: header, message type • date and time • missing or incorrect station numbers • ship and buoy positions • unknown aviation location points • duplicates
MARS: Meteorological Archive/Retrieval System Future Architecture • Supports archive, retrieval and post-processing of meteorological data. • Used both in operational and research environments and stored in the following formats: • BUFR (Binary Universal Form Representation): Observation data maintained by NMOC. The “MARSA” archive contains daily, monthly or periodic data extracted from model run, analysis or other sources that are use operationally. • GRIB (Grid in Binary): Model data maintained by BMRC. The “MARSB” archive contains data that are forwarded by researchers and scientists in BMRC, either as as means of keeping the bulk of their data for the duration of test run in their research, or for long terms archive. • Using Storage Area Network (SAN) to provide a “bottomless” storage of model and observation data. • MARS is integrated with operational visualisation tools for efficient access to its database. • MARS Client communicates with Server via requests consisting of a delimited ASCII string the clearly identifies the action, and pairs of parameters & values (search pattern). This can be via either the Data Browser (see picture) or command line. • The client/server architecture is flexible and adaptive to changes in the environment. Hence clients may be embedded in different software applications, e.g. Metview, McIDAS, etc. • This system is similar to Distributed Oceanographic Data System (DODS).
Commercial Confidence CADAM (NCC) CMSS GTS MTN SitesDB WMC RSMC ClimateDB Proposed MARS Implementation Real Time Systems Climate and Metadata Systems AIFS QADAM Further QC Receiving obs data: BUFR, BUOY, SHIP, TESAC, BATHY, WAVEOB, SATOB, McIDAS, etc. 10 days of real time data Obs & NWP Decoder RTDB (NMOC) RTH SHIP / BUOY Data convert to sfc_ship_obs format and MQC CMSS QC DADAM Processed Obs Convert data: Observation – BUFR Model – GRIB Convert data Mass Store ADAM DODS Server MARS Bottomless storage of observation & model data Post processed products EVE TCZ
Data Analysis Application DODS server DODS: Distributed Oceanographic Data System • The Distributed Oceanographic Data System allows you to access data over the internet, from programs that weren't originally designed for that purpose, as well as some that were. • The DODS architecture uses a client/server model, with a client that sends requests for data out onto the network to some server , that answers with the requested data. • With DODS, data is accessed using a URL. • DODS data is stored in binary form, and by default, it is transmitted that way, too. • Using flexible data types suitable for many uses, including scientific data, the DODS servers deliver data directly to the client program in the format needed by that client. • DODS provides sophisticated sub-sampling capabilities to prevent requesting the whole dataset that may not meet the user’s requirements. • By developing network versions of commonly used data access Application Program Interface (API) libraries, such as NetCDF , JGOFS , BUFR, GRIB, and others, the DODS project can capitalize on years of development of data analysis and display packages that use those APIs, allowing users to continue to use programs with which they are already familiar. • DODS server and DODS GRADS client (installed on gale) will be managed by BMRC, with IT support from COSB. • DODS is similar to Meteorological Archive/Retrieval System (MARS). DODS Client Dataset DODS server Internet link (http) Dataset DODS server Local Dataset
Surface Based Observations Section: Marine Operations Group • The role of the Surface Based Observations Section is to ensure that Bureau and national requirements for meteorological observations are satisfied by reviewing, developing and implementing meteorological observing systems, in collaboration with other Sections, Branches and Regional Offices in the Bureau, and by liaison with other organisations. • A sub-section of this section that specifically handles marine and oceanographic data is the Marine Operations Group. • The role of the Marine Operations Group (MOG) is to manage the Bureau of Meteorology's Marine Observing Program, comprising: • Port Meteorological Agents and the Australian Voluntary Observing Fleet (AVOF); • the Expendable Bathythermograph Ship-of-Opportunity Programme (XBT SOOP); • drifting, moored and wave rider buoy programs; • profiling float program (Argo); • automated shipboard weather observing systems (ASAP); and • all future operational marine networks.
INMARSAT CMSS Decoder RTDB ASAP L1: Automated Shipboard Aerological Programme • The Automated Shipboard Aerological Programme (ASAP) is a WMO marine programme providing upper air soundings at sea. • It involves the generation of upper air profile data from data sparse ocean areas using automated sounding systems carried on board merchant ships plying regular ocean routes. • The profile data are all made available in real time on the GTS, for use by operational centres. • Australia had operated an ASAP unit, and the programme was coordinated through the ASAP Panel, a component of the JCOMM Ship Observations Team. • Most of the soundings are presently from the North Atlantic and North West Pacific Oceans, but the programme is also expanding into other ocean basins, most notably through a new, cooperative Worldwide Recurring ASAP Project (WRAP). • The ASAP Panel (ASAPP) publishes an Annual Report, giving programme status and statistics on data return and data quality. • There are currently no WRAP ships involved in the Australian region. Raw observation data TEMP SHIP messages Converts into TEMP SHIP messages (UUAA format) Perth LES receives raw data and forwards to CMSS COSB relay observation data to interested parties NMOC use observation data and insert into Upper Air models Wind Temperature Humidity Air pressure MOG holds metadata for ship only (Not launches) Develop Annual report to be sent to UK Met Office
VOS/AVOF L1: Voluntary Observing Ships/ Australian Voluntary Observing Fleet Data Flow diagram • Member countries of the World Meteorological Organization (WMO) are encouraged to maintain national fleets of Voluntary Observing Ships (VOS) to take, record and report weather observations whilst at sea. • The international Voluntary Observing Ships' Scheme is coordinated by the VOS Panel, a component of the JCOMM Ship Observations Team (SOT). • Australia participates in the VOS scheme through the Australian Voluntary Observing Fleet (AVOF). The AVOF consists of Australian and foreign owned merchant, research, passenger and private vessels, which operate mainly in the Australian region as shown on the accompanying map. Member vessels of the AVOF often assist the Bureau in other areas of marine meteorology and oceanography, such as deploying drifting buoys or profiling floats, or participating in the XBT SOOP. • VOSClim is an ongoing project within the Voluntary Observing Ships' Scheme. It aims to provide a high-quality subset of marine meteorological data, with extensive associated metadata, to be available in both real-time and delayed mode to support global climate studies. • Data from the project will be invaluable for climate change studies and research. In particular it will be used to: • input directly into air-sea flux computations, as part of coupled atmosphere-ocean climate models; • provide ground truth for calibrating satellite observations; • provide a high quality reference data set for possible re-calibration of observations from the entire VOS fleet.
PMA Global Collecting Centres (GCC): Ensuring Minimum Quality Control (MQC) GCC Bracknell,GB GCC Hamburg,DE Log Book Turbo AMDCP ShipAWS NOAA INMARSAT CMSS Argos Toulouse FRGPC GTS MTN AVOF L2: Data Flow Diagram Inspections metadata?? VOS Metadata Delayed mode data Data, logbooks, graphs and inspections data MOG FTP/ WWW (IMMT2 & BBXX codes) Monthly Report for monitoring Scientific Research Community CADAM Archive National Climate Centre VOSClim Delayed mode QC data Real time data AIFS NMOC HQC Responsible Members (RM): Applying second level QC HK IN GB US RU DE JP NL DAC (US): Data Assembly Centres RFC Model results data Near real time raw data Air pressure and air temperature, humidity, wind speed and wind direction Climatological Summaries for Ocean Areas of Responsibility Data & Std climatological products on request RTDB Obs data RTMC NMC Near RT data Near RT data via Perth LES (SHIP message: BBXX) GTS sub-system QC process RTH Products and Services: -Real Time Data -Non Real Time Data -Climatological Statistics Near RT data (SHIP message: BBXX) transferred to GTS Meteo France RTH RTH Near RT data (SHIP report: BBXX)
PMA AVOF L3A: Ship Inspections • Ship inspection data is obtained via the following methods: • Recruitment of a new ship • As part of a routine inspection • Via the PMA (e-mail or telephone) • This metadata is updated on the ShipsDB (now migrated to SitesDB), maintained by MOG • Equipment failures and replacements are also handled by OEB via SitesDB. • Some of the inspections are also performed by OICs and regional staff, and inspectors in non-PMA states. • In future, this data will be sent to the Data Assembly Centres for global archive. Inspections metadata (inspection/survey reports) Inspections metadata (inspection/ survey reports) DAC (US): Data Assembly Centres MOG CADAM Archive
PMA Global Collecting Centres (GCC): Ensuring Minimum Quality Control (MQC) GCC Bracknell,GB GCC Hamburg,DE Log Book Turbo ShipAWS AMDCP AVOF L3B: (Delayed mode) VOS Metadata Paper logbooks; Electronic logbooks (IMMT2 & BBXX codes); Barograph charts MOG FTP/ WWW (IMMT2 & BBXX codes) - Yearly VOSClim Quarterly Paper logbooks DAC (US): Data Assembly Centres HQC Responsible Members (RM): Applying second level QC HK IN GB US RU DE JP NL CADAM Archive National Climate Centre Data & Std climatological products on request Paper logbooks (SHIP & additional group code) Climatological Summaries for Ocean Areas of Responsibility Scientific Research Community Air pressure and air temperature, humidity, wind speed and wind direction Monitoring and predictions products: -WWW -Publications -Community Notices Products and services NMC Products and Services: -Non Real Time Data -Climatological Statistics
Log Book Turbo AMDCP ShipAWS Argos Melb INMARSAT NOAA CMSS GTS MTN AVOF L3C: (Real time) Products and Services: -Real Time Data -Near Real Time Data -Climatological Statistics CADAM Archive National Climate Centre Air pressure and air temperature, humidity, wind speed and wind direction Near real time data (SHIP message: BBXX) Near real time raw data Seasonal outlook Model results data AIFS NMOC RFC Other NMC RTDB Obs data Near real time data (SHIP message: BBXX) DAC (US): Data Assembly Centres Scientific Research Community Near real time data via Perth LES (SHIP message: BBXX) GTS sub-system QC process RTH BUFR code & Model data Near RT data (SHIP message: BBXX) Monitoring Statistics Meteo France RTH Real Time Monitoring Centre RTH Near RT data (SHIP message: BBXX) transferred to GTS Near real time data (SHIP message: BBXX)
AVOF L3D: Ship Monitoring Future Process • At the beginning of each month, the following components are actioned automatically from an MS Access database (ship_monitoring.mdb) to generate a monthly ship performance monitoring report: • Data required from ADAM (DATA_HOLDER_SFC_SHIP_92ON) and Ships97.mdb, I.e. observations count and metadata, respectively. • Query compiled to list AVOF ships during report month and observation period. • Extracts observations data file (number of BBXX messages received by CMSS) and generates corresponding table (for faster calculation). • Counts total of observations per ship for each synoptic hour (0000 0600 1200 1800). Observation totals for ships that record observations manually are also entered. • Report is generated and posted on MOG website. The production of the report provides a count of the observations submitted and present in the NCC ADAM database by the Australian Voluntary Observing Fleet (AVOF) at the time of compilation. • UK Met Office provide a monthly report on the data quality of observations via the GTS. Report details include total observations per callsign, % gross error, average observation bias and normalised standard deviation. • Receive monthly Data Quality Monitoring Report (txt file) from UK Met Office at the beginning of the following month. • Perform visual QC of statistics and note any major exceptions for further investigation (e.g. may notify PMA to contact ship officer to identify the cause of bad measurements). • Copy values for pressure, wind speed and non-reporting ships and paste into Excel template. • Format report and save as pdf file. • Post pdf file on MOG website and send to interested parties (some parties as a hardcopy). ADAM 2.Extract obs for AVOF ships during month 3.Count obs total for each ship per synoptic hour Obs data 5.MOG generate Ship Performance Monitoring Report and post on website Monthly Performance Report for AVOF ships Ships97 1.List AVOF ships during month 4.Manual observation counts are entered Metadata Monthly Data Quality Report for AVOF ships 10.Post on website and send to interested parties 6.Receive text file from UK Met Office 7.Perform visual QC of result to id exceptions for further investigations. 8.Copy pressure & wind speed values into Excel 9.Format report and save as pdf.
Commercial Confidence AVOF L3D: Ship Monitoring (Future) Current Process • Monthly Performance Report and Monthly Data Quality Monitoring Report will be sourced from NEONS-RTDB and can be combined into one report. • Metadata will be obtained automatically from MOG as part of the report generation process. • Data Management will develop a report that replicates the details from both the Performance Report and the Data Quality Monitoring Report. • Reporting parameters will be set by Data Management and MOG (e.g. ability to add observations manually for vessels with callsign SHIP, account for late observations, ). • The report will be generated automatically, I.e. third day of the following month. • The report will be posted on a BoM website as HTML and pdf. • MOG can link to DM webpage and can send url to interested parties, instead of files. MOG Meta data and misc observations Monthly Data Quality & Performance Report for AVOF ships DM generate Ship Data Quality Monitoring & Performance Report MOG links report to website and notifies interested via url. RTDB Observation data
AVOF L4A: (Real time) Data Assembly Centre – NOAA, US • Extract GTS reports of project ships (by call sign) and decode (including additional project data). • Receive the real time reports from the Real Time Monitoring Centre • Collect delayed mode reports of project ships from participants. • Merge real time and delayed mode reports, eliminate duplicates and compile a complete project data set. • Collect metadata and survey reports for project ships and compile a complete data set. • Make project data sets available to users on request. • Maintain a project web site, information exchange mechanism and electronic newsletter. 1.Extract & decode GTS reports 6.Provide project datasets to users (e.g. NMC) 4.Merge real time and delayed mode reports 4.Eliminate duplicates 4.Compile complete project data set Real Time Monitoring Centre 2.Receive real time reports 6.Maintain project website, info exchange & newsletter 3.Collect delayed mode reports 5.Collect metadata and survey reports
AVOF L4B: (Real time) Real Time Monitoring Centre - UK • Extract GTS reports of project ships (by call sign) and decode. • Associate project observed variables (pressure, air temperature, humidity, SST, wind speed and direction) for each project ship with co-located model field values (4 times daily). • Compile data sets of observations and associated model field values and transfer to the Data Assembly Centre. • Provide ship monitoring statistics for all VOSClim ships to the Data Assembly Centre (monthly). 3.Compile datasets & transfer to DAC 1.Extract & decode GTS reports 2.Associate variables into models (4 times daily) DAC (US): Data Assembly Centres 4.Provide monitoring statistics to DAC (monthly)
Turbo • Developed by KNMI (Royal Dutch Meteorological Institute) and endorsed by WMO, Turbo is similar in concept to the Bureau's Electronic Field Book. • The Observing Officer manually enters the observed weather details into a computer running the Turbo software. • The program then performs quality control and consistency checks on the data before appending the data to an archive file on the computer. • A BBXX message is also generated which can be saved to floppy disk ready for transmission by Inmarsat. • TurboWin is a user-friendly system with over 200 built-in quality checks.
ShipAWS • The ShipAWS is based on the Vaisala Milos 500 Automatic Weather Station (AWS). • Sensors for air pressure, air temperature, humidity, wind speed and wind direction. • A Global Positioning System (GPS) • A serial data connection to the ship's compass (together with the GPS enable the true wind to be derived from the apparent wind). • Communications is via a dedicated Inmarsat C terminal. • Displays the instantaneous and averaged (remote and derived) data. • A manual console for entering the visual parameters to complete the weather observation.
AMDCP • The Argos Marine Data Collection Platform (AMDCP) consists of a transmitter, a data entry keypad, an omni-directional antenna, and sensors for air pressure and air temperature. • Automatic - remotely sensed air pressure and air temperature, averaged over 10 minutes, • Manual - manually encoded ship's weather observation, including the visual parameters, entered using a hand-held keypad. • Only one vessel is still currently using this system. It will be decommissioned and replaced by end of 2003 financial year.
XBT SOOP L1: Ship-Of-Opportunity Program Data Flow diagram • The XBT SOOP is a network of merchant and research ships that launch expendable bathythermographs (XBTs) to measure/record the thermal structure of the upper 900m of the oceans. • The primary goal of the Ship-of-Opportunity Programme (SOOP) is to fulfil upper ocean data requirements which have been established by GOOS and GCOS, and which can be met at present by measurements from ships of opportunity (SOO). • Data management is taken care of through the Global Temperature Salinity Profile Programme (GTSPP). • The upper ocean thermal data collected by the XBT SOOP is a major reason for our increased knowledge of the interaction between the atmosphere and the oceans, and is an indicator of major climate events such as El Nino. Apart from climate prediction, the XBT SOOP supports other operational needs (including fisheries, shipping and defence) by providing upper ocean thermal data for assimilation in models and ocean analysis schemes. • A standard XBT system consists of a launcher, an expendable probe (see the schematic diagram at right), a control unit located on the bridge for data processing, recording and message transmission. An electrical connection is formed between the probe and the control unit when the canister containing the probe is placed in the launcher and secured. A thermistor is located in the nose of the XBT probe. • Following the launch of an XBT, signal wire unravels simultaneously from the spools inside the probe and canister to ensure the probe free falls from the sea surface unaffected by ship motion or sea state. The depth is determined from an assumed fall-rate of the probe. • A small current travels down the signal wire from the control unit which detects changes in the water temperature as changes in the resistance of the circuit as the XBT probe descends. The changes in resistance are processed by the control unit and converted into measurements of absolute temperature. The XBT can measure temperature to an accuracy of +/- 0.15o C, with a resolution of better than 0.01o C. • On completion of an XBT drop, and after the profile has been checked for ambiguities or gross errors, a standard BATHY message containing low resolution XBT data is prepared automatically and transmitted using the Argos system for distribution on the Global Telecommunications System (GTS).
PMA Temperature profiles Argos Toulouse FRGPC NOAA/TIROS-N CMSS Joint Aust Facility for Ocean Observing Systems (JAFOOS) Products and Services: -Non Real Time Data -Climatological Statistics GTS MTN XBT SOOP L2: Data Flow Diagram Delayed mode data Feedback Raw delayed mode data (Full resolution) Disk Ship Greeter’s log Voyage summary Observer’s log sheet XBT processing unit SOOP-TC SOOPIP Feedback XBT Summary Report (6mths) QC following CMR QC Cookbook MOG Real time data E-mail report/ delayed mode data RTDB NMOC QC Data Archive Raw real time data (Low resolution) Annual QC dataset via FTP/Web Near RT data (BATHY message: JJYY) GTS sub-system QC process Dataset (Weekly- Monthly) NODC Global Temp Salinity Profile Project Scientific QC following CMR QC Cookbook GTSPP USERS Near RT data (BATHY message: JJYY) MEDS RTH QC following GTSPP QC Manual Near real time data (draft BATHY) RTH Near RT data (BATHY message: JJYY) Dataset (3 times a week) Meteo France RTH Real Time Data Archive Near real time data (BATHY message: JJYY) transferred to GTS
F634:XBT Ship Greeter’s Log F636:XBT Voyage Summary F635:XBT Observer’s Log Sheet NODC GSTPP MEDS XBT SOOP L3: (Delayed mode) – BOM QC process Future Process Send XBT plot profile to PMA PMA feedbacks results to Ship crew Info includes: -position of the cast -weather -sea conditions at drop time -equipment malfunctions Random Access data MEDS ASCII data Determine if disk is readable Raw data XBT Data Manipulation QC Process Per semester, send to users via web/email Log sheets filled in by the ships crew as the probes are released. QC following CMR QC Cookbook Approximately 1 month turnaround per cruise Transect XBT data from ships via PMA MA data (Twice yearly) MA data (Yearly) Send yearly XBT dataset Convert .qclist into Summary Report Useful if an error in logging the data onto the computer has been made or if the weather/sea-state has affected the quality of the data. QC SOOP documents for potential errors during voyage Send XBT Summary Report to SOOPIP / WMO Dataset (Weekly- Monthly) JAFOOS Web/data server CMR: Ann G AODC GTSPP USERS Log sheets are archived CD back-up after report generated JAFOOS: Red MOG: Green Dataset (3 times a week)
Commercial Confidence F634:XBT Ship Greeter’s Log F636:XBT Voyage Summary F635:XBT Observer’s Log Sheet NODC GSTPP MEDS XBT SOOP L3: (Delayed mode) – BOM QC process (Future) Current Process Send XBT plot profile to PMA PMA feedbacks results to Ship crew Info includes: -position of the cast -weather -sea conditions at drop time -equipment malfunctions Replaced with new Java based system Random Access data MEDS ASCII data Raw data Determine if disk is readable XBT Data Manipulation QC Process Per semester, send to users via web/email Log sheets filled in by the ships crew as the probes are released. QC following CMR QC Cookbook Approximately 1 month turnaround per cruise Transect XBT data from ships via PMA MA data (Twice yearly) MA data (Yearly) Send yearly XBT dataset Convert .qclist into Summary Report Useful if an error in logging the data onto the computer has been made or if the weather/sea-state has affected the quality of the data. QC SOOP documents for potential errors during voyage Send XBT Summary Report to SOOPIP / WMO Dataset (Weekly- Monthly) JAFOOS Web/data server CMR: Ann G AODC GTSPP USERS Log sheets are archived CD back-up after report generated JAFOOS: Red MOG: Green Dataset (3 times a week)
PMA F634:XBT Ship Greeter’s Log F636:XBT Voyage Summary F635:XBT Observer’s Log Sheet XBT SOOP L4A: (Delayed mode) – Document Review The following documents/disk are received from the PMA, and the details are checked accordingly to determine if there are any possible issues with the data captured: • Diskette with full resolution XBT data checked if readable • If not readable, the PMA is contacted to retrieve the information from the Hard Disk on the ship’s system • Ship Greeter’s Log (F634) • Visit information correct • Check equipment condition (e.g. Launcher, cable, test canister, computer, antenna, ground connection) • Test system (e.g. internal clock meets GMT, temperature approx. 1.5o, transmission of message) • General comments: To detail any other issues and information • Supplies details and check-list • Observers Voyage Summary (F636) • Voyage information correct • Supplies remaining at end of voyage • Test system (e.g. temperature reading similar at beginningand end of voyage) • General comments: To detail any other issues andinformation • Observers Log Sheet (F635) • Each launch meets operational cruise requirements • Check weather conditions • Check comments for malfunctions 1.If disk not readable, request PMA to retrieve data from HDD 1.Determine if disk is readable 2/3/4.Review SOOP documents for potential errors during voyage
XBT SOOP L4B: (Delayed mode) – Data manipulation • Assign cruise identifier • XX###### where X identifies the ship and # identifies the voyage • Run “XBTDisk” • Create directory with same name as cruise identifier • Create XX######.SIP file which is concatenation of all casts into one file • Transfers .sip file to UNIX server • Run “mixedBOM” • Create .MA (MEDS ASCII) file • Create .SUM file • Run “plot_sip” IDL program • Creates a map of the station positions and a plot of each drop (4-casts-to-a-page) from the .sip data file. • Provide XBT cruise plots/maps to PMA who relay report to ship’s personnel for feedback on success of drops. • Run “convertM2RA” • Converts MEDS ASCII file to random access format • Incorporate into dataset for QUEST (Quality Evaluation for Subsurface Temperatures) which is used for QC • Adds prefix to indicate period under review, e.g. BOMyyyy • Archive current dataset (pre-QC) on XBT server prior to loading new data Approximately 1 month turnaround per cruise 2.Create .SIP file via “XBTDisk” 3.Create .SUM & .MA files via “mixedBOM” 5.Convert file to QUEST RA file via “convertM2RA” & auto load into BOM2003 6.Archive dataset (pre-QC) on XBT server 1.Assign cruise identifier 4.Run “plot_sip” 4.Send XBT plot profile to PMA 4.PMA feedbacks results to Ship crew
XBT SOOP L4C: (Delayed mode) – QC process • Review graphical contents and metadata of files via QUEST to perform engineering QC (I.e. system error trace) • Archive dataset before scientific QC process • Execute scientific QC process via QUEST • QUEST is a Fortran program designed to visualise the casts in the context of both climatology and neighbouring casts along the ship track. This is in order to identify real features of a given region and to help distinguish between such features and erroneous data. • It allows QC codes (as per the CMR QC Cookbook) to be allocated to the XBT dataset to flag possible errors. • Archive dataset after scientific QC process • Run “convertRA2M” program • Creates .qclist file which is in MEDS-ASCII format • Perform post-check • Run “Waterfall” IDL program prior to distributing the QC listing • Waterfall creates a plot of casts from QC’d data, which allows the operator to review the drop Approximately 1 month turnaround per cruise When all data processed for each semester Redo if post check negative 1.Review graphical contents of files to trace for possible system errors 2.Archive dataset before scientific QC process 3.Execute scientific QC via QUEST 4.Archive dataset after scientific QC process 5.Convert .RA file to .MA using “convertRA2M” 6.Run “Waterfall” and perform post-check
NODC GSTPP MEDS XBT SOOP L4D: (Delayed mode) – Data Dissemination • JAFOOS disseminate QC processed data to users via web and e-mail • Twice yearly to MOG • Yearly to GTSPP (NODC), AODC, CMR, and JAFOOS web/data server (archive) • MOG uses .qclist to create Summary Report • MOG sends XBT Summary Report to SOOPIP / WMO • MOG back-ups dataset onto CD after Summary report sent to SOOPIP / WMO • GTSPP forwards reports to users of data. • MEDS has 6 users that receive data three times a week. • The US/NODC has more than a dozen users receiving data either weekly or monthly. 2.Convert .qclist into Summary Report 3.Send XBT Summary Report to SOOPIP / WMO 4.CD back-up after report generated MOG JAFOOS web/data server 1.Per semester, send to users via web/email Dataset (Weekly- Monthly) AODC GTSPP USERS CMR Dataset (3 times a week)
Australian Buoy Programme L1 • The Australian Buoy Programme consists of two sub-programs: • Drifting Buoy Program • The Bureau is involved in the deployment and maintenance of drifting and moored buoys within our assigned region by WMO. • The Bureau also participates at an international level via DBCP and IBPIO. • The Data Buoy Cooperation Panel (DBCP) is a official joint body of the World Meteorological Organization (WMO) and the Intergovernmental Oceanographic Commission (IOC) which was formally established in 1985. The most important task of the DBCP is to ensure a satisfactory the coordination at the international level of the Drifting and Moored Buoy programs. • Waverider Buoy Program • The Bureau has a strategy to develop a National Waverider Buoy Program based around existing waverider networks operated privately or by state or local authorities. These networks will supplement the Bureau's own waverider buoy located off the west coast of Tasmania near Strahan. • Wave data is also received from networks operated by external agencies in other states: • Manly Hydraulics Laboratory (MHL - NSW) • Beach Protection Authority (BPA - Qld) • Department of Planning & Infrastructure (DPI – WA) • Woodside Petroleum (WOOD – SA) • WA Petroleum (WAPET – WA) • Sydney Water (SYDWT – NSW) • Esso Australia Ltd (ESSO)
SVP Moored FGGE Buoy L1: Drifting and Moored Data Flow diagram • The Bureau deploys two types of drifting buoys; FGGE and SVP • The FGGE, or spar type buoy is available as either: • standard, with air pressure, air temperature and sea surface temperature, or • wind buoy, with air pressure, air temperature, sea surface temperature, wind speed and wind direction. • and comes either: • drogued with a weighted line attached to the base, and drifts with the sub-surface currents, or • undrogued, and drifts with the surface wind and waves. • The SVP (Surface Velocity Program) buoy was originally developed for oceanography and consists of a surface float (approximately 40cm in diameter), a smaller sub-surface float and a holey sock drogue. • Currently no moored buoys owned by BoM. The Bureau had plannedto take delivery of a 3 metre discus buoy from Axys Technologiesin 2003, however that has been deferred because of the Bureau'srelocation. (Refer to time-series station)
SVP Moored PMOC Argos-France DBCP - TC FGGE NMCs Argos Toulouse FRGPC NOAA/TIROS-N Other CMSS GTS MTN Buoy L2: Data Flow Diagram Delayed mode data Visual QC of Drifting Buoy data for equipment problems External Parties E-mail QC change requests Daily telnet for AOML SVP BUOY observation data MOG PTT sends 2 identical values for verification DBCP QC process for GTS data Upper Air & Marine Telemetry Group E-mail new version Monthly Report for monitoring Develop report Near RT raw data Air pressure Air pressure tendency Air temp Sea surface temp Sub-surface temp Sea surface salinity Sub-surface salinity Wind speed Wind direction Wave period Wave height Solar radiation Ocean currents Conductivity Turbidity Fluorescence Daily telnet for BUOY observation data (Date, Time, Location) using ELSA99 Monthly Buoy Status Report for platforms using BUOY code Via e-mail & snail mail CADAM GTS sub-system QC process PI / PGC Relay new Buoy details NCC Seasonal outlooks Real time data Near RT data (BUOY message: ZZYY) Model results data RTDB NMOC AIFS RFC Meteo France Near real time data (BUOY message:ZZYY) Near RT data (BUOY message: ZZYY) transferred to GTS Obs data RNODC/ DB Real Time Data Archive RTH RTH RTH Near real time data (BUOY message: ZZYY)
Service Argos-US Service Argos-Melb NOAA NOAA Buoy L3A: Daily Buoy monitoring • FGGE and SVP Buoys: • Daily telnet to Service Argos (Melbourne) for TX BUOY data using ELSA99. ELSA99 displays the platform locations based on observations data from CLS Argos. • MOG uses an MS Access database to upload the data to develop a cartography map (ELSA99) to track the buoys. Buoy tracking information is posted on MOG website. • Back-up ELSA99 database onto another PC every Monday & Thursday using Event Manager. • AOML SVPB Buoys: • Several AOML SVPB buoys have BoM “sponsored” barometers installed. This data are only available via Service Argos (US). Limited profile access to Service Argos (US) requires the following automated process to be actioned rather than direct processing by ELSA99. • Daily telnet session at 07:00 AEST. File “AOML.txt” created, which cannot be read by ELSA99. • MOG run Clipper program “AOMLSVPB.exe” used to convert data. It runs at 07:05 AEST. • Scans “AOML.txt” for new PTT/Date/Time. • If new, adds details to “AOMLSVPB.dbf” (holds all historic data). • Creates another database called “NEWDATA.dbf” which overwrites text file “NEW.txt”. NEW.txt has PTT code translated with AOML reference and is in csv format. • If Sunday/Monday, then append to existing file “NEW.txt”. A copy of this file stored on Orville server. • Query within ELSADAT.mdb uses NEW.txt to append to ELSA99 database. Again, MOG use ELSA99 to track the buoy locations. 1.Daily telnet from Melb Service Argos for BUOY data 2.ELSA99 used by MOG to track buoys 3.Back-up ELSA database every Mon & Thurs on other PC MOG 2.Telnet file on AOML SVPB buoys from US Service Argos 3.Run AOMLSVPB.exe to convert data for ELSA99 4.ELSA99 used by MOG to track buoys
Buoy L3B: Monthly Buoy monitoring • Prior to generation of the monthly buoy reports, all issues identified (for buoys the Bureau is interested in) via the Buoy-QC mailing list are reflected in the metadata database. • On a daily basis, a process running on server Gale extracts all buoy data for the previous 24 hours from the RTDB into a flat file. • At the beginning of each month (2nd day), an Excel application is utilised to extract the required data for the previous month from each flat file into a new Excel workbook. • MOG uses this workbook to formulate the statistics contained in the monthly buoy report. • For example, platform additions & omissions, deployments, listing of active platforms with sensor status, owner listing, and platform location. • A visual QC check to confirm the validity of the WMO number and other data items (I.e. metadata and observational data). • MOG also uses MapINFO to plot the Buoy locations at month end. This is attached to the monthly buoy report. • The report is saved as a pdf file and posted on the MOG website, sent to several users via email and post. 1.Update buoy metadata db with confirmed Buoy-QC issues 4.Make changes to metadata database BUOY97.mdb 2.Extract daily buoy data and generate flat file 3.Start of month Excel workbook generated containing flat files 4.QC check on validity of data 4.MOG generate Status Report based on data from workbook 5.Use MapINFO to generate plots of Buoy locations at month end 6.Report saved as pdf and posted on website and sent to users Monthly Buoy Status Report for platforms using BUOY code RTDB
DCBP QC Guidelines for GTS data • The scheme is based on an Internet distribution list (i.e. mailing list) used by all parties involved in the process. • Principal Meteorological or Oceanographic Centres (PMOC) responsible for Buoy data Quality Control can make status change proposals by the mean of an Internet mailing list (BUOY-QC@VEDUR.IS). • The Technical Co-ordinator of the DBCP, acting as a focal point between these centres and the owners of the buoys forwards the proposals to them. • For more information refer to http://www.dbcp.noaa.gov/dbcp/2qgd.html • There is a web-based QC process (prototype) called QCRelay which will replace this e-mail based Buoy QC. Currently only PIs have access to this website “bulletin board”.
Waverider Buoy L1 Data Flow diagram • The Waverider is used to measure surface wave height of the ocean. It is possible to measure sea surface temperature and wave direction, but these are not used by the Bureau. • The basis of the system is a spherical buoy tethered by a mooring to follow the vertical motions of the water surface. Within the buoy is an accelerometer which is mounted so as to only detect the vertical movement of the buoy. The signal from the accelerometer is converted to a radio signal and transmitted through the whip antenna to a base station. The data can also be recorded by an optional logger mounted inside the buoy. • A directional Waverider detects the horizontal motion of the buoy as well as the vertical movement and hence is able to calculate the direction of the wave motion. Typically such a buoy would output the heave, east and north component of movement as well as the directional energy spectrum in digital form. • The mooring of a Waverider buoy consists of a long rubber cord (15 m minimum for a non-directional buoy and 30 m for a directional system), in addition to wire or fibre rope and chain. These robust components of the mooring are normally twice the water depth. The Waverider buoy is fitted with a navigation warning light to comply with normal navigation requirements. • Waverider buoys are very sensitive to rotation and must neverbe rolled or spun as this is likely to damage the suspension ofthe accelerometer and render the Waverider buoy unserviceable. • There are currently 28 stations. All are sending messages to theBureau in WAVEOB format (except two stations, that transmitin BUOY format). • The Bureau currently own only two Waveriders. The rest areowned by the other agencies.
Base Station Wave height Wave direction Wave CMSS GTS MTN Waverider Buoy L2: Data Flow Diagram Metadata ADAM MOG Metadata • Base stations receive and distribute the data with no pre-defined timing to the Agencies (normally via dial-up). No QC performed on raw data. • Agency converts raw data to BUOY/WAVEOB format. • Agencies own, maintain and store the data as they own most of the Waveriders. There is no central storage of the processed data (e.g. BoM agreed not to archive this data). • Agencies/RFC sends data to NMOC via ftp to CMSS Assign WMO number Metadata FTP Wave rider data BUOY/ WAVEOB messages (ZZYY/MMXX) Agencies RFC Near real time data Waverider Buoy Observations for platforms using BUOY/WAVEOB codes Model results External Parties (e.g. Joint operators) NMOC RTDB FTP Wave rider data BUOY/ WAVEOB messages (ZZYY/MMXX) RTH External Parties RTH
Waverider Buoy L3: Agencies Data Flow diagram • Data from waverider is obtained via dial-up. Each agency dials up at times agreed with the Bureau. • Convert raw data into BUOY/WAVEOB format automatically. • Send data to RFC or NMOC via ftp. • No visual QC of data due to “real-time data” requirements. QC would have been performed manually. • Generate Waverider Buoy Observations report (Sometimes by RFC) • Each of the following agency owns, maintains and archives data: • Manly Hydraulics Laboratory, NSW (MHL) • Environmental Protection Agency, Qld (EPA) • Department for Planning and Infrastructure, WA (DPI) • Woodside Petroleum, (WOOD) • West Australian Petroleum Pty. Ltd, (WAPET) • Sydney Water, (SYDWT) • Esso Australia Ltd, (ESSO) • Bureau owns two waveriders.
Argo L1: Australian Programme Data Flow diagram • Argo is a global array of 3,000 proposed free drifting profiling floats designed to measure the temperature and salinity of the upper 2000m of the ocean. • The floats will repeatedly monitor the oceans and supplement the existing upper ocean thermal (XBT SOOP) networks by extending the spatial and temporal coverage, depth range and accuracy, and enhancing them by providing salinity measurements. • A profiling float is a one metre long upright cylinder with a onemetre antenna mounted on top. The float contains a batterypowered pump and bladder which allows it to control its ownbuoyancy. The batteries have a nominal five-year lifespan. • The ARGO cycle requires the float to sink to 1,500 or 2,000mbelow the surface and drift for 10 days. It will then change itsbuoyancy and record the temperature and salinity profiles asit ascends to the surface. • Once at the surface, the recorded profile data willbe transmitted to a processing centre via satellite,before the float alters its buoyancy and descendsto restart the cycle. Each float is expected to lastbetween 4 - 5 years.
NOAA/TIROS-N Argos Toulouse FRGPC CMSS GDAC GTS Sub-System; Coriolis, France (Sensor data processing, QC and encoding) Godae, US GTS MTN Profiling Float (10 day cycle) Argo L2: Data Flow Diagram Future Process Real time data MOG Telnet Raw PTT data TESAC messages sent for time stamp Products and Services: -Real Time Data -Climatological Statistics NMOC FTP / WWW TESAC messages sent for time stamp Monthly status report on floats RTDB Data Mgt FTP / WWW TESAC messages: KKYY (12hrs x 2 per day) CMR GTS Bulletins (TESAC message: KKYY) MEDS Real Time Data Archive Telnet Raw PTT data E-mail Near RT data (TESAC message: KKYY) Location & processed sensor data PI GTS Bulletins (TESAC message: KKYY) Registration of new float info GTS sub-system QC process RTH Launch details via online form RTH GTS Bulletins (TESAC message: KKYY) NODC RTH Near real time raw data (3hrs) AIC France Yearly archive QC Data Archive Metadata in NetCDF
Commercial Confidence Profiling Float (10 day cycle) NOAA/TIROS-N Argos Toulouse FRGPC CMSS GDAC GTS Sub-System; Coriolis, France (Sensor data processing, QC and encoding) Godae, US Argo L2: Data Flow Diagram (Future) Current Process Delayed mode data QC Data Archive Metadata in NetCDF PI/ RDAC: CMR Registration of new float info Delayed QC NetCDF obs data for specific region (6mths) Yearly Monthly status report on floats Launch details via online form MOG NetCDF RT QC Location & processed sensor data (TESAC message: KKYY) in 24hrs NODC AIC France Message time stamp MARS Real time data JAFOOS TESAC message Products and Services: -Real Time Data -Climatological Statistics Telnet raw PTT data Data Mgt NMOC FTP / WWW E-mail QC Location & processed sensor data (TESAC message: KKYY) TESAC message: KKYY (12hrs x 2 per day) GTS sub-system QC process FTP / WWW GTS Bulletins (TESAC message: KKYY) Real Time Data Archive MEDS GTS Bulletins (TESAC message: KKYY) Near real time Raw data (3hrs) RTH RTH GTS MTN RTH GTS Bulletins (TESAC message: KKYY)
Service Argos Argo L3A: Registration of New Float • The registration information is sent to Service Argos to confirm delivery of the float. • After the new float has been deployed, the Principal Investigators (PI) requests a new WMO identifier from the Marine Observations Group (MOG). • Each unique float number is pre-issued from the WMO (WMO identifier: 59#####) to represent a region. • There is no confirmation from the WMO of the identifier being issued. • CMR send the Argo Information Centre details of the launch via online form. • Both CMR and MOG hold a list of instruments, profiles and current float status for the region. The metadata may be different due to their requirements. • The data transmitted on the GTS uses this WMO number as the unique identifier. 1.Relay registration information to Service Argos 2.Request unique WMO float identifiers from MOG 59##### 2.Register new float details with MOG 4.Store & maintain metadata PI/CMR MOG 3.Launch details sent to AIC AIC
Service Argos Argo L3B: Real Time processing • CMR telnets raw PTT data from Service Argos whenever a float is expected to submerge (4 hours lag time is added to this process; up to 22 hours for data retrieval). • MOG also obtains raw data, for analysis of processing time and monthly status report on floats: • Clipper program used to strip data and calculate transmission times and location. This report is sent to CMR and other parties interested in float status. • Data is decoded and stripped for calibration. Data is calibrated using Matlab program. • Matlab program is used to apply Real-time QC, e.g. Spike and out-of-range tests. Errors in the calibrated data are replaced by missing values and flagged as “Bad” in the raw dataset. • QC’d, calibrated data is converted to TESAC format. • Message is sent to COSB via e-mail and picked up by CMSS. • Message is also sent to MOG as part of analysis of processing time. • COSB verifies message, strips unwanted data (e.g. headers and footers) and concatenates it with other TESAC data. • COSB releases bulletin unto the GTS once within every 12 hour block. • MOG picks up the transmission and measures the processing time between CMR to BoM to GTS. An alarm is generated if messages are not received in a timely basis. 2.MOG obtain raw data at same time as CMR 2.Develop monthly report on status of floats 2.Send to CMR and other interested parties 7.Message sent to MOG for time stamp checking 10.MOG analyses & compare transmission times 1.CMR obtain raw PTT data 3.Decode, strip and calibrate data 4.Real time QC on calibrated data 5.Convert data to TESAC 6.Message sent to COSB via e-mail 8.COSB verifies, strips & concatenates TESAC data 9.COSB release GTS bulletin every 12 hours