1 / 36

BLUElink>: Operational Issues and Product Delivery

BLUElink>: Operational Issues and Product Delivery. Graham Warren National Meteorological and Oceanographic Centre Central Operations Branch Bureau of Meteorology. Overview. NMOC Role Scheduling Observational Data Requirements and Management Status of Current Systems Product Delivery

Anita
Download Presentation

BLUElink>: Operational Issues and Product Delivery

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. BLUElink>: Operational Issues and Product Delivery Graham Warren National Meteorological and Oceanographic Centre Central Operations Branch Bureau of Meteorology

  2. Overview • NMOC • Role • Scheduling • Observational Data Requirements and Management • Status of Current Systems • Product Delivery • Internal Users • External Users • Research Users Second BLUElink> Users Forum

  3. NMOC Overview • Most forecasts/warnings are generated manually in the Bureau’s 7 Regional Offices, but all depend on centralized guidance from NMOC. • NMOC is the 8th 24x7 office and is responsible for: • Running of the NWP suite • ITC Help Desk • Aviation forecasts • Tsunami Warnings • Implementation & support of NWP systems Second BLUElink> Users Forum

  4. National Meteorological and Oceanographic Centre (NMOC) • 60 staff • Central Operations (24x7): • Southern Hemisphere Analysis • Tsunami Warning Centre • Aviation Desk • Help Desk (IT and Comms) • Operational Development • NWP model implementation & support • Oceanographic Systems • Ocean model implementation and support • Software Systems • Real-time Database and Systems Administration • National Weather Graphics Unit Second BLUElink> Users Forum

  5. Oceanography • Support for Marine Meteorology • Wave models (and surface winds) • Support for Climate • SST analysis • Sub-surface analysis • Seasonal prediction model (POAMA) • BLUElink> Ocean Forecasting • Marine Hazards • Tsunami • Storm surge Second BLUElink> Users Forum

  6. The Operational Forecast System OceanMAPS

  7. Data Flows – Global Models e) OceanMAPS Global SST Analysis O.I Ocean Model POAMA Australian Region MOF GASP Global Assimilation And Prognosis 1DVAR+GenSi+Spectral Manual Intervention (MSLP) f) PME Boundary Conditions To TXLAPS & LAPS UV OCF (Sites Gridded) EER HYSPLIT47 GASP_EPS Ensemble Prediction System Sea State Model WAM • Most other models rely on GASP model output. • Boundary conditions from GASP feed into TXLAPS and LAPS • Model outputs from some systems above get stored in RTDB • and MARS Verification

  8. Scheduling – Supervisor Monitor Scheduler (from ECMWF)

  9. OceanMAPS schedule

  10. In Situ data Drifting Buoys Moored Buoys Ship of Opportunity (XBT) ARGO profiling floats Sea-level gauge data Data Requirements Second BLUElink> Users Forum

  11. Satellite data SST AVHRR Local retrievals Global AATSR Microwave (AMSR-E) Altimetry Scatterometer Data Requirements Second BLUElink> Users Forum

  12. Data Monitoring (Quantity)

  13. Data Monitoring (Coverage) Second BLUElink> Users Forum

  14. Data Monitoring (Timeliness)

  15. Meteorological Forcing • Need to have access to surface fields from the meteorological models used for forcing • GASP • LAPS • Wind Stress • Heat Fluxes • Useful to have integrated DM system Second BLUElink> Users Forum

  16. Meteorological Archive and Retrieval System • Developed and Operated by ECMWF • Suited to handling large data sets and operational model analyses and forecast products • Provides seamless access to real-time and historical data • Allows sub-setting in time and space of large datasets Second BLUElink> Users Forum

  17. Ocean Analysis and Modelling Systems Internal Data Viewer In situ data (ARGO, SOOP, Buoys, Mass storage Database File delivery SST Data (AVHRR, ATSR, AMSR-E,…) MARS Altimeter Data (Jason-1, ENVISAT, GFO) Internet Access NWP Output (GASP, LAPS,…) P1 Data and Product Management

  18. Status of BLUElink> Systems • High resolution SST analysis • Regional (1/12 deg resolution - operational) • Run daily ~ 0100 UTC • Global (1/4 deg resolution – testing) • To be run daily ~ 0400 UTC • OceanMAPS – Ocean Forecast System • Operational 7 day forecast run twice weekly (Monday and Thursday) • One major failure fixed (15/18 Jan) • Some data drop-outs cause problems Second BLUElink> Users Forum

  19. Product Delivery • Internal Users • Diagnostic products • Data monitoring • System performance • Graphical Viewer • Intranet • Tailored products Second BLUElink> Users Forum

  20. Product Delivery • External Users • Basic Products (freely available) • Static Web images • Registered users (cost of access) • File delivery via ftp server • Research Partners • OpenDAP Second BLUElink> Users Forum

  21. Graphical Products Second BLUElink> Users Forum

  22. OceanMAPS Data Products • Best daily ocean in-situ profile observations • Remotely sensed sea surface height (Jason, Envisat, GFO) • Remotely sensed sea surface temperature (GHRSST) • Surface flux products (NWP GASP/LAPS) • Basic model products • sea surface height (SSH) • ocean currents (U,V,W) and states (T,S,Rho) • mixed layer depths • surface momentum, heat, and fresh water fluxes • Model restart files Second BLUElink> Users Forum

  23. Data and Product Storage • Behind real-time analysis • Run once per day • 4GB per day • 32GB per week • 138GB per month • 1680GB per year • 192 hour forecast • Run twice per week • 92GB per week • 368GB per month • 4788GB per year

  24. OceanMAPS file/data formats • File Formats - NetCDF v3 file format (all data products) • Climate and Forecast (CF-1.0) conventions • COARDS conventions • Products • Best daily Observations (Argo netCDF format) • Sea surface temperature products • GHRSST L2P, L3P, and L4 formats (netCDF, CF) • Bureau NWP GASP and LAPS surface fluxes (netCDF, COARDS) • MOM4 model products (netCDF, COARDS, CF-1.0 coming) • Conversion to GRIB/BUFR (WMO Standards)? Second BLUElink> Users Forum

  25. OceanMAPS data/file size • Basic product data array size per time point • 2-D array, single precision is 4.4 MB • 3-D array, single precision is 206 MB • Restart data array size • 2-D array, double precision is 8.8 MB • 3-D array, double precision is 412 MB • Basic data product (daily mean, forcing 3hrly) • U,V,T,S,eta,forcing is ~1129 MB • Basic data product (3 hourly mean) • U,V,T,S,eta,forcing is ~7229 MB Second BLUElink> Users Forum

  26. Data Transport Protocols • FTP - file transfer protocol • Time tested and robust • But…user downloads entire file • HTTP - hyper transport protocol • Works thru most firewalls • Wrapped in nice interface for user • But…user downloads entire file • OPeNDAP - Open Network Data Access Protocol(formerly DODS) • Catalogs of data for users and crawlers • User interface for viewing file metadata • Virtual data sets and aggregation • User downloads selected spatial and temporal indices, as well as variables Second BLUElink> Users Forum

  27. Products Available via ftp(to Registered Users) • Fixed Domains (as for graphics) • NetCDF format (GRIB coming?) • Temperature (1-10 Mbytes/time slice) • Salinity • Zonal Wind • Meridional Wind • Sea Level Anomaly (20-200kBytes) Second BLUElink> Users Forum

  28. Distributed Oceanographic Data System (DODS)or OpenDAP • Conceived in 1993 at a workshop held at URI. • Objectives were: • to facilitate access to PI held data as well as data held in national archives and • to allow the data user to analyze data using the application package with which he or she is the most familiar. • Basic system designed and implemented in 1993-1995 by Gallagher and Flierl with NASA and NOAA funding. • From 1994 to present it has been extended with NASA, NOPP, NSF and NOAA funding. Second BLUElink> Users Forum

  29. Considerations with regard to the development of OPeNDAP • Many data formats • Many different client types Many data providers • Many different semantic representations of the data • Many different security requirements

  30. Our (BoM) History of OPeNDAP • 2004 - First DODS server installed for TPAC digital library to serve near-real time SST and MSLP data sets (~ 20 GB ) • 2005 - Trialed enhanced service to serve MARS data thru DODS (closed due to lack of security) • 2006 - Upgrade to OPeNDAP server 3 and THREDDS server to deliver last 7 days of NWP and ocean forecasts (~ 1.0 TB) • 2007 - Added Tsunami scenario database (~ 5.0 TB) • 2008 - To add second phase of Tsunami scenario database (~55 TB) Second BLUElink> Users Forum

  31. Bureau netCDF CSIRO HDF MARS GRIB Matlab IDL GrADS Ferret IDV Excel VisAD Access ncBrowse OPeNDAP Second BLUElink> Users Forum

  32. OPeNDAP Clients • Network addresses based on URLs • url = 'http://www.opendap.org/opendap/nph-dods/data/fnoc1.nc?u,v’ • url_path = 'http://www.opendap.org/opendap/nph-dods/data/ • MATLAB • Command: “loaddap url” • Structure is loaded into memory with file metadata names • IDL • Command: “stat = OPENDAP_GET(url, data)” • Structure is loaded into memory with file metadata names • NCO • Command: ncks -o ./output -O -C -q -p $url_path fnoc1.nc?u,v • Open Source: www.opendap.org Second BLUElink> Users Forum

  33. LAS ------ Live Access Server TDS ------ THREDDS Data Server WCS ----- Web Coverage Service for GIS MARS ---- Met. Archival Retrieval System SIMS ------ Services Information Mgmt System GODAE Services LAS WCS HTTP TDS OPeNDAP gateway OPeNDAP gateway: User Authentication: yes DataSet Authorization: yes Resource Mgmt: yes SIMS Disk Cache BoM Firewall / Router TDS TDS TDS SQL MARS Global File System SAN Disk Array Disk Array Tape Silo Fast Disk Array Fast Disk Array SATA Disk Array Tape Archive SATA Disk Array Virtualized Storage - data aging GODAE Service Architecture

  34. Conclusion • OceanMAPS has proved to be robust and reliable • Accuracy dependant on input data • In situ • altimetry • Product availability • Fixed Images (freely available) • Fixed NetCDF files (available to Registered Users) • OpenDAP Access (available to research users) Second BLUElink> Users Forum

More Related