1 / 11

Output of UM Diagnostics Directly in CF NetCDF; an end to costly data conversions

NCAS C omputational M odelling S ervice (CMS) Group providing services to the UK academic modelling community. Output of UM Diagnostics Directly in CF NetCDF; an end to costly data conversions. Jeff Cole, Grenville Lister, NCAS-CMS, Dept. of Meteorology, University of Reading.

williamsapril
Download Presentation

Output of UM Diagnostics Directly in CF NetCDF; an end to costly data conversions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community Output of UM Diagnostics Directly in CF NetCDF; an end to costly data conversions Jeff Cole, Grenville Lister, NCAS-CMS, Dept. of Meteorology, University of Reading

  2. NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community • Typical workflow • Run the model to create 64-bit fields-file output • Convert fields files to 32-bit pp (delete fields files) • Convert pp data to CF NetCDF (all of it, ad hoc, delete pp files?) • Analyze, visualize, process NetCDF data • Problems? • Low volumes of data - probably not • Large volumes of data – greater potential for inefficiency

  3. NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community • Cascade • 80TB of pp data to NetCDF • conversion rate to NetCDF 10hr/TB on the HECToR NERC post processor • 800hrs converting pp-NetCDF (doesn’t include fields-file – pp conversion) • Model data production 10 wall clock hrs/TB • Leeds experience • Faster conversions - 4hr/TB • “As far as I'm concerned the main issue is the amount of extra disk space the conversion takes up. It essentially doubles what is needed because I am never 100% sure the conversion has worked properly so I never delete the original pp files.”

  4. NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community Objectives • Directly write UM STASH output in NetCDF format • Implement the whole UMUI STASH panel • Convert PP file metadata into CF metadata • Support standard and mean PP files • Support NetCDF versions 3 and 4, including data compression • Support multiple UM versions

  5. NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community Coding decisions • All NetCDF library calls contained in one f90 module • Minimize changes to existing UM code • All model metadata is written when NetCDF file is first opened • Use STASH domain and time profile names as vertical and time coordinate names • Standard names and units are taken from file STASH_to_CF.txt, produced by David Hassell

  6. NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community Writing metadata Subroutine ncfile_init(called from initial) Loop1: loop over Fortran units if PP file is to be output on this unit Open NetCDF file Write global attributes Loop2: loop over all possible STASH codes if this STASH code is to be written to a PP file Write horizontal dimension information (ncfile_write_horiz_dim) Write vertical dimension information (ncfile_write_vert_dim) Write time dimension information (ncfile_write_time_dim) Create NetCDF variable Save NetCDF variable id, to be used when writing data Write NetCDF variable attributes

  7. NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community Writing metadata Loop3: loop over Fortran units if PP file is to be output on this unit Put NetCDF file into data mode Write horizontal dimension data (ncfile_write_horiz_var) Write vertical dimension data (ncfile_write_vert_var) Write time dimension data (ncfile_write_time_var)

  8. NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community netcdf_mod ncfile_init ncfile_write_data stwork initial Atmosphere

  9. NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community NetCDF timings HadGEM3A, vn8.2, N96L85, 3 day run, 118 diagnostics, 9 PP files

  10. NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community

  11. NCAS Computational Modelling Service (CMS) Groupproviding services to the UK academic modelling community To Do • Finish implementing CF attributes (bounds, cell_methods) • More work needed on time dimension • Complete STASH panel options (meaning, grid subsection, timeseries) • Add support for mean PP files • Put in code so PP output works when files are re-initialised and for CRUNs • Add support for writing NetCDF data in I/O Server code • Add UMUI panel to select NetCDF options and code to read NetCDF namelist variables • Add support for NetCDF4 data compression • Port code to other UM versions, including adding support for the old ocean model • Comprehensive testing

More Related