1 / 10

An HDF5-WRF module -A performance report

An HDF5-WRF module -A performance report. MuQun Yang, Robert E. McGrath, Mike Folk National Center for Supercomputing Applications University of Illinois, Urbana-Champaign ymuqun@ncsa.uiuc.edu URL : http://hdf.ncsa.uiuc.edu/apps/WRF-ROMS/. The uniqueness about the study.

luka
Download Presentation

An HDF5-WRF module -A performance report

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An HDF5-WRF module-A performance report MuQun Yang, Robert E. McGrath, Mike Folk National Center for Supercomputing ApplicationsUniversity of Illinois, Urbana-Champaign ymuqun@ncsa.uiuc.edu URL: http://hdf.ncsa.uiuc.edu/apps/WRF-ROMS/

  2. The uniqueness about the study • The HDF5 is not used to store satellite data, but the output from a sophisticated numerical weather model • Explore the performance of parallel HDF5 in parallel computing environments • Investigate the performance of the compression feature inside HDF5 when applying to the numerical model

  3. From WRF tutorial

  4. SchematicHDF5 File Structure of Sequential HDF5-WRF Output / Dim_group WRF_DATA TKE RHO_U Time ETC… HDF5 dataset(WRF field) HDF5 group(WRF dataset) Solid line: HDF5 datasets or sub-groups (the arrow points to) that are members of the HDF5 parent group. Dash line: The association of one HDF5 object to another HDF5 object through dimensional scale table.

  5. Schematic HDF5 File Structure of Parallel HDF5-WRF Output / WRF_DATA Time_stepN Dim_group Time_step0 Time_step1 attr1 Time attr2 RHO_U TE RHO_U TE … HDF5 dataset(WRF field) HDF5 group(WRF dataset) Solid line: HDF5 datasets or sub-groups (the arrow points to) that are members of the HDF5 parent group. Dash line: the association of one HDF5 object to another HDF5 object through dimensional table.

  6. Summary • Parallel IO is not trivial • Effective chunking with MPI-IO inside HDF5 library is the key to improve parallel HDF5 performance • Szip compression can improve performance for WRF application • Shuffling algorithm with gzip compression can further improve compression ratio

More Related