1 / 15

Storing EOS data using HDF5 Archival Information Package

Storing EOS data using HDF5 Archival Information Package. MuQun Yang (The HDF Group) Choonghwan Lee (The HDF Group) Ruth Duerr (NSIDC ). Definitions of some terms. CDM(Common Data Model) Unification of the data models of OpenDAP , netCDF and HDF5 netCDF-4(network Common Data Form 4)

ellie
Download Presentation

Storing EOS data using HDF5 Archival Information Package

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Storing EOS data using HDF5 Archival Information Package MuQun Yang (The HDF Group) Choonghwan Lee (The HDF Group) Ruth Duerr (NSIDC ) HDF and HDF-EOS Workshop XII, Aurora, CO

  2. Definitions of some terms • CDM(Common Data Model) • Unification of the data models of OpenDAP, netCDF and HDF5 • netCDF-4(network Common Data Form 4) • Library manipulating array-oriented scientific data • CF-Conventions • Convention of metadata providing spatial properties, among climate research community… • CLASS (Comprehensive Large Array-data Stewardship System) • Electronic library of NOAA environmental data HDF and HDF-EOS Workshop XII, Aurora, CO

  3. Prerequisite • METS(Metadata Encoding & Transmission Standard) • Standard for encoding structural metadata digital library • ISO-19115 • Schema for describing geographic information(former FGDC) • File-level Metadata • Metadata about the individual file or granule • Dataset-level Metadata • Metadata that applies to each and every granule/file in the whole data set(product) HDF and HDF-EOS Workshop XII, Aurora, CO

  4. HDF5 Archive Information Package Data file HDF5 METS Metadata file Primary SchemaExtension Schema |<mets> |---<dmdSec>----------------<MODS> |---<amdSec>--------------|--<techMD> | |--<rightsMD> | |--<sourceMD> |----<fileGrp> |----<structMap> HDF5 AIP Components http://www.hdfgroup.uiuc.edu/papers/papers/AIP/HDF5_AIP_White_Paper.pdf HDF and HDF-EOS Workshop XII, Aurora, CO

  5. NOAA SDS Program CDM/NetCDF4 ISO-19115 NCDC:CLASS ECS to ISO-19115 HDF5-AIP NSIDC/ECSMetadata NetCDF4 / HDF5 Data METS ECS to METS NSIDC/ ECS HDF4-data H4toH5 NetCDF4/HDF5-data HDF and HDF-EOS Workshop XII, Aurora, CO 5

  6. Enhanced H4toH5 conversion tool • Convert HDF-EOS2 data to NetCDF4-compliant HDF5 data • Official release (2.0) can be found at http://hdfgroup.org/h4toh5/ $ ./h4toh5 –eos –nc4 input.he2 output.nc4 HDF and HDF-EOS Workshop XII, Aurora, CO

  7. Challenges to do the conversion • Retrieve geo-location information from HDF-EOS2 data • Conform to NetCDF4 data model in the existing H4toH5 conversion tool • …… HDF and HDF-EOS Workshop XII, Aurora, CO

  8. Challenges: Handle EOS - Grid • Grid lacks geolocation fields • Use predefined projections • Geographic • Sinusoidal • Polar stereographic • … • New converter creates geolocation fields • HDF-EOS2 API GDij2ll() Data [4][12] Lon[12] Data [4][8] Lon[4][8] Geographic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sinusoidal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . HDF and HDF-EOS Workshop XII, Aurora, CO

  9. Challenges: Handle EOS - Swath • The size of geolocation fields can be different from data fields • New converter has to handle geolocation fields correctly . . . . . . . . . . . . . . . . . . . . . . . . . . . HDF and HDF-EOS Workshop XII, Aurora, CO

  10. Challenges in conforming to NetCDF4 Longitude field has two columns Data field has three columns New longitude has three columns • Follow CF conventions • Create two variables: NewLongitude and NewLatitude • Add to the data field an attribute coordinates=“NewLongitudeNewLatitude” • Keep the original Latitude and Longitude . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . HDF and HDF-EOS Workshop XII, Aurora, CO

  11. Now some examples to show NetCDF4 files converted from EOS2 HDF and HDF-EOS Workshop XII, Aurora, CO

  12. A netCDF-4 file converted from EOS2 data at NSIDC HDF and HDF-EOS Workshop XII, Aurora, CO

  13. A netCDF-4 file converted from EOS2 data at NSIDC HDF and HDF-EOS Workshop XII, Aurora, CO

  14. Deliverables and future work • Deliverables • Enhanced HDF4 to HDF5 conversion tool http://hdfgroup.org/h4toh5/ • A validation tool to verify the correctness of the conversion Will be released soon! • Potential future work 1. Reduce the output file size 2. Improve the performance HDF and HDF-EOS Workshop XII, Aurora, CO

  15. Acknowledgement • This work was supported under NOAA Scientific Stewardship Program grant number NA07OAR4310286. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NOAA.  HDF and HDF-EOS Workshop XII, Aurora, CO

More Related