1 / 14

Challenges to Archive and Access NASA HDF-EOS Data in the long Term

Challenges to Archive and Access NASA HDF-EOS Data in the long Term. MuQun Yang (The HDF Group) Choonghwan Lee (The HDF Group) Ruth Duerr (NSIDC ). Prerequisite. METS(Metadata Encoding & Transmission Standard) Standard for encoding structural metadata ISO-19115

jaime-bryant
Download Presentation

Challenges to Archive and Access NASA HDF-EOS Data in the long Term

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Challenges to Archive and Access NASA HDF-EOS Data in the long Term MuQun Yang (The HDF Group) Choonghwan Lee (The HDF Group) Ruth Duerr (NSIDC ) 25th IIPS Conference

  2. Prerequisite • METS(Metadata Encoding & Transmission Standard) • Standard for encoding structural metadata • ISO-19115 • International Schema for describing geographic information • File-level Metadata • Metadata about the individual file or granule • Dataset-level Metadata • Metadata that applies to each and every granule/file in the whole data set(product) 25th IIPS Conference

  3. HDF5 Archive Information Package Data file HDF5 METS Metadata file Primary SchemaExtension Schema |<mets> |---<dmdSec>----------------<MODS> |---<amdSec>--------------|--<techMD> | |--<rightsMD> | |--<sourceMD> |----<fileGrp> |----<structMap> HDF5 AIP Components http://www.hdfgroup.uiuc.edu/papers/papers/AIP/HDF5_AIP_White_Paper.pdf 25th IIPS Conference

  4. NOAA SDS Program CDM/NetCDF4 ISO-19115 NCDC:CLASS ECS to ISO-19115 HDF5-AIP NSIDC/ECSMetadata NetCDF4 / HDF5 Data METS ECS to METS NSIDC/ ECS HDF4-data H4toH5 NetCDF4/HDF5-data 25th IIPS Conference 4

  5. Enhanced H4toH5 conversion tool • Convert HDF-EOS2 data to NetCDF4-compliant HDF5 data • Official release (2.0) can be found at http://hdfgroup.org/h4toh5/ $ ./h4toh5 –eos –nc4 input.he2 output.nc4 25th IIPS Conference

  6. Challenges to do the conversion • Retrieve geo-location information from HDF-EOS2 data • Conform to NetCDF4 data model in the existing H4toH5 conversion tool • …… 25th IIPS Conference

  7. Challenges: Handle EOS - Grid • Grid lacks geolocation fields • Use predefined projections • Geographic • Sinusoidal • Polar stereographic • … • New converter creates geolocation fields • HDF-EOS2 API GDij2ll() Data [4][12] Lon[12] Data [4][8] Lon[4][8] Geographic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sinusoidal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25th IIPS Conference

  8. Challenges: Handle EOS - Swath • The size of geolocation fields can be different from data fields • New converter has to handle geolocation fields correctly . . . . . . . . . . . . . . . . . . . . . . . . . . . 25th IIPS Conference

  9. Challenges in conforming to NetCDF4 Longitude field has two columns Data field has three columns New longitude has three columns • Follow CF conventions • Create two variables: NewLongitude and NewLatitude • Add to the data field an attribute coordinates=“NewLongitudeNewLatitude” • Keep the original Latitude and Longitude . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25th IIPS Conference

  10. Now some examples to show NetCDF4 files converted from EOS2 25th IIPS Conference

  11. A netCDF-4 file converted from EOS2 data at NSIDC 25th IIPS Conference

  12. A netCDF-4 file converted from EOS2 data at NSIDC 25th IIPS Conference

  13. Deliverables and future work • Deliverables • Enhanced HDF4 to HDF5 conversion tool http://hdfgroup.org/h4toh5/ • A validation tool to verify the correctness of the conversion Will be released soon! 25th IIPS Conference

  14. Acknowledgement • This work was supported under NOAA Scientific Stewardship Program grant number NA07OAR4310286. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NOAA.  25th IIPS Conference

More Related