1 / 34

Improving the usability of HDF-EOS2 data

Improving the usability of HDF-EOS2 data. Kent Yang, Joe Lee, Choonghwan Lee The HDF Group March 31 st , 2009. EOSDIS Evolution 2015 Vision Tenets. Feb 3, 2005. Purpose of the talk. Identify problems to access HDF-EOS2 data Share our plan to help improve the EOS2 usability.

rschafer
Download Presentation

Improving the usability of HDF-EOS2 data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Improving the usability of HDF-EOS2 data Kent Yang, Joe Lee, Choonghwan Lee The HDF Group March 31st, 2009 Annual briefing to ESDIS

  2. EOSDIS Evolution 2015 Vision Tenets Feb 3, 2005 Annual briefing to ESDIS

  3. Purpose of the talk • Identify problems to access HDF-EOS2 data • Share our plan to help improve the EOS2 usability Annual briefing to ESDIS

  4. The success of EOS mission MODIS HDF-EOS data to detect sand storm from Gobi china Courtesy NASA Earth Observatory Annual briefing to ESDIS

  5. The success of EOS mission MODIS HDF-EOS data to detect fires in Africa Courtesy NASA Annual briefing to ESDIS

  6. Identify problems : technical part Annual briefing to ESDIS

  7. Beauty of HDF-EOS2 • Very close to physical model • Convenient to handle geo-location projections • Files are generally compact • No needs to store latitude and longitude for grid • Reduce the size of latitude and longitude for swath Annual briefing to ESDIS

  8. A compact HDF-EOS2 Grid • MODIS (MYD10A2) Grid file • Two 2400 X 2400 grid data fields with gzip compression • File size: 75 KB • If lat/lon are kept in the file • File size may become 44 MB, • 500 times bigger than the original file size Annual briefing to ESDIS October 15-18, 2008 Workshop Name (edit footer to change) 8

  9. A compact HDF-EOS2 swath coarse_cross with dimension maps • Small file size – Swath coarse_along along along cross cross CloudCover(along, cross) Latitude(along, cross) Longitude(along, cross) CloudCover(along, cross) Latitude(coarse_along, coarse_cross) Longitude(coarse_along, coarse_cross) Annual briefing to ESDIS

  10. There are issues to access some HDF-EOS2 files Aims to improve the usability Annual briefing to ESDIS

  11. First of all, there should be more communications among providers, tool developers, users and data storage software developers. Focus on HDF-EOS2 and HDF4 in the next few slides. Annual briefing to ESDIS

  12. Schematic of Grid XDim YDim StructMetadata.0 = “… XDim= 14 YDim = 8 UpperLeftPointMtrs =(-156.0,71.0)LowerRightMtrs =(180.0,-75.0) … Projection = HE5_GCTP_GEO …” HDF-EOS5 GRIDS: Float32 Ozone(YDim, XDim) Annual briefing to ESDIS

  13. Issue 1: lack of strict checking in HDF-EOS2 library Annual briefing to ESDIS

  14. Lack of strict checking in HDF-EOS2 library • XDim and YDim exist and have values • .e.g, OMI (OMUVBL3) • Generally, XDim and YDim do not exist because they are implied. • XDim and YDim do not need to have values because they can be calculated. Values are just redundant. Annual briefing to ESDIS

  15. Lack of strict checking in HDF-EOS2 library StructMetadata … Band_1KM_RefSB[4] … Data Field Band_1KM_RefSB 4 elements Data Field Band_1KM_RefSB 15 elements • Inconsistency in the number of elements HDF-EOS2 informs and HDF-EOS2 actually reads • e.g., MODIS (MAC021S0, MAC02QS0, MAC02QS1, MAC021S1) • We believe that this data field had four elements once and it was appended later. consistent INCONSISTENT! Annual briefing to ESDIS

  16. Lack of strict checking in HDF-EOS2 library • Wrong grid parameters • e.g., MISR (MI3DAER) • Projection is defined by code, projection parameters, etc. • For Lambert Azimuth projection, the fifth parameter specifies “longitude of center of projection”. • Its value is entered in DDDMMMSSS.SS format. • -92970000.00 minutes 970? degrees -92 seconds 0 Annual briefing to ESDIS

  17. Issue 2: lack of standard way to handle some geolocation fields Annual briefing to ESDIS

  18. Schematic of Swath StructMetadata.0 = “ … DimensionName=”along” size= 11 DimensionName=”cross” size= 3 …” Data fields: Float32 CloudCover(along,cross) Geolocation fields: Float32 lat(along,cross) Float32 lon(along,cross) along cross Annual briefing to ESDIS

  19. No standard way to handle geolocation fields • No standard way to figure out the vertical coordinate in an HDF-EOS2 file • A data field • TAirStd[GeoTrack][GeoXTrack] • StdPressureLev is a dimension • Additional data field supplies values, probably • pressStd • How can a user know for sure that the field pressStd provides the vertical coordinate information? [StdPressureLev] [StdPressureLev] [StdPressureLev] Annual briefing to ESDIS

  20. Issue 3: No enforcement for the use of standard names Annual briefing to ESDIS

  21. Schematic of Grid XDim YDim StructMetadata.0 = “… XDim= 14 YDim = 8 UpperLeftPointMtrs =(-156.0,71.0)LowerRightMtrs =(180.0,-75.0) … Projection = HE5_GCTP_GEO …” HDF-EOS5 GRIDS: Float32 Ozone(YDim, XDim) Annual briefing to ESDIS

  22. No enforcement for the use of standard names • Use of non-standard dimensions in grid • e.g., MOPITT (MOP03) • To be interpreted properly, each data field must make use of the two predefined dimensions: XDim and YDim • However, none of data fields in that file refers to XDim or YDim. Instead, they refer to nlon and nlat, which are not standard way. Annual briefing to ESDIS

  23. No enforcement for the use of standard names • Object names are not restrictive • e.g., MODIS (MAC03S0) • Name of a data field called • Land/SeaMask • A dimension called • nscans*10 • These names may not be safe in other APIs. Especially, HDF5 does not allow slash(/). Annual briefing to ESDIS

  24. Other bad practices • Extreme long attributes • Chunk size much bigger than dimensional size for fixed size SDS • These are not enforced by HDF4 library Annual briefing to ESDIS

  25. Other technical difficulties • Hybrid HDF-EOS2 files • Some data cannot be accessed by HDF-EOS2 library • Need to help users access these files Annual briefing to ESDIS

  26. Identify problems : service part Annual briefing to ESDIS

  27. A message from DAAC User Survey I am a student and I need to read MODIS HDF-EOS data. But it is hard for me to extract the data. I need a fortran program to extract the data. I wish DAAC can provide this. Annual briefing to ESDIS

  28. DAAC tool sites • Provide most information users need Annual briefing to ESDIS

  29. What’s left? • Some tools are out of date • On some sites, no C/fortran examples to access the data • Information on the tool page out of date Annual briefing to ESDIS

  30. What’s the problem? • A small percentage of EOS2 users needs help. • Access “difficult” files • Compensate tools/examples for DAACs Annual briefing to ESDIS

  31. The HDF Group’s plan to DAACs • Compensate fortran/C example code for DAACs • C/fortran examples for hybrid files and “difficult” files • Evaluate/update tools listed at hdfeos.org • Evaluate tools at DAACs • EOS-forum • Web forum? • EOS2 user and developer mailing lists? Annual briefing to ESDIS

  32. Thank You! Annual briefing to ESDIS

  33. Acknowledgements This work was supported by the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX08A077A. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA. Annual briefing to ESDIS

  34. Questions/comments? Annual briefing to ESDIS

More Related