1 / 32

HDF OPeNDAP Project Update

HDF OPeNDAP Project Update. MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009. Outline. HDF5 – OPeNDAP work review HDF4 – OPeNDAP handler progress report Future work plan. Access data via OPeNDAP. Users. Hyrax. IDV. libnc -dap. HDF HANDLER. AIRS. DAP. Visualization Tools.

burke
Download Presentation

HDF OPeNDAP Project Update

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 Annual briefing to ESDIS

  2. Outline • HDF5 – OPeNDAP work review • HDF4 – OPeNDAP handler progress report • Future work plan Annual briefing to ESDIS

  3. Access data via OPeNDAP Users Hyrax IDV libnc-dap HDF HANDLER AIRS DAP Visualization Tools OPeNDAP Clients OPeNDAP Servers HDF4 Files Annual briefing to ESDIS

  4. HDF5 – OPeNDAP Work Annual briefing to ESDIS

  5. HDF5-OPeNDAP handler • Update with new Hyrax • Help GES DISC serve TES and OMI data • Finish HDF5 – DAP2 mapping document and submit it as a technical note to NASA SPG Annual briefing to ESDIS

  6. HDF5-Friendly OPeNDAP Client Library • Retrieve the key HDF5 information • Easy to access other HDF5 data (e.g. EOS5 swath) Annual briefing to ESDIS

  7. Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it. Example: Groups in HDF5 HDF5-Friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5 Annual briefing to ESDIS

  8. MLS swath via GrADS CO level near Beijing, China before, during and after Olympics lower stratosphere Before COat these elevations is a measure of pollution During After upper stratosphere Annual briefing to ESDIS

  9. HDF4 – OPeNDAP Work Annual briefing to ESDIS

  10. Current HDF4-OPeNDAP handler • Contributed by JPL in 1998 • Part of the OPeNDAP distribution • Visualization client tools cannot display most HDF-EOS2 and HDF4 data Annual briefing to ESDIS

  11. Our work • Enhance the HDF4-OPeNDAP handler • More NASA HDF-EOS2 and HDF4 data can be accessed via visualization clients Annual briefing to ESDIS

  12. Challenges – more products Hundreds of HDF-EOS2 and HDF4 Data products Annual briefing to ESDIS

  13. Challenges – Requirement from Clients • CF conventions • - Clients expect a certain data structure • - Clients expect some key attributes Annual briefing to ESDIS

  14. Challenges – Grid • Various HDF-EOS2 Grid projections • Geographic • Sinusoidal • Polar stereographic • … Data [4][12] Lon[12] Data [4][8] Lon[4][8] Geographic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sinusoidal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Annual briefing to ESDIS

  15. Challenges - Swath Longitude field has two columns Data field has three columns Longitude should have three columns • HDF-EOS2 Swath - Dimension map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Annual briefing to ESDIS

  16. Challenges – various products • “Special” HDF-EOS2 files • Hybrid HDF-EOS2 files • Some HDF4 objects that cannot be read by EOS library • HDF4 data products Annual briefing to ESDIS

  17. Enhanced HDF4 Handler • Follow CF conventions • Generate the correct DAP data type. • Add shared geo-location information. • Insert additional key attributes. • How: Parse Metadata or Use EOS2 library Annual briefing to ESDIS

  18. Implementation choice • Use Metadata parser or HDF-EOS2 library • Metadata parser • Pro: simple • Con: limited products • HDF-EOS2 library • Pro: maximum products • Con: install HDF-EOS2 library first Annual briefing to ESDIS

  19. Solution Comparison Annual briefing to ESDIS

  20. Quick Implementation • Parse Metadata – hdf5_handler • Use HDF-EOS2 library – h4toh5 IDL ncBrowse Ferret MATLAB NCL IDV GrADS ODC Annual briefing to ESDIS

  21. Current Status • MODIS / AIRS Grid and Swath • Other Projections (Polar, Sinusoidal) • Testing IDL ncBrowse Ferret MATLAB NCL IDV GrADS ODC Annual briefing to ESDIS

  22. HDF-EOS2 in CF via DAP AIRS Grid AIRS.2008.10.27.L3.RetStd001.v5.2.2.0.G08303124144.hdf http://daac.gsfc.nasa.gov/daac-bin/whom/mk_page_cgi.pl?PATH=datapool/AIRS/Level3_V005/01_AIRX3STD/2008/10 AIRS Swath ftp://airspar1u.ecs.nasa.gov/data/s4pa//Aqua_AIRS_Level2/AIRX2RET.005//2008/330/AIRS.2008.11.25.143.L2.RetStd.v5.2.2.0.G08331121849.hdf MODIS Grid MOD08_D3.A2008284.005.2008289105354.hdf http://modis-atmos.gsfc.nasa.gov/MOD08_D3/acquiring.html MODIS Swath MOD06_L2.A2001222.0025.004.2003127111841.hdf http://modis-atmos.gsfc.nasa.gov/MOD06_L2/acquiring.html Annual briefing to ESDIS

  23. More challenges: Missing Attributes • MODIS Grid – Rain fall • AURA OMI Grid - Ozone No Unit! Annual briefing to ESDIS

  24. More challenges Match dimension names • Horizontal geolocation information: - Non-standard names ( e.g. nlon, nlat) How to match? • Vertical geolocation information: - Users provide the geolocation name and vaule but no conventions are enforced. How to know? Annual briefing to ESDIS

  25. More challenges – Missing Values • In a swath file, a data field has three dimensions. But no dimension value for the third dimension. MISR_AM1_TC_CLASSIFIERS_P020_O000995_F06_0011.hdf Annual briefing to ESDIS

  26. A Solution to challenges • Provide an option for the handler to generate fully CF-compliant output • Create a separate XML input file for the handler and feed it into the handler • Insert attributes, renaming rules, and missing values by putting them in the XML input • This requires a knowledge about the characteristics of data in advance Annual briefing to ESDIS

  27. One Solution An input file from data provider • Example: Attributes, Renaming, Values <xml> <variable name=‘RrLandRain’> <attr name=‘unit’ type=‘String’> inches</attr> </variable> <rename from=‘pressStd’ to=‘level’></rename> <insert name=‘NAltitudeDim’ size=10 type=‘Int8’> 0,1,2,3,4,5,6,7,8,9 </insert> </xml> Annual briefing to ESDIS

  28. Future Work (Short Term) • Expand test-suite with data pool • Release through OPeNDAP website • Support for Swath via Parsing • Support for Point data type via HDF- EOS2 library Annual briefing to ESDIS

  29. Future Work (Long Term) • General HDF4 Files Support • Using an input XML file to provide full • CF support • HDF-EOS2 to DAP2 Mapping Document Annual briefing to ESDIS

  30. Thank You! Workshop Name (edit footer to change)

  31. Acknowledgements This work was supported by the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A, NNX08A077A and NNX06AG75A. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA. Workshop Name (edit footer to change)

  32. Questions/comments? Workshop Name (edit footer to change)

More Related