1 / 29

HDF5 OPeNDAP Project Update and Demo

MuQun Yang and Hyo -Kyung Lee (The HDF Group) James Gallagher ( OPeNDAP , Inc.). HDF5 OPeNDAP Project Update and Demo. OPeNDAP Review. A software framework that allows simple access to remote data Data Access Protocol Client-server model Widely used by Earth Science Community.

vance
Download Presentation

HDF5 OPeNDAP Project Update and Demo

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) HDF5 OPeNDAP ProjectUpdate and Demo

  2. OPeNDAP Review • A software framework that allows simple access to remote data • Data Access Protocol • Client-server model • Widely used by Earth Science Community

  3. Example Usage HDF4 OPeNDAP Client OPeNDAP HDF4 Server Actual Content of Data Semantic Meaning of Data Syntactic Structure of Data OPeNDAP Visualization Client

  4. Why Important? • HDF5: NASA EOS / NPOESS • NASA’s MISSION: ACCESS! • Our MISSION: Build a DAP-HDF5 Bridge

  5. Goals • 1st: Access HDF5 data via OPeNDAP • 2nd: Construct HDF5 data from OPeNDAP

  6. OPeNDAP HDF5 Server Goal: Access HDF5 data via OPeNDAP • Map HDF5 Datatypesinto DAP Datatypes • Plus, tweaks for OPeNDAP Clients • Plus, tweaks for HDF-EOS Files

  7. Mapping HDF5 Datatypes • Prototype server in 2001 • NASA Grant from Nov. 2006 • Support for Compound Datatype • Support for Group • Support for References / Links • First Product Release in Mar. 2008

  8. OPeNDAP HDF5 Server Goal: Access HDF5 data via OPeNDAP • Map HDF5 Datatypes in DAP Datatypes • Plus, tweaks for OPeNDAP Clients • Plus, tweaks for HDF-EOS Files

  9. OPeNDAP Clients Not all OPeNDAP clients are created equal! • I hate long variable names. • I need special attributes on dataset. • I care only Grid datatypes. • I care only well-formed attributes.

  10. Tweaks for OPeNDAP Clients • Two configuration options • --enable-short-name • --enable-CF

  11. OPeNDAP HDF5 Server Goal: Transform HDF5 access in DAP-way • Map HDF5 Datatypes in DAP-way • Plus, tweaks for OPeNDAP Clients • Plus, tweaks for HDF-EOS5 Files

  12. HDF EOS Not all HDF5 files are created for DAP! • Grid with No Dimension data • Clients expect Grid with Dimension data • Some Attributes stored in Extremely Long String(s) (e.g. StructMetada.0) • Clients expect well-structured attributes

  13. Tweaks for HDF-EOS • Two more configuration options • --enable-eos-grid • --enable-eos-meta

  14. Day After Server Tweaks • Finally, Happy Clients! IDL ncBrowse Ferret MATLAB NCL IDV GrADS ODC

  15. Goals • 1st : Access HDF5 via OPeNDAP • 2nd: Construct HDF5 data from OPeNDAP • Bonus: Help clients to better access EOS Swath data

  16. HDF5 Served in OPeNDAP • There are some things money can’t buy. For everything else, there is MasterCardTM. Minimize • There are some things DAP can’t serve. For everything else, there is _________? Maximize

  17. Some Things (that OPeNDAP HDF5 server can’t serve) • Hard: Opaque, Bitmap, Enum, 64 bit Integer, Variable Length types • Illegal: Reserved characters in DAP are used in Dataset/Group name in HDF5

  18. For Everything Else • Let there be HDF5-Friendly OPeNDAP client library! • Package and Deliver in DAP first. • Then, let the client library handle it. • Concept of Trojan Horse

  19. Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it. Example: Group in HDF5 HDF5-Friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5

  20. Example: Reference in HDF5 • Object / Regional Reference • Map to DAP URL at server • No de-referencing of URL at client library • Important for NPOESS

  21. HDF5-Friendly OPeNDAP Client Library • One more reason: Help Clients to view Swath • Only ODC can display Swath properly. No Latitude and Longitude • Easy but RigidOPeNDAP NC Client Library! • easy: same as netCDF C interface • rigid: follows netCDF model • Our client library must be Easy but Flexible Courtesy of NASA

  22. NC-Friendly OPeNDAP Client Library HDF5-Friendly OPeNDAP Client Library ??? View NetCDF View G/R/Swath Group/Ref/Swath NetCDF View HDF5 HDF5 GrADS GrADS OPeNDAP HDF5 Server OPeNDAP NC Server liboc-dap libnc-dap dapserver libdap

  23. Example: dap2h5 • A test application for the client library • It can construct HDF5 from DAP output ORIGINAL CONSTRUCTED

  24. Demo: GrADS • GrADS is modified to use our library • The client library provides grid mapping from swath data • GrADS displays swath through HDF-EOS5 specific client library API calls

  25. Caution • Our Client Library is ONLY PROTOTYPE! • It does NOT support all DAP data types • It does NOT support all AURA files • It does NOT support all Visualization clients

  26. Summary • HDF5 Access via OPeNDAP • Is Easy / Efficient / Cool • Loses some things • Requires HDF5-friendly OPeNDAP Client Library • HDF5-Friendly OPeNDAP Client Library Prototype • Serves HDF5 better • Benefits visualization clients

  27. Future Work • HDF5 to DAP2 Mapping Document • Release HDF5-friendly OPeNDAP Client Library Prototype • URL: http://hdfgroup.org/projects/opendap/index.html

  28. Credits Mike Folk (THG) Robert McGrath (NCSA) Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET) Christopher Lynnes, James Johnson, Denis Nadeau, Fan Fang (NASA) Jennifer Adams (GrADS) Dave Brown (UCAR)

  29. Acknowledgement • This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX06AG75A..  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA. 

More Related