1 / 26

Enhancing OPeNDAP with HDF5 for Remote Data Access

A project update and demo showcasing how integrating HDF5 with OPeNDAP simplifies remote data access in Earth Science. Challenges, tweaks, and client enhancements explored. Prototype client library demonstrated.

frew
Download Presentation

Enhancing OPeNDAP with HDF5 for Remote Data Access

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HDF5 OPeNDAP ProjectUpdate and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) HDF and HDF-EOS Workshop XII

  2. OPeNDAP Review • A software framework that allows • simple access to remote data • Data Access Protocol • Client-server model • Widely used by Earth Science • Community HDF and HDF-EOS Workshop XII

  3. OPeNDAP Concept Typical Remote Access View Data (User) Remote Data (HDF4) FTP/HTTP Visualization Tools (gradsdap) Visualization tools (e.g. grads) Handler (hdf4_handler) Client Library (libnc-dap) Server (apache) DAP Protocol (via http) HDF and HDF-EOS Workshop XII

  4. OPeNDAP HDF5 Handler View Data (User) Remote HDF5 HDF5 Handler Server (apache) DAP Protocol (via http) HDF and HDF-EOS Workshop XII

  5. Challenges Mapping HDF5 to DAP • Compound Datatype • Groups • Object/Regional References HDF and HDF-EOS Workshop XII

  6. OPeNDAP HDF5 Handler with HDF-EOS5 View Data (User) Remote HDF5 Remote HDF-EOS5 Visualization tools (grads) HDF5 Handler Client Library (libnc-dap) Server (apache) DAP Protocol (via http) HDF and HDF-EOS Workshop XII

  7. Challenges: HDF- EOS5 • Grid with No Geolocation data • Clients expect Grid with Geolocation data • Some Attributes stored as Extremely Long • String(s) (e.g. StructMetada.0) • Clients expect structured attributes HDF and HDF-EOS Workshop XII

  8. Tweaks for HDF-EOS5 • Added two default HDF5 handler configuration options: • --enable-eos-grid • --enable-eos-meta HDF and HDF-EOS Workshop XII

  9. Challenges: OPeNDAP Clients • Need special attributes on dataset. • Need shared geolocation variables. HDF and HDF-EOS Workshop XII

  10. Tweaks for OPeNDAP Clients • Added two optional handler configuration options: • --enable-short-name • --enable-CF HDF and HDF-EOS Workshop XII

  11. OPeNDAP HDF5 Handler with HDF-EOS5 View Data (User) Remote HDF5 Remote HDF-EOS5 Visualization tools (grads) HDF5 Handler HDF5 Handler w/ CF options Client Library (libnc-dap) Server (apache) DAP Protocol (via http) HDF and HDF-EOS Workshop XII

  12. Day After Server Tweaks • Finally, Happy Clients! IDL ncBrowse Ferret MATLAB NCL IDV GrADS ODC HDF and HDF-EOS Workshop XII

  13. Problems of Tweaking HDF5 Handler Remember that we added two optionalHDF5 handler configuration options: --enable-short-name --enable-CF Causes ambiguity among variable names (e.g. /GroupA/ozone vs. /GroupB/ozone) Drops some key attributes (e.g. StructMetdata, HDF_ROOT_GROUP) HDF and HDF-EOS Workshop XII

  14. HDF5-Friendly OPeNDAP Client Library View Groups ??? HDF5 Groups View Data (User) Remote HDF5 Visualization tools (grads) HDF5 Handler HDF5 Handler w/ CF options Client Library (libnc-dap) HDF5-Friendly Client Library (liboc-dap) Server (apache) DAP Protocol (via http) HDF and HDF-EOS Workshop XII

  15. Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it. Example: Groups in HDF5 HDF5-Friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5 HDF and HDF-EOS Workshop XII

  16. Example: Reference in HDF5 • Important for NPOESS • Object / Regional Reference • Map to DAP URL HDF and HDF-EOS Workshop XII

  17. Example: dap2h5 View Group/Ref. Group/Ref. View Data (User) Remote HDF5 • A test application for the client library • It can construct HDF5 from DAP output HDF5-F. Client Library App. (dap2h5) HDF5 Handler ORIGINAL CONSTRUCTED HDF5-Friendly Client Library (liboc-dap) Server (apache) DAP Protocol (via http) HDF and HDF-EOS Workshop XII

  18. Our Client Library(prototype) • One more reason: • Help Clients to view Swath No Latitude and Longitude Courtesy of NASA HDF and HDF-EOS Workshop XII

  19. Visualizing HDF-EOS5 Swath Problem Visualizing HDF-EOS5 Grids View Swath ??? Swath View Data (User) Remote HDF-EOS5 Remote HDF5 Visualization tools (grads) Visualization Tools (gradsoc) HDF5 Handler HDF5 Handler w/ CF option HDF5-Friendly Client Library (liboc-dap) Client Library (libnc-dap) Server (apache) DAP Protocol (via http) HDF and HDF-EOS Workshop XII

  20. Demo: MLS swath via GrADS • GrADS coupled with our client • library • The client library provides grid • mapping from swath data • GrADS displays swath through • HDF-EOS5 specific client library • API calls HDF and HDF-EOS Workshop XII

  21. Summary of Client Library (Prototype) • Finished coding • Tested with the GrADS visualization client • Documented • Working on a demo DAP to HDF5 tool by using the client library prototype • Will test with NCL if time allows HDF and HDF-EOS Workshop XII

  22. Caution • Our Client Library is ONLY A PROTOTYPE! • It does NOT support all DAP data types • It does NOT support all AURA files • It does NOT support all Visualization clients HDF and HDF-EOS Workshop XII

  23. Summary • HDF5 Access via OPeNDAP • Is Easy • Is used by GES DISC to serve Aura • files • HDF5-Friendly OPeNDAP Client Library • Serves HDF5 better(EOS swath) • Benefits visualization clients HDF and HDF-EOS Workshop XII

  24. Future Work • HDF5 to DAP2 Mapping Document • Release HDF5-friendly OPeNDAP • Client Library (Prototype) • URL: http://hdfgroup.org/projects/opendap HDF and HDF-EOS Workshop XII

  25. Credits Mike Folk (The HDF Group) Robert McGrath (NCSA) Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET) Christopher Lynnes, James Johnson, Denis Nadeau, Fan Fang (NASA) Jennifer Adams (GrADS) Dave Brown (UCAR) HDF and HDF-EOS Workshop XII

  26. Acknowledgement • This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A, NNX08A077A and NNX06AG75A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.  HDF and HDF-EOS Workshop XII

More Related