HDF5 OPeNDAP Project Update and Demo

1. HDF5 OPeNDAP ProjectUpdate and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) HDF and HDF-EOS Workshop XII, Aurora, CO

2. OPeNDAP Review • A software framework that allows simple access to remote data • Data Access Protocol • Client-server model • Widely used by Earth Science Community HDF and HDF-EOS Workshop XII, Aurora, CO

3. Example Usage HDF4 OPeNDAP Client OPeNDAP HDF4 Server Actual Content of Data Semantic Meaning of Data Syntactic Structure of Data OPeNDAP Visualization Client HDF and HDF-EOS Workshop XII, Aurora, CO

4. Why Important? • HDF5: NASA EOS / NPOESS • NASA’s MISSION: ACCESS! • Our MISSION: Build a DAP-HDF5 Bridge HDF and HDF-EOS Workshop XII, Aurora, CO

5. OPeNDAP HDF5 Server Goal: Access HDF5 data via OPeNDAP • Map HDF5 Datatypes into DAP Datatypes • Plus, tweaks for OPeNDAP Clients to access HDF-EOS5 Files HDF and HDF-EOS Workshop XII, Aurora, CO

6. Mapping HDF5 Datatypes to DAP • Compound Datatype • Support for Group • Object/regional References HDF and HDF-EOS Workshop XII, Aurora, CO

7. Some OPeNDAP Clients Not all OPeNDAP clients are created equal! • Cannot handle long variable names. • Need special attributes on dataset. • Handle only DAP Grid datatype. • Handle only well-formed attributes. HDF and HDF-EOS Workshop XII, Aurora, CO

8. Tweaks for OPeNDAP Clients • Two configuration options • --enable-short-name • --enable-CF HDF and HDF-EOS Workshop XII, Aurora, CO

9. HDF EOS Not all HDF5 files are created for DAP! • Grid with No Dimension data • Clients expect Grid with Dimension data • Some Attributes stored in Extremely Long String(s) (e.g. StructMetada.0) • Clients expect well-structured attributes HDF and HDF-EOS Workshop XII, Aurora, CO

10. Tweaks for HDF-EOS • Two more configuration options • --enable-eos-grid • --enable-eos-meta HDF and HDF-EOS Workshop XII, Aurora, CO

11. Day After Server Tweaks • Finally, Happy Clients! IDL ncBrowse Ferret MATLAB NCL IDV GrADS ODC HDF and HDF-EOS Workshop XII, Aurora, CO

12. Some Things (that OPeNDAP HDF5 server can’t serve) • Hard: HDF5 Opaque, Bitmap, Enum, Variable Length types • Illegal: Reserved characters in DAP are used in Dataset/Group name in HDF5 HDF and HDF-EOS Workshop XII, Aurora, CO

13. Goals • 1st : Access HDF5 via OPeNDAP • The OPeNDAP HDF5 handler was released on March, 2008 at OPeNDAP website http://www.opendap.org/ • 2nd: Construct HDF5 data from OPeNDAP • Bonus: Help clients to better access EOS Swath data HDF and HDF-EOS Workshop XII, Aurora, CO

14. HDF5-Friendly OPeNDAP client library HDF and HDF-EOS Workshop XII, Aurora, CO

15. Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it. Example: Group in HDF5 HDF5-Friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5 HDF and HDF-EOS Workshop XII, Aurora, CO

16. Example: Reference in HDF5 • Object / Regional Reference • Map to DAP URL • Important for NPOESS HDF and HDF-EOS Workshop XII, Aurora, CO

17. HDF5-Friendly OPeNDAP Client Library • One more reason: Help Clients to view Swath No Latitude and Longitude • Only ODC can display Swath properly. • Easy but Rigid OPeNDAP NC Client Library! • easy: same as netCDF C interface • rigid: follows netCDF model • Our client library must be Easy but Flexible Courtesy of NASA HDF and HDF-EOS Workshop XII, Aurora, CO

18. NC-Friendly OPeNDAP Client Library HDF5-Friendly OPeNDAP Client Library ??? View G/R/Swath View NetCDF NetCDF Group/Ref/Swath View HDF5 HDF5 GrADS GrADS OPeNDAP NC Server OPeNDAP HDF5 Server libnc-dap liboc-dap dapserver libdap HDF and HDF-EOS Workshop XII, Aurora, CO

19. Summary of Client Library Prototype • Finished coding • Tested with GrADS visualization client • Documented • Working on a demo DAP to HDF5 tool via the client library prototype • May test with NCL HDF and HDF-EOS Workshop XII, Aurora, CO

20. Example: dap2h5 • A test application for the client library • It can construct HDF5 from DAP output ORIGINAL CONSTRUCTED HDF and HDF-EOS Workshop XII, Aurora, CO

21. Demo: MLS swath via GrADS • GrADS is modified to use our library • The client library provides grid mapping from swath data • GrADS displays swath through HDF-EOS5 specific client library API calls HDF and HDF-EOS Workshop XII, Aurora, CO

22. Caution • Our Client Library is ONLY PROTOTYPE! • It does NOT support all DAP data types • It does NOT support all AURA files • It does NOT support all Visualization clients HDF and HDF-EOS Workshop XII, Aurora, CO

23. Summary • HDF5 Access via OPeNDAP • Is Easy • Is used by GES DISC to serve Aura files • HDF5-Friendly OPeNDAP Client Library Prototype • Serves HDF5 better(EOS swath) • Benefits visualization clients HDF and HDF-EOS Workshop XII, Aurora, CO

24. Future Work • HDF5 to DAP2 Mapping Document • Release HDF5-friendly OPeNDAP Client Library Prototype • URL: http://hdfgroup.org/projects/opendap/index.html HDF and HDF-EOS Workshop XII, Aurora, CO

25. Credits Mike Folk (THG) Robert McGrath (NCSA) Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET) Christopher Lynnes, James Johnson, Denis Nadeau, Fan Fang (NASA) Jennifer Adams (GrADS) Dave Brown (UCAR) HDF and HDF-EOS Workshop XII, Aurora, CO

26. Acknowledgement • This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX06AG75A..  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.  HDF and HDF-EOS Workshop XII, Aurora, CO