1 / 42

HDF5 OPeNDAP Project

MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.). HDF5 OPeNDAP Project. Question of the Day. How can I get One Sub from SUBWAY? C hicken teriyaki with no onion . How can I get One Subset of EOS data image from NASA? Today’s satellite image right above the roof.

leann
Download Presentation

HDF5 OPeNDAP Project

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.) HDF5 OPeNDAP Project

  2. Question of the Day • How can I get One Sub from SUBWAY? • Chicken teriyaki with no onion • How can I get One Subset of EOS data image from NASA? • Today’s satellite image right above the roof

  3. Subway How can I get One Sub from SUBWAY? • Easy: anywhere / accepts Visa|MC • Efficient: your own recipe • Cool: eat fresh and look good

  4. Dumb-way How can I get One Subset of EOS data image from NASA? • Difficult: bring a USB drive and visit NASA • Inefficient: download 10 Gbyte and search for 10 byte dataset • Not cool: wrong or useless dataset

  5. DAP-way! How can I get One Subset of EOS data image from NASA? • Easy: anywhere / accepts IE|FF • Efficient: WYSIWYG – less fat! • Cool: Visualization clients – see fresh!

  6. DAP-way Implementation • OPeNDAP: Free DAP-way Software • Server-Client Model APIs • Widely used by Earth Science Community

  7. OPeNDAP server sites OPeNDAP • Market Dominance: • http://www.opendap.org Courtesy of OPeNDAP, Inc.

  8. Flat Binary CODAR General netCDF HDF4 Matlab DSP Tables SQL FITS CDF CEDAR Data Data Data Data Data Data Data Data Data Data Data Data ESML netCDF Matlab JGOFS FITS FreeFrom CODAR HDF4 DSP JDBC CDF CEDAR OPeNDAP Servers Courtesy of OPeNDAP, Inc.

  9. IDL Client Matlab Client netCDF Java netCDF C Ferret GrADS IDV VisAD ncBrowse Matlab IDL Access Excel OPeNDAP Clients Courtesy of OPeNDAP, Inc.

  10. How OPeNDAP works NetCDF OPeNDAP NetCDF Server HDF4 OPeNDAP Client OPeNDAP HDF4 Server

  11. Architecture of OPeNDAP Server • Server 3 (CGI) • C/C++ based HTTP Server • Insecure / Inflexible • Server 4 (Hyrax) • Java based TomCat Servlet Engine • Secure / Flexible

  12. Server 4 (Hyrax) Architecture OLFS BES Java Servlet Engine Unix Daemon Server 3 (CGI) Architecture HTTP Server BES Commands DAP XML, GIS, KML HTML XML- encapsulated object Data Store Optional catalogs: XML, GIS, KML File system with data files like HDF, NetCDF and SQL Database, … http://docs.opendap.org/index.php/Hyrax Courtesy of OPeNDAP, Inc.

  13. Example Usage HDF4 OPeNDAP Client OPeNDAP HDF4 Server Actual Content of Data Semantic Meaning of Data Syntactic Structure of Data OPeNDAP Visualization Client

  14. What about HDF5? NetCDF OPeNDAP NetCDF Server HDF4 OPeNDAP Client OPeNDAP HDF4 Server HDF5 OPeNDAP HDF5 Server

  15. Why Important? • HDF5: NASA EOS / NPOESS • NASA’s MISSION: ACCESS! • Our MISSION: Build DAP-way Bridge

  16. Goals • 1st: Transform HDF5 access in DAP-way • 2nd: Yet preserve the beauty of HDF5

  17. OPeNDAP HDF5 Server Goal: Transform HDF5 access in DAP-way • Map HDF5 Datatypes in DAP-way • Plus, tweaks for OPeNDAP Clients • Plus, tweaks for HDF-EOS Files

  18. Mapping HDF5 in DAP-way • Prototype server in 2001 • NASA Grant from Nov. 2006 • Support for Compound Datatype • Support for Group • Support for References / Links • First Product Release in Mar. 2008

  19. OPeNDAP HDF5 Server Goal: Transform HDF5 access in DAP-way • Map HDF5 Datatypes in DAP-way • Plus, tweaks for OPeNDAP Clients • Plus, tweaks for HDF-EOS Files

  20. Subway Customers • I want 3 inch-long sub • I need Wasabi sauce • I eat only wheat bread • I eat only meat balls

  21. OPeNDAP Clients Not all OPeNDAP clients are created equal! • I hate foot-long variable names. • I need special attributes on dataset. • I care pre-defined Grid data type. • I care only well-formed attributes.

  22. Tweaks for OPeNDAP Clients • Two configuration options • --enable-short-name = cut bread • --enable-CF = put Wasabi sauce

  23. OPeNDAP HDF5 Server Goal: Transform HDF5 access in DAP-way • Map HDF5 Datatypes in DAP-way • Plus, tweaks for OPeNDAP Clients • Plus, tweaks for HDF-EOS5 Files

  24. Subway Customers • I want 3 inch-long • I need Wasabi sauce • I eat only wheat bread • I eat only meat balls

  25. Subway Customers - OK • I eat only wheat bread • I eat only meat balls Subway Suppliers Problem • No breads, only wheat! • No meat balls, only meat!

  26. HDF EOS Not all HDF5 files are created for DAP-way! • Grid with No Dimension data • Clients expect Grid with Dimension data • Metadata Attribute in Two Strings • Clients expect One Structured Format

  27. Tweaks for HDF-EOS • Two more configuration options • --enable-eos-grid = bake bread • --enable-eos-meta = make balls

  28. OPeNDAP HDF5 Server Goal: Transform HDF5 access in DAP-way • Map HDF5 Datatypes in DAP-way • Plus, tweaks for OPeNDAP Clients • Plus, tweaks for HDF-EOS5 Files

  29. Day After Server Tweaks • Finally, Happy Clients! ncBrowse Ferret MATLAB NCL IDV GrADS ODC

  30. Live Demo Ozone concentration level over the south pole

  31. Goals • 1st: Transform HDF5 access in DAP-way • 2nd: Yet Preserve the beauty of HDF5

  32. HDF5 Served in DAP-way (Yet Preserve the beauty of HDF5) • There are some things money can’t buy. For everything else, there is MasterCardTM. Minimize • There are some things DAP can’t serve. For everything else, there is _________? Maximize

  33. Some Things (that OPeNDAP HDF5 server can’t serve) • Hard: Opaque, Bitmap, Enum, 64 bit Integer, Variable Length types • Illegal: Reserved characters in DAP are used in Dataset/Group name in HDF5

  34. For Everything Else • Let there be HDF5-friendly OPeNDAP client library! • Package and Deliver in DAP-way first. • Then, let the client library handle it. • Concept of Trojan Horse

  35. Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it. Example: Group in HDF5 HDF5-friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5

  36. Example: Reference in HDF5 • Object / Regional Reference • Map to DAP URL at server • No de-referencing of URL at client library • Important for NPOESS

  37. HDF5-Friendly OPeNDAP Client Library No Latitude and Longitude • One more reason: Tame Clients • Only ODC can display Swath properly. • Easy but Evil OPeNDAP NC Client Library! • easy: nc style API • evil: fixed dimensional attributes • Our client library must be Easy but Good Courtesy of NASA

  38. NC-Friendly OPeNDAP Client Library HDF5-Friendly OPeNDAP Client Library ??? View NetCDF View G/R/Swath Group/Ref/Swath NetCDF View HDF5 HDF5 GrADS GrADS OPeNDAP HDF5 Server OPeNDAP NC Server liboc-dap libnc-dap dapserver libdap

  39. Summary • DAP-way access of HDF5 • Is Easy / Efficient / Cool • Loses some things • Requires HDF5-friendly OPeNDAP Client Library • HDF5-friendly OPeNDAP Client Library • Serves HDF5 better • Tames evil visualization clients

  40. Future Work • HDF5 DAP Mapping Document • Finish HDF5-friendly OPeNDAP Client Library prototype • Test it on GrADS and display Swath

  41. Project Website • http://hdfdap.hdfgroup.uiuc.edu/joomla • Beta preview • Feedbacks are more than welcome!

  42. Credits Mike Folk (THG) Robert McGrath (NCSA) Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET) Christopher Lynnes, James Johnson, Denis Nadeau (NASA) Jennifer Adams (GrADS) Dave Brown (UCAR)

More Related