1 / 22

Access HDF5 Datasets via OPeNDAP’s Data Access Protocol (DAP)

Learn how to easily access remote data using OPeNDAP's client-server model, widely used by the Earth Science Community.

zacharyj
Download Presentation

Access HDF5 Datasets via OPeNDAP’s Data Access Protocol (DAP)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Access HDF5 Datasets via OPeNDAP’s Data Access Protocol (DAP) MuQun Yang The HDF Group HDF and HDF-EOS Workshop X, Landover, MD

  2. OPeNDAP • A software framework that allows simple access to remote data • Client-server model • Servers make the local data accessible remotely • Clients make applications easily access to remote served data • Widely used by Earth Science Community • URL: http://www.opendap.org/ HDF and HDF-EOS Workshop X, Landover, MD

  3. The Data Request • Data are requested with a URL. http://www.cdc.noaa.gov/cgi-bin/nph-nc/datasets/Reynolds_sst Protocol Machine name OPeNDAP server Directory File name ?sst[10:10][0:90][0:180] Constraint • The user can impose a constraint on the data that s/he would like to acquire from a data set by appending a constraint expression to the end of the URL. HDF and HDF-EOS Workshop X, Landover, MD

  4. OPeNDAP Data Types OPeNDAP provides three basic data objects: • data descriptor structure (.dds) - syntactic metadata - rigid • data attribute structure (.das) - semantic metadata - flexible • data (.dods) - the actual data in a binary structure HDF and HDF-EOS Workshop X, Landover, MD

  5. OPeNDAP Servers Flat Binary CODAR General netCDF HDF4 Matlab DSP Tables SQL FITS CDF CEDAR Data Data Data Data Data Data Data Data Data Data Data Data ESML netCDF Matlab JGOFS FITS FreeFrom CODAR HDF4 DSP JDBC CDF CEDAR No fully-supported HDF5 server HDF and HDF-EOS Workshop X, Landover, MD

  6. IDL Client Matlab Client netCDF Java netCDF C Ferret GrADS IDV VisAD ncBrowse Matlab IDL Access Excel OPeNDAP Clients No HDF5 client HDF and HDF-EOS Workshop X, Landover, MD

  7. Why HDF5-OPeNDAP server and client? Users! Users!! Users!!! HDF and HDF-EOS Workshop X, Landover, MD

  8. A User Case • Task: • 10 elements of 17 MB HDF5 data • 200 elements of 10 MB NetCDF data • 150 elements of 5 MB HDF4 data HDF and HDF-EOS Workshop X, Landover, MD

  9. General approach HDF5 data FTP HDF4 data FTP NetCDF data FTP HDF and HDF-EOS Workshop X, Landover, MD

  10. Approach with OPeNDAP HDF5 data FTP HDF4 data server DAP client NetCDF data DAP server HDF and HDF-EOS Workshop X, Landover, MD

  11. Conclusion • HDF5-DAP server is required • HDF5-DAP client is stronglyrecommended HDF and HDF-EOS Workshop X, Landover, MD

  12. Previous work - A Prototype of DAP2-HDF5 server • Incomplete mapping from HDF5 to OPeNDAP • No comprehensive test suite • Only tested a few examples on two platforms • Incomplete documentation HDF and HDF-EOS Workshop X, Landover, MD

  13. From HDF4 server at JPL From HDF5 server at NCSA An example from SSMI HDF and HDF-EOS Workshop X, Landover, MD

  14. Current HDF5-OPeNDAP Support HDF5 DAP Prototyped Server No Client HDF and HDF-EOS Workshop X, Landover, MD

  15. HDF5-OPeNDAP Project • Two-year project awarded by NASA ROSES NRA • Funding much less than expected • Original proposed work kept • with significant contributions from ESDIS • Starting date: Nov. 15th, 2006 HDF and HDF-EOS Workshop X, Landover, MD

  16. Proposed Work – software development • Implementation • HDF5-DAP2 server • HDF5-DAP4 server • Deployment • Release servers to OPeNDAP community HDF and HDF-EOS Workshop X, Landover, MD

  17. HDF5-DAP2 server • Update and complete the prototype, including make files, instructions, documentation, etc • Add a few selected features to handle NASA Aura data and NPOESS data • Map HDF5 group to DAP(?) • Map Object/regional reference to DAP(?) • Implement a comprehensive test suite • Test on platforms supported by OPeNDAP • Demonstrate with sample NASA data HDF and HDF-EOS Workshop X, Landover, MD

  18. Fully-Supported HDF5 Server HDF5 HDF5 server DAP HDF5 HDF5 server Matlab/Ferret Client DAP HDF and HDF-EOS Workshop X, Landover, MD

  19. Proposed Work – software development • Implementation • DAP4 to HDF5 conversion utility • Deployment • Release utility to OPeNDAP community • Design study • Feasibilityof DAP-aware HDF5 library HDF and HDF-EOS Workshop X, Landover, MD

  20. Proposed Work - DAP to HDF5 utility • A command line tool that converts DAP to HDF5 file • individual DAP variable • subsetting Conversion tool HDF5 HDF5 server DAP HDF and HDF-EOS Workshop X, Landover, MD

  21. Proposed work - documentations • DAP4 specification • HDF5-DAP4 mapping document • DAP4 to HDF5 specification • RFC for DAP-aware HDF5 library • Submit specifications to ESDSWG HDF and HDF-EOS Workshop X, Landover, MD

  22. Acknowledgement This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNG05GC60A.  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.  HDF and HDF-EOS Workshop X, Landover, MD

More Related