1 / 18

DAP+NETCDF Using the netCDF-4 Data Model

DAP+NETCDF Using the netCDF-4 Data Model. Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009. Overview. RECAP: netCDF-DAP Integration RECAP: Architecture DAP Elements Example DDS and DAS DAP to netCDF-4 Translation netCDF-3 versus netCDF-4 Translation

teige
Download Presentation

DAP+NETCDF Using the netCDF-4 Data Model

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DAP+NETCDFUsing the netCDF-4 Data Model Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009

  2. Overview • RECAP: netCDF-DAP Integration • RECAP: Architecture • DAP Elements • Example DDS and DAS • DAP to netCDF-4 Translation • netCDF-3 versus netCDF-4 Translation • Extending the URL Format • Constraints: Projections • Constraints: Selections • Figuring Out Constraints • Client Parameters Revisited • Building and Installing netCDF-4 + DAP • Debugging Notes

  3. RECAP: NETCDF-DAP Integration • Unidata has integrated the DAP protocol into the standard netCDF distribution. • It replaces the OPeNDAP provided libnc-dap library • It closely mimics the original libnc-dap translation • Implemented completely in C, versus libnc-dap, which is C++ • This talk is primarily about the support for the netCDF-4 data model

  4. GRIB netCDF-3 HDF5 . . . DAP Converter Data Request Using DAP URL Remote Data Server Web Server Internet Data in DAP Format DAP to netCDF Converter Part of the netCDF Library netCDF library NCO (NC operators) NCDUMP Other Programs RECAP: Architecture / Flow

  5. DAP Elements • DDS - describes the format of the data • DAS - describes associated attributes • DATADDS - the actual data; aka DODS • DATADDS contains two things: • A subset of the DDS followed by • The actual data

  6. Web Debugging • If you are knowledgeable about DAP, you can look at the underlying DDS, DAS, DATADDS, and more, using a web browser • Procedure: • Remove the client parameters • Add to the base url one of the following extensions: • .dds, .das, .asc (or .ascii), .html, .info

  7. Example DDS and DAS Dataset { Sequence { String Drifter_ID; String Date_Sampled; Float64 Latitude; Float64 Longitude; Float64 SST; ... } Drifters; } EOS.DBO; Attributes { FacilityGlobal { String DataCenter "COAS"; String DrifterType "MetOcean"; } Drifter_ID { String Description “Instrument unique identifier } Date_Sampled { String Description "Date/time"; String Timezone "GMT"; }...

  8. DAP to NETCDF-4 Translation • The client parameter [mode=netcdf4] will cause the DAP DDS and data to be translated to the netCDF-4 data model • netCDF-4 model is more powerful than DAP, so the translation only uses part of netCDF-4 • Versus the netCDF-3 (classic) translation • Lossless • All DAP primitive types handled (no upgrading) • DAP sequences are translated to combinations of netCDF-4 compound types and vlen types • Performance improvement • No need for string or sequence dimensions

  9. NETCDF-3 vs netCDF-4 translation netcdf Drifters { types: compound Drifters_record_t { string Drifter_ID ; double Latitude ; double Longitude ; double SST; }; Drifters_record_t(*) Drifters_t ; variables: Drifters_t Drifters ; netcdf Drifters { dimensions: Drifters = 5 ; // Sequence // dimension stringdim64 = 64 ; // String // dimension variables: char Drifters.Drifter_ID(Drifters, stringdim64) ; double Drifters.Latitude(Drifters); double Drifters.Longitude(Drifters); double Drifters.SST(Drifters); … …

  10. Extending the DAP URL Format • <clientparams><baseurl>?<constraint> • Additional client parameters • Constraint tells the OPeNDAP server what subset of the dataset to send • Constraint is divided into two parts • Projections • Selections • Currently disabled Data Specified By Full DDS Data Specified By Constraints

  11. Client Parameters Revisited • Explicitly specify [mode=netcdf4] in order to get the netcdf-4 translation • Or specify NC_NETCDF4 mode to nc_open() • Additional [show=…] cases • [show=dds] (attribute “_dds”) • [show=das] (attribute “_das”) • [show=datadds] (attribute “_datadds”) • String attribute defining the DDS, DAS, or DDS part of the DATADS, respectively, as obtained from the OPeNDAP server

  12. Constraints: Projections • Projections specify array slices of underlying OPeNDAP variables to send • http://test.opendap.org/dods/dts?windW[0:10:2] • Format: ?<projection>,<projection>,… • Projection is DAP variable name followed by a slice specification [first:stride:last] • Note: last, not count

  13. Constraints: Selections • Selection constraints tell the OPeNDAP server to subset parts of the dataset based on the values in the dataset • Can only be applied to DAP Sequences • http://test.opendap.org/dap/data/&CS02.light>0 • Format: &<selection>&<selection>&… • Multiple selections are treated as logical “and” • Selections always follow the projections, if any in the URL

  14. Figuring out Constraints • How does one know what constraints to apply? • The problem is that they are defined in terms of the underlying DAP data model => Breaks the netCDF abstraction • For projections, the “show=projection” parameter can help by showing the underlying DAP variables • For Selections, you need to know something about the values in the original dataset • Look at the .info file?

  15. Build and Install NETCDF4 + DAP • Add --enable-dap and --enable-netcdf-4 to your list of ./configure flags • Specify the curl library (libcurl) if ./configure cannot find it --with-curl=<path to curl library> • As usual, of course, specify --with-hdf5 and --with-zlib

  16. Example OPeNDAP Data Sources • http://test.opendap.org:8080/dods/dts • http://test.opendap.org • http://test.opendap.org/dap/data/ff • http://www.opendap.org/data/datasets.cgi ?xmlfilename=datasets.xml&exfunction=none

  17. Questions?

  18. Demo Sites • http://test.opendap.org:8080/dods/dts/Drifters • http://test.opendap.org • http://test.opendap.org/dap/data/ff • http://www.usgodae.org/dods/GDS/coamps_cent_am • http://www.opendap.org/data/datasets.cgi?xmlfilename=datasets.xml&exfunction=none

More Related