260 likes | 471 Views
Data Interoperability. The HDF Group Staff. Interoperability. Interoperability is a property of a product or system, whose interfaces are completely understood, to work with other products or systems, present or future, without any restricted access or implementation. (From Wiki).
E N D
Data Interoperability The HDF Group Staff HDF/HDF-EOS Workshop XIV
Interoperability • Interoperability is a property of a product or system, whose interfaces are completely understood, to work with other products or systems, present or future, without any restricted access or implementation. (From Wiki) HDF/HDF-EOS Workshop XIV
Background for data interoperability • Focus on Earth Sciences • Data needs to be shared across communities • Self-described data format • Many different types of data • Grids, moving-sensor multidimensional fields, time series, profiles, trajectories, geospatial framework data, points HDF/HDF-EOS Workshop XIV
Why data interoperability? • Data format • Several data formats available HDF4/5, netCDF3/4, GRIB, BUFR, Binary • Data are created with some data formats by following different physical models • End users would like to use their familiar tools to access data via different formats HDF/HDF-EOS Workshop XIV
Why data interoperability? • Semantic/content interoperability • An example: the missing value of a physical variable • Tools or applications need to know the fill value of a physical variable • How can they identify? Normally via an attribute that stores the fill value • The name can be “Fill_value, _FillValue, badValue,……” HDF/HDF-EOS Workshop XIV
Goals • Discuss a few cases about data interoperability • Hear opinions or issues about data interoperability from audiences HDF/HDF-EOS Workshop XIV
Data Format Interoperability HDF/HDF-EOS Workshop XIV
Case 1 • One set of APIs to handle multiple data formats HDF/HDF-EOS Workshop XIV
Case 1 • netCDF4 • Combine powerful model and simplicity of netCDF with features of HDF5 • netCDF interface on top of HDF5 • CDM(Common Data Model) • Harmonize netCDF, HDF4, HDF5, OPeNDAP, GRIB and others HDF/HDF-EOS Workshop XIV
Discussions • Any comments/sharings about the experiences with netCDF4 and CDM? HDF/HDF-EOS Workshop XIV
Case 2 • Format conversions and translations HDF/HDF-EOS Workshop XIV
Data format Conversions/Translations • Format conversions • The HDF4 to HDF5 conversion tool • The HDF-EOS2 to netCDF3/4 conversion tool • HDF-EOS to GeoTIFF conversion tool • …… • Data format Translations • netCDF tools to access HDF4/5 files via OPeNDAP • …… HDF/HDF-EOS Workshop XIV
Use netCDF tools to access HDF via OPeNDAP Aqua/Aura Translation Layers Users HDF4/5 Handlers IDV libnc-dap AIRS/OMI DAP Visualization Tools OPeNDAP Clients OPeNDAP Servers HDF4/5 Files HDF/HDF-EOS Workshop XIV
netCDF4 to access HDF-EOS5 files • Augmentation • One file can be used for both EOS5 and NetCDF-4. • Note that EOS5 users are not affected at all. NetCDF-4 file Augmentation HDF-EOS5 file HDF-EOS5 Augmented HDF-EOS5 file NetCDF4 HDF5 HDF5 Sep. 28-30, 2010 HDF/HDF-EOS Workshop XIV 14
Discussions • Any comments/sharings about the experiences with data format conversions and translations HDF/HDF-EOS Workshop XIV
Semantic/Content Interoperability HDF/HDF-EOS Workshop XIV
An example • Degree of latitude and longitude • Champaign’s latitude is 40°6´38" N • It can be represented in two formats: • 400638 in DDDMMSS format • 40.1105556 in decimal format • How can the applications know which format the file is used? • Better to have a common standard to facilitate the exchange this kind of information HDF/HDF-EOS Workshop XIV
Semantic/Content Interoperability • CF conventions become such a standard • Many applications/tools follow CF conventions to interpret the metadata HDF/HDF-EOS Workshop XIV
Use netCDF tools to access HDF via OPeNDAP Aqua/Aura Not follow CF conventions Users HDF4/5 Handlers IDV libnc-dap AIRS/OMI DAP Visualization Tools OPeNDAP Clients OPeNDAP Servers HDF4/5 Files HDF/HDF-EOS Workshop XIV
Discussions • Any comments about semantic/content interoperability • Any comments about the usage of CF conventions HDF/HDF-EOS Workshop XIV
HDF5 and netCDF-4 Libraries InteroperabilityTwo sources of interoperability problems and how to deal with them HDF/HDF-EOS Workshop XIV
netCDF-4 library and arbitrary HDF5 files • netCDF-4 is built on top of HDF5 • netCDF-4 files are HDF5 files with specific characteristics • Tracked creation order of objects and their attributes • Absence of datasets with complex compound, and region and object references • Presence of dimension scales • netCDF-4 expects to find properties 1 and 2 in an HDF5 file and fails to open or read the file when those are not present (e.g., the NPOESS files) • Solution: TB discussed with the netCDF-4 folks HDF/HDF-EOS Workshop XIV
Programming models and libraries settings • Use case 1 (libraries settings): Simultaneous access to an HDF5 file via HDF5 and netCDF-4 • One of the libraries will not be able to open the file due to the different access properties used by each library (H5F_CLOSE_WEAK in HDF5 vs. H5F_CLOSE_ STRONG in netCDF-4) • Solution: • Design improvements in HDF5 • New APIs to detect situation • Automatic detection and correction of the access properties • Document the best practices for applications HDF/HDF-EOS Workshop XIV
Programming models and libraries settings • Use case 2 (programming models): Usage of HDF5 wrapper libraries and netCDF-4 library in the same application • C++ H5Topen call fails due to the netCDF-4 library shutting down the HDF5 library underneath by calling H5close • Solution: • Resolved in the newest netCDF-4 • Document the best practices for applications • For more detailed discussion see “NetCDF-4/HDF5 Libraries Interoperability Issues” at http://www.hdfgroup.uiuc.edu/RFC/HDF5/netCDF4-HDF5/ • Let us know your use case to improve testing! HDF/HDF-EOS Workshop XIV
Thank you ! HDF/HDF-EOS Workshop XIV