Data Interoperability

1. Data Interoperability The HDF Group Staff HDF/HDF-EOS Workshop XIV

2. Interoperability • Interoperability is a property of a product or system, whose interfaces are completely understood, to work with other products or systems, present or future, without any restricted access or implementation. (From Wikipedia) HDF/HDF-EOS Workshop XIV

3. Background for data interoperability • Focus on Earth Sciences • Data needs to be shared across communities • Self-described data format • Many different types of data • Grids, moving-sensor multidimensional fields, time series, profiles, trajectories, geospatial framework data, points HDF/HDF-EOS Workshop XIV

4. Why data interoperability? • Data format • Several data formats available HDF4/5, netCDF3/4, GRIB, BUFR, Binary • Data are created with some data formats by following different physical models • End users would like to use familiar tools to access data via different formats HDF/HDF-EOS Workshop XIV

5. Why data interoperability? • Semantic/content interoperability • An example: the missing value of a physical variable • Tools or applications need to know the fill value of a physical variable • How can they identify? Normally via an attribute that stores the fill value • The name can be “Fill_value, _FillValue, badValue,……” HDF/HDF-EOS Workshop XIV

6. Goals • Discuss a few cases about data interoperability • Hear opinions or issues about data interoperability from audience. HDF/HDF-EOS Workshop XIV

7. Data Format Interoperability HDF/HDF-EOS Workshop XIV

8. Case 1 • One set of APIs to handle multiple data formats HDF/HDF-EOS Workshop XIV

9. Case 1 • netCDF4 • Combine powerful model and simplicity of netCDF with features of HDF5 • netCDF interface on top of HDF5 • CDM(Common Data Model) • Harmonize netCDF, HDF4, HDF5, OPeNDAP, GRIB and others HDF/HDF-EOS Workshop XIV

10. Discussions • Any comments/sharings about the experiences with netCDF4 and CDM? HDF/HDF-EOS Workshop XIV

11. Case 2 • Format conversions and translations HDF/HDF-EOS Workshop XIV

12. Data format Conversions/Translations • Format conversions • The HDF4 to HDF5 conversion tool • The HDF-EOS2 to netCDF3/4 conversion tool • HDF-EOS to GeoTIFF conversion tool • …… • Data format Translations • netCDF tools to access HDF4/5 files via OPeNDAP • …… HDF/HDF-EOS Workshop XIV

13. Use netCDF tools to access HDF via OPeNDAP Aqua/Aura Translation Layers Users HDF4/5 Handlers IDV libnc-dap AIRS/OMI DAP Visualization Tools OPeNDAP Clients OPeNDAP Servers HDF4/5 Files HDF/HDF-EOS Workshop XIV

14. netCDF4 to access HDF-EOS5 files • Augmentation • One file can be used for both EOS5 and NetCDF-4. • Note that EOS5 users are not affected at all. NetCDF-4 file Augmentation HDF-EOS5 file HDF-EOS5 Augmented HDF-EOS5 file NetCDF4 HDF5 HDF5 Sep. 28-30, 2010 HDF/HDF-EOS Workshop XIV 14

15. Discussions • Any comments/sharings about the experiences with data format conversions and translations HDF/HDF-EOS Workshop XIV

16. Semantic/Content Interoperability HDF/HDF-EOS Workshop XIV

17. An example • Degree of latitude and longitude • Champaign’s latitude is 40°6´38" N • It can be represented in two formats: • 400638 in DDDMMSS format • 40.1105556 in decimal format • How can the applications know which format the file is used? • Better to have a common standard to facilitate the exchange this kind of information HDF/HDF-EOS Workshop XIV

18. Semantic/Content Interoperability • CF conventions become such a standard • Many applications/tools follow CF conventions to interpret the metadata HDF/HDF-EOS Workshop XIV

19. Use netCDF tools to access HDF via OPeNDAP Aqua/Aura Not follow CF conventions Users HDF4/5 Handlers IDV libnc-dap AIRS/OMI DAP Visualization Tools OPeNDAP Clients OPeNDAP Servers HDF4/5 Files HDF/HDF-EOS Workshop XIV

20. Discussions • Any comments about semantic/content interoperability • Any comments about the usage of CF conventions HDF/HDF-EOS Workshop XIV

21. HDF5 and netCDF-4 Libraries InteroperabilityTwo sources of interoperability problems and how to deal with them HDF/HDF-EOS Workshop XIV

22. netCDF-4 library and arbitrary HDF5 files • netCDF-4 is built on top of HDF5 • netCDF-4 files are HDF5 files with specific characteristics • Tracked creation order of objects and their attributes • Absence of datasets with complex compound, and region and object references • Presence of dimension scales • netCDF-4 expects to find properties 1 and 2 in an HDF5 file and fails to open or read the file when those are not present (e.g., the NPOESS files) • Solution: TB discussed with the netCDF-4 folks HDF/HDF-EOS Workshop XIV

23. Programming models and libraries settings • Use case 1 (libraries settings): Simultaneous access to an HDF5 file via HDF5 and netCDF-4 • One of the libraries will not be able to open the file due to the different access properties used by each library (H5F_CLOSE_WEAK in HDF5 vs. H5F_CLOSE_ STRONG in netCDF-4) • Solution: • Design improvements in HDF5 • New APIs to detect situation • Automatic detection and correction of the access properties • Document the best practices for applications HDF/HDF-EOS Workshop XIV

24. Programming models and libraries settings • Use case 2 (programming models): Usage of HDF5 wrapper libraries and netCDF-4 library in the same application • C++ H5Topen call fails due to the netCDF-4 library shutting down the HDF5 library underneath by calling H5close • Solution: • Resolved in the newest netCDF-4 • Document the best practices for applications • For more detailed discussion see “NetCDF-4/HDF5 Libraries Interoperability Issues” at http://www.hdfgroup.uiuc.edu/RFC/HDF5/netCDF4-HDF5/ • Let us know your use case to improve testing! HDF/HDF-EOS Workshop XIV

25. Thank you ! HDF/HDF-EOS Workshop XIV