1 / 60

HDF Update

HDF Update. Mike Folk, Kent Yang, Elena Pourmal The HDF Group March 31st, 2009. HDF Project Update Helpdesk HDF Releases Library release highlights Quality Assurance Java Product Highlights Other Tools Activities Outreach Related projects and activities EOS Support OPeNDAP

cnantz
Download Presentation

HDF Update

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HDF Update Mike Folk, Kent Yang, Elena Pourmal The HDF Group March 31st, 2009 Annual HDF Briefing to ESDIS

  2. HDF Project Update • Helpdesk • HDF Releases • Library release highlights • Quality Assurance • Java Product Highlights • Other Tools Activities • Outreach • Related projects and activities • EOS Support • OPeNDAP • netCDF Support • Other projects of interest • Platform support issues

  3. HDF Project Update Annual HDF Briefing to ESDIS

  4. Helpdeskstatistics Annual HDF Briefing to ESDIS

  5. Number of HDF4 and HDF5 issues since 2000 Annual HDF Breifing to ESDIS

  6. HDF4 issues by category Annual HDF Breifing to ESDIS

  7. HDF5 issues by category Annual HDF Breifing to ESDIS

  8. HDF issues by domain Annual HDF Breifing to ESDIS

  9. HDF issues resolution time Annual HDF Breifing to ESDIS

  10. HDF open and closed bugs Annual HDF Breifing to ESDIS

  11. HDF5 open and closed bugs Annual HDF Breifing to ESDIS

  12. HDF4 open and closed bugs Annual HDF Breifing to ESDIS

  13. Questions? Annual HDF Breifing to ESDIS

  14. HDF Releases Annual HDF Breifing to ESDIS

  15. HDF Releases in 2008 - 2009 • HDF4 • HDF4r2.4 (February 2009) • HDF5 • HDF5-1.8.1 (May 2008) • HDF5-1.6.8 (November 2008) • HDF5-1.8.2 (November 2008) • HDF- Java Products • Version 2.5 (February 2009) • H4toH5 • Version 2.0 (June 2008) • Version 2.1 (February 2009) Annual HDF Breifing to ESDIS

  16. HDF Libraries Release Highlights Annual HDF Breifing to ESDIS

  17. HDF4r2.4 • Fixed a flaw in the HDF4 design exposed by applications that use HDF-NetCDF interface to access both HDF4 and NetCDF-3 files with datasets/variables with unlimited dimensions. • Added several new APIs including API to get compressed size of data. • Updated hdp and hrepack to display compression information. • Added support for Fedora 10, Ubuntu 8.10, and OpenSuse 11.1 with GNU C and Fortran. • Better support for Mac OSX (GNU compilers and 32-bit Intel). • http://www.hdfgroup.org/release4/obtain.html Annual HDF Breifing to ESDIS

  18. HDF5-1.6.8 • Many improvements in tools to address ESDIS requests • Availability of compression information in h5dump • Files comparison with h5diff • Support for strings in h5import • Support for user block in h5repack • Added compatibility macros (e.g., H5Gcreate1) for applications to work the same with 1.6.* and 1.8.* • Support for object and region references in C++ • Bug fixes • http://www.hdfgroup.org/HDF5/release/obtain516.html Annual HDF Breifing to ESDIS

  19. HDF5-1.8.1&2 • Tools improvements • Support for external links • Support for a simplified way to produce binary output by h5dump (ESDIS) • Better support for user block with h5repack • Improved performance for free space tracking in a file • Fixed bugs with file mounting • Many improvements in configuration, installation and testing • Better support for shared C, Fortran and C++ libraries • http://www.hdfgroup.org/HDF5/release/obtain5.html Annual HDF Breifing to ESDIS

  20. HDF Java Products • New released is based on JDK 1.6.x, HDF4r2.4 and HDF5-1.8.2 with 1.6 compatibility • Java products (i.e. HDFView) can read files created by HDF5 1.8.2 • New HDFView: • Supports access to remote HDF5 files stored at the iRODS server (Windows and Linux only). For information on HDF5-iRODS module, see http://www.hdfgroup.org/projects/irods/index.html. • Shows large dataset (e.g. 1,000,000 x 100) in a table viewer • Converts JPEG, GIF, PNG, and BMP images to HDF5 or HDF4 • Saves HDF4 or HDF5 images to JPEG, GIF, PNG, and BMP images • Writes variable length string data • Filters fill values for images Annual HDF Breifing to ESDIS

  21. H4toH5 tools and library • Based on HDF4r2.4 and HDF5-1.8.2 • h4toh5 utility • Recognizes HDF-EOS2 files (--with-hdfeos2 configuration option) • Can generate HDF5 files that can be read by netCDF-4 • h4toh5 library • Bug fixes • Performance improvements Annual HDF Breifing to ESDIS

  22. Quality Assurance Some work of interest to ESDIS Annual HDF Breifing to ESDIS

  23. QA Efforts • We test at least 250 different configurations for HDF software (OSs x compilers x HDF configurations) every day • Regression testing of HDF5 released branches (1.6. and 1.8) and development branch (1.9), and HDF4 • Platforms • Linux 32 and 64, Solaris 32 and 64, Free BSD 32 and 64, AIX 32 and 64 (at LLNL), Windows 32 and 64, Linux Clusters, SGI Altix • Backward/forward compatibility testing • Third part software testing Annual HDF Breifing to ESDIS

  24. Backward/forward compatibility testing • Goal: assure interoperability between 1.6 and later versions of the HDF5 library used to create AURA files • Regression tests with and without strict format checks using known problem files (currently MLS Cloud series with corrupted headers) • Download sample from subdirectories of OMI, MLS, HIRDL, TOMS and TES files. Sample consists of files from the current day of the year for available years in two of the subdirectories listed for the above instruments. Annual HDF Breifing to ESDIS

  25. Backward/forward compatibility testing • FTP sites (about 540 000 files) • OMI Level 2 files • ftp://aurapar2u.ecs.nasa.gov • OMI Level2G, MLS and HIRDL files • ftp://acdisc.gsfc.nasa.gov  • TOM files • ftp://acdisc.sci.gsfc.nasa.gov • TES files • ftp://l4ftl01.larc.nasa.gov Annual HDF Breifing to ESDIS

  26. Backward/forward compatibility testing • Each known subdirectory will be visited ~5 times per year. • hdf5 tools from versions 1.6.9, 1.8.3, and 1.9.0 are checked for proper operation with downloaded files. • Downloaded files are copied with h5copy and repacked with h5repack. The resulting files are checked again with all 3 versions of the tools. Annual HDF Breifing to ESDIS

  27. Third party software testing • NetCDF-4 • Daily tests of the netCDF4 snapshot • Built against current HDF5 v1.8.* • Tested on Linux 32/64 bit and solaris • HDF-EOS2 and HDF-EOS5 testing • HDFEOS2 with h4cc, 32/64 bit Linux, 32 bit solaris • HDFEOS5 with hdf5 v1.6 and v1.9, 32/64 bit Linux, 32/64 bit solaris • IDL wrappers • Created IDL/HDF module binaries based on HDF5 1.8.2 and HDF4.2r3 Annual HDF Breifing to ESDIS

  28. Questions? Annual HDF Breifing to ESDIS

  29. Java Product Highlights Annual HDF Briefing to ESDIS

  30. HDF Java Products Highlights • V2.5 released in February 2009 • 50 bug fixes/enhancements • Support for HDF5 1.8-with-1.6 compatibility(Does not support new HDF5 1.8 features) Annual HDF Briefing to ESDIS

  31. HDFView Highlights • Support for iRODS • Show large dataset in table • Write variable length string • Convert image to/from HDF4/5 • JPEG • GIF • PNG • BMP Annual HDF Briefing to ESDIS

  32. Future work for Java • 26 bugs/enhancements • hdf-java 2.6 release • Include latest HDF4 and HDF5 1.8 libraries • Fix bugs and add new enhancements • Full support for HDF5 1.8 (next slide) Annual HDF Briefing to ESDIS

  33. Future work for Java • Full support for HDF5 1.8 • Add HDF5 1.8 APIs to the HDF5 Java wrapper • Initial implementation (done) • Review and revision is in progress • Implement and test HDF5 JNI C layer • Implementation (200+ APIs), 1 month fte • Testing, 1 month fte • Support new features in HDF5 1.8 • Revise hdf-java object package • Add new features to HDFView April 18-19, 2007 Annual HDF Briefing to ESDIS

  34. Project Plan for 1.8 support Annual HDF Briefing to ESDIS

  35. Other Tools Activities Annual HDF Briefing to ESDIS 35

  36. Improvements in HDF4 tools • Display compression information in hdf4 tools • Preserve unlimited dimensions in hrepack • Fix bugs in hdp and hdiff reported by DAACs. Annual HDF Briefing to ESDIS

  37. HDF5 tools highlights • 35 bug fixes/enhancements to HDF5 tools • All bug fixes/enhancements are applied to both HDF5 1.6 and 1.8 release Annual HDF Briefing to ESDIS

  38. Highlights of HDF5 Tools enhancements • h5dump • Display compression information • Follow external links • Add ability to specify precision output • Use MEMORY option for default binary output • h5diff • Display the precision the differences • Handle NaN values • h5repack • Keep the original user block information • Multiple filters can be invoked Annual HDF Briefing to ESDIS

  39. Outreach Annual HDF Briefing to ESDIS

  40. Earth Science Related Outreach IEEE Transactions on Geosciences and Remote Sensing, vol. 47, issue 1 Paper: “Ensuring Long Term Access to Remotely Sensed Data with Layout Maps” by Duerr, Cao, Folk, Lynnes, Yang. ESIP Federation Summer meeting, July 2008 Presented a poster about OPeNDAP work Meeting with ITTVis, Oct. 2008 Presentation about HDF Worked with them to test the IDL-HDF5 module with a new HDF5 release 7th NASA ESDSWG conference, Oct. 2008 Reported on the OPeNDAP project, participated in several working groups AGU Fall meeting, Dec. 2008 Presentation: “Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps,” by Folk and Duerr American Meteorological Society Annual Meeting, Jan. 2009 Presentations on OPeNDAP and the HDF-EOS2 to netCDF-4 conversion tool April 1-2, 2008 Annual HDF Briefing to ESDIS Annual HDF Briefing to ESDIS 40

  41. HDF-EOS Workshop XII • Tutorials • Introduction to HDF5 Data and Programming Models • Advanced HDF5 Features • Migrating from HDF5 1.6 to 1.8 • Introduction to HDF5 Tools • HDFView and HDF Java Products • What will be new in HDF5 • Status • HDF Status and Developments • HDF Group Support for NPP/NPOESS • Applications and discussions • NOAA Science Data Stewardship project update • HDF5-OPeNDAP Project Update and Demo • Independent HDF4 mapping project update • HDF5 IRODS • Users: What are your needs • No programming language left behind Annual HDF Briefing to ESDIS

  42. Other outreach - sample • Presentation “HDF5: Toward a Universal Platform for Science and Engineering Data,” Folk • National Renewable Energy Lab Scientific and Technical Data Management and Analysis Workshop ’08, May 2008 • “HDF Experiences with I/O Bottlenecks,” Folk • NSF/NARA Collaborative Expedition Workshop: Toward Scalable Data Management – Overcoming I/O Bottlenecks, June 10, 2008 • HDF5 Tutorial, Pourmal and Cheng • 37th SPEEDUP Workshop on HPC, Zurich, Sept. 2008 • “HDF5: Past, Present, and Future”“Tuning Parallel HDF5 for HPCApplications,” Koziol, Aydt • National Energy Research Scientific Computing Center HDF5 Workshop on Scalable Performance Annual HDF Briefing to ESDIS

  43. Related projects and activities Annual HDF Briefing to ESDIS

  44. EOS Support Annual HDF Briefing to ESDIS

  45. EOS support • HDF-EOS2 and HDF-EOS5 • Automatic configuration with szip enabled/disabled • EOS2 and EOS5 are tested daily with HDF4 and HDF5 development code • HDF-EOS website • workshop coordination • EOS product release 1/6/2020 The HDF Group for 2008 annual briefing Annual HDF Briefing to ESDIS 45

  46. OPeNDAP Annual HDF Briefing to ESDIS

  47. OPeNDAP – current status • HDF5-OPeNDAP handler • Served OMI and TES data • HDF5 –awareness OPeNDAP client library • HDF4-OPeNDAP handler • Tested with AIRS data and some MODIS data • More information in the afternoon session 1/6/2020 The HDF Group for 2008 annual briefing Annual HDF Briefing to ESDIS 47

  48. netCDF Support Annual HDF Briefing to ESDIS

  49. NetCDF4 support • Help Unidata NetCDF4 developers • Frequent communications with NetCDF4 developers • Test NetCDF4 daily with the HDF5 development code • NetCDF4 performance benchmark report • http://www.hdfgroup.org/pubs/papers/2008-06_netcdf4_perf_report.pdf 1/6/2020 The HDF Group for 2008 annual briefing Annual HDF Briefing to ESDIS 49

  50. Long Term Access to Remotely Sensed HDF4 Data with Layout Maps Discussed later Annual HDF Briefing to ESDIS

More Related