1 / 72

HDF Project Update

This briefing provides an update on the activities of the HDF project in supporting NASA HDF data providers and users, including maintenance and quality assurance for HDF4 and HDF5 libraries, integration with complementary technologies, and prototype study of web technologies to access HDF data.

bittinger
Download Presentation

HDF Project Update

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HDF Project Update Mike Folk, Elena Pourmal, Kent Yang The HDF Group Briefing to ESDIS October 16-17, 2012 HDF Briefing to NASA

  2. Project Goals • Support for NASA HDF data providers and data users, and the EOSDIS Core System (ECS) • Maintenance and quality assurance for HDF4 and HDF5 libraries and utilities • Evolving the HDF4 and HDF5 libraries and utilities to meet new NASA EOS requirement • Integration with complementary technologies and application domains • Prototype study of web technologies to access HDF data HDF Briefing to NASA

  3. Support activities for NASA HDF data providers and users and ECS HDF Briefing to NASA

  4. Activities • Help Desk • Outreach • Mailing lists, Web site • Conferences, Workshops, Other • EOS support • netCDF support and collaboration • Later today: • HDF OPeNDAP support • Support for NASA data products • SMAP, ICESat-2, MEaSURES) HDF Briefing to NASA

  5. Help desk HDF Briefing to NASA

  6. The HDF Group Help Desk • High-priority support for ESDIS users • Fast response • Problem investigation and resolution • Performance tuning • Access to HDF developers • Basic support for entire HDF user community, as resources permit • help@hdfgroup.org HDF Briefing to NASA

  7. Help Desk issues resolution time *Number of messages up do August 31, 2012 HDF Briefing to NASA

  8. Help Desk issues by years *Number of messages up do August 31, 2012 HDF Briefing to NASA

  9. outreach HDF Briefing to NASA

  10. Outreach • Web pages improvements • FAQ support for HDF4 and HDF5 http://www.hdfgroup.org/HDF5-FAQ.html http://www.hdfgroup.org/HDF-FAQ.html • Later today: Detailed talk on support for EOS (Kent) • Organizing material based on the ESDIS survey results and feedback from the users • Example: Tools by category (work in progress) • How do I edit, copy, import/export data with HDF? HDF Briefing to NASA

  11. Tools by category HDF Briefing to NASA

  12. Outreach • Mailing list and archives news@hdfgroup.org http://www.hdfgroup.org/news/ hdf-forum@hdfgroup.org http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/ HDF Briefing to NASA

  13. hdf-forum@hdfgroup.org • Started in Summer 2007 • Monitored by the members of the HDF group • Number of messages: • Most discussed topics: • Parallel HDF5 • Advanced HDF5 features (drivers, complex types, performance) • .NET and C++ wrappers, HL libraries HDF Briefing to NASA

  14. hdf-forum@hdfgroup.org • Members of the FORUM help with • Releases testing on a regular basis • Maintaining CMake build systems on platforms beyond Windows • Answering questions • The Help Desk focuses on ESDIS and other paying customers while referring others to the FORUM for topics that require domain knowledge or specific HDF5 usage • Reporting defects • Securing funding for The HDF Group HDF Briefing to NASA

  15. Conferences, Workshops, Other • 2011 IDV/THREDDS Workshop, July 2011 • Learned how to use IDV and THREDDS • 10th NASA ESDSWG Conference, Nov 2011 • Poster on HDF and netCDF interoperability • 2012 ESIP Summer Meeting, July 2012 • Presentations • HDF/HDF-EOS Data for Dummies • What does Open Source mean for HDF? • Poster • What’s new at hdfgroup.org? HDF Briefing to NASA

  16. Conferences, Workshops, Other • 15th HDF and HDF-EOS Workshop, April 2012 • Five tutorials, nine presentations and lightning talks, and one special session • Materials: http://www.hdfeos.org/workshops/ws15/agenda.php HDF Briefing to NASA

  17. Conferences, Workshops, Other • Advised ICESat-2, MOPITT and several MEaSURES projects on interoperability with netCDF-4 • Participated in the SMAP ISO-19115 work • Participated in ESDSWG SPG telecons • Joined several ESDSWG mailing lists related to HDF HDF Briefing to NASA

  18. netCDF support and collaboration HDF Briefing to NASA

  19. NetCDF-4 support • Help Unidata netCDF-4 developers • Frequent communications with netCDF-4 developers • Provided a fix for ncdump critical to the ESDIS users • Test netCDF-4 daily with the HDF5 development code • Plan to participate in Unidata Advanced netCDF Workshop on October 26, 2012 HDF Briefing to NASA

  20. EOS support HDF Briefing to NASA

  21. EOS Support • Submitted several bugs and the corresponding fixes to the HDF-EOS team • EOS2 and EOS5 are tested daily with HDF4 and HDF5 development code • HDF-EOS website • Actively maintain the website • Improve the quality of the example codes and plots to access NASA HDF and HDF-EOS data via MATLAB, IDL and NCL • More information in a separate talk HDF Briefing to NASA

  22. Maintenance and quality assurance for HDF4 and HDF5 libraries and utilities HDF Briefing to NASA

  23. Activities • Manage and resolve issues • Perform maintenance releases of HDF software • Support OSs and compilers identified by ESDIS • Address issues and software evolution requests from ESDIS • Perform SQE and SQA activities • Put in place missing processes • Improve existing processes • Build in software quality by performing testing and using the best software development practices HDF Briefing to NASA

  24. HDF issues HDF Briefing to NASA

  25. Issues and their priorities • Must Fix • Data corruption • Portability • Backward and Forward Compatibility • Funded Requests • Fix after “Must Fix” • Power User Requests • Tools • Library issues • Build Infrastructure • When resources permit • Wrappers • HL Libraries • Other Need your input on priorities! HDF Briefing to NASA

  26. HDF4 opened and closed issues HDF Briefing to NASA

  27. HDF5 opened and closed issues HDF Briefing to NASA

  28. SQE and SQA at the HDF group HDF Briefing to NASA

  29. Activities • CMMI appraisal and process improvement action plan (Dec. 2011) • Created SQE position • Focus on • Implementing CMMI best practices and procedures • Enforcing current practices and procedures • Focused on improving software testing HDF Briefing to NASA

  30. Activities • Worked on establishing a culture of code reviews for the HDF libraries and tools • Improved releases planning and monitoring for HDF5 • Used static analysis tools to identify defects • Coverity (used at THG) • Klocwork (used at LLNL) HDF Briefing to NASA

  31. HDF Maintenance releases HDF Briefing to NASA

  32. General maintenance for 2011-2012 • Implemented CMake for all HDF software • Worked on closing reported issues • Focused on performance enhancements • Improved HDF documentation • http://www.hdfgroup.org/HDF5/doc/Advanced.html • Explored the needs for internal documentation • Preventive measures: • Coverity static analysis; Rational tools; valgrind • Have been exploring new testing tools to sustain testing for HDF products (CDash, Bamboo, Cloud) HDF Briefing to NASA

  33. Maintenance Releases 2011 – 2012 CF – code freeze 6 weeks for HDF5 4 weeks for other products HDF Briefing to NASA

  34. HDF4 maintenance releases • HDF 4.2.7 released in February 2012 • Extra functions to support H4 mapping project • Support for Linux PPC64 with IBM XL Fortran • Minor bug fixes and documentation improvements • HDF 4.2.8 release in August 2012 • Support for H4mapping project • Information about releases available from http://www.hdfgroup.org/ftp/HDF/prev-releases/ReleaseFiles/ HDF Briefing to NASA

  35. HDF5 maintenance releases • HDF5 1.8.7-1.8.9 • Many improvements in tools • ESDIS request to handle “/” in attributes names • Fortran 2003 support • Fortran wrappers for Dimension Scales APIs • Information about releases available from http://www.hdfgroup.org/ftp/HDF5/prev-releases/ReleaseFiles/ • HDF5 1.8.10 November 15, 2012 • Many improvements in h5diff, minor bug fixe HDF Briefing to NASA

  36. H4h5tools maintenance releases • 2.2.1 released February, 15th, 2012 • Based on HDF 4.2.7 and HDF5 1.8.8 • Fixed bugs reported by ESDIS users • Upgraded to HDF5 1.8 APIs • Updated documents HDF Briefing to NASA

  37. Java maintenance releases • 2.8 release December 2011 • Addressed all issues reported by ESDIS users • 2.9 release December 2012 • Based on HDF 4.2.8 and HDF5 1.8.10 • ESDIS requests: • Show groups/attributes in creation order • Export data to a binary/ascii file without having to open the object in the TableView HDF Briefing to NASA

  38. future directions HDF Briefing to NASA

  39. General maintenance for 2013-2014 • Document and improve maintenance processes and procedures • Focus on documentation for HDF4 and HDF5 • Improve quality • Establish code review procedures for all HDF products • Expand static analysis to all HDF software packages • Enhance testing framework HDF Briefing to NASA

  40. Future Maintenance Releases CF – code freeze 6 weeks for HDF5 4 weeks for other products HDF Briefing to NASA

  41. Future HDF4 maintenance releases • HDF 4.2.9 planned for February 2013 • Close as many issues as possible • HDF 4.2.10 planned for February 2014 • Complete “self-configuration” work for library maintainability and portability • ESDIS priorities? HDF Briefing to NASA

  42. Future HDF5 maintenance releases • HDF5 1.8.11 May 2013 • Features sponsored by European Synchrotron Community (PSI/DECTRIS and DESY) • Writing compressed chunks directly to HDF5 file bypassing library internals • Finished prototype; achieved order of magnitude boost in I/O • Support for dynamically loaded third party libraries (compression and other filters) • Preliminary studies soon; several prototypes are already available HDF Briefing to NASA

  43. Future HDF5 maintenance releases • How soon will MATLAB and IDL be able to support 1.8.11? • ESDIS priorities? HDF Briefing to NASA

  44. Future h4h5tools maintenance releases • 2.2.2 release March 2013 • Use latest HDF4 and HDF5 releases • Address any ESDIS issues if arise • 2.2.3 release March 2014 • Possible improvements • Conversion of dimension scales • Leveraging HDF5 1.8 features • ESDIS priorities? HDF Briefing to NASA

  45. Future Java maintenance releases • 2.10 release December 2013 • We are gathering requirements (next slides) HDF Briefing to NASA

  46. Tools update HDF Briefing to NASA

  47. HDFView User survey HDF Briefing to NASA

  48. Background • HDFView was created for browsing and editing HDF files on a local machine • The most popular HDF tool • August 2012 downloads • HDFView – 2,805 • HDF5 binaries for all platforms – 1,073 • Twelve releases since the first release in 1998 • Time to reevaluate its capabilities and find out users’ expectations HDF Briefing to NASA

  49. HDFView Survey Goals • Understand users needs and future directions for of HDFView maintenance and evolution • New requirements • Current deficiencies • Most desired improvements • Help with tasks prioritization HDF Briefing to NASA

  50. HDFView Survey • Conducted a survey at https://www.surveymonkey.com/s/PLGRLJY • Over 40 questions in 8 different areas • About 200 responses HDF Briefing to NASA

More Related