HDF Product Designer

1. HDF Product Designer Aleksandar Jelenak, H. Joe Lee, Ted Habermann The HDF Group

2. HDF Data Producer’s Conundrum Interoperability HDF Features • Conventions • Metadata • Software • netCDF • Datatypes • Groups • Attributes • Dimension scales • Compression • Chunking • Scale/offset • Etc. Mission Requirements • Science objectives • Data processing, discovery & distribution • Data documentation • User engagement, preparedness, feedback HDF Product Designer

3. HDF Product Designer Background • Originated from the HDF5 Earth Science (H5ES) Builder/Creator toolset developed by Jeffrey Lee for the ICESat-2 mission • Built using the MySQL, Apache, PHP software stack • Runs as a desktop application • Under review by The HDF Group to generalize the concept

4. What is HDF Product Designer? • Application for organizing HDF5 file content to data and metadata • Built-in support for conventions • Create and re-use parts of file content • Collaborative design process • Generate HDF5 template file, or code that produces the designed file for several programming languages • HDF5 template has structure and metadata

5. Design Process Create and organize content: • Files, groups, datasets, attributes • Many parameters of datasets and attributes can be set • Re-use any part of content elsewhere in the same or another file • All the work is saved in a database

6. After a File is Designed… • Export description of the file’s content into a tab-delimited text file • This description file can be opened with Excel • Content description can be edited • Description file can be imported back into the Product Designer

7. After a File is Designed… • A template HDF5 file can be generated • Code can be produced for reading and writing designed file for: • FORTRAN • IDL • The generated write code can be edited to fill in real data

8. Typical HDF Product Designer Workflow

9. Expected Benefits • Easy way to design HDF file content • Interoperability issues addressed during the design phase • End-to-end data flow testing can start earlier • Reduced risk for costly redesign of processing pipeline based on stakeholder feedback later • Team collaboration • Promotion of design best practices across mission teams

10. Future Work • Local (desktop) or central (web site) app? • Reduce app’s required software footprint • MySQL already replaced with SQLite • Support Python, MATLAB for template file code • Web service for generating HDF5 templates and code • Generate portable/free production code using Python and h5py • Re-use content infofrom old products: • Import HDF4 Content Map.

11. Future Work • Support for compound datatypes • Vdatain HDF4 maps to attribute or dataset? • Convention compliance checkers • CF, HDF-EOS, etc. • Interoperability checkers • netCDF-4, visualization tools, etc.

12. Local vs. Central Application? • Local: • No need for Internet connection • Richer/more responsive user interface • No user accounts management • Requires a developer’s computer • Central: • Only web browser required • Better collaboration workflow • Only one place to update the app • Dual mode – like email software?

13. Looking for Partners • New missions • Projects upgrading from HDF4 to HDF5 • Projects reprocessing their data to increase interoperability • Use cases Contact: eoshelp@hdfgroup.org

14. Acknowledgement This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.