HDF5: State of the Union

1. HDF5: State of the Union Mohamad Chaarawi chaarawi@hdfgroup.org The HDF Group

2. What is HDF5? A versatile data model that can represent very complex data objects and a wide variety of metadata. A completely portable file format with no limit on the number or size of data objects stored. An open source software library that runs on a wide range of computational platforms, from cell phones to massively parallel systems, and implements a high-level API with C, C++, Fortran 90, and Java interfaces. A rich set of integrated performance features that allow for access time and storage space optimizations. Tools and applications for managing, manipulating, viewing, and analyzing the data in the collection.

3. HDF5 Technology Platform • HDF5 Abstract Data Model • Defines the “building blocks” for data organization and specification • Files, Groups, Links, Datasets, Attributes, Datatypes, Dataspaces • HDF5 Software • Tools • Language Interfaces • HDF5 Library • HDF5 Binary File Format • Bit-level organization of HDF5 file • Defined by HDF5 File Format Specification

4. HDF5 Data Model • Groups – provide structure among objects • Datasets – where the primary data goes • Data arrays • Rich set of datatype options • Flexible, efficient storage and I/O • Attributes, for metadata Everything else is built essentially from these parts.

5. “Groups” 3-D array lat | lon | temp ----|-----|----- 12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6 Table palette Raster image Raster image 2-D array “Datasets” Structures to organize objects “/”(root) “/TestData”

6. Why use HDF5? • Challenging data: • Application data that pushes the limits of what can be addressed by traditional database systems, XML documents, or in-house data formats. • Software solutions: • For very large datasets, very fast access requirements, or very complex datasets. • To easily share data across a wide variety of computational platforms using applications written in different programming languages. • That take advantage of the many open-source and commercial tools that understand HDF5. • Enabling long-term preservation of data.

7. Who uses HDF5? • Examples of HDF5 user communities • Astrophysics • Astronomers • NASA Earth Science Enterprise • Dept. of Energy Labs • Supercomputing centers in US, Europe and Asia • Financial Institutions • NOAA • Manufacturing industries • Many others • For a more detailed list, visit • http://www.hdfgroup.org/HDF5/users5.html

8. Topics

9. Brief History of HDF 1987 At NCSA (University of Illinois), a task force formed to create an architecture-independent format and library: AEHOO (All Encompassing Hierarchical Object Oriented format) Became HDF Early NASA adopted HDF for Earth Observing System project 1990’s 1996 DOE’s ASC (Advanced Simulation and Computing) Project began collaborating with the HDF group (NCSA) to create “Big HDF” (Increase in computing power of DOE systems at LLNL, LANL and Sandia National labs, required bigger, more complex data files). “Big HDF” became HDF5. 1998 HDF5 was released with support from DOE Labs, NASA, NCSA 2006 The HDF Group spun off from University of Illinois as non-profit corporation

10. The HDF Group • Established in 1988 • 18 years at University of Illinois’ National Center for Supercomputing Applications • 8 years as independent non-profit company, “The HDF Group” • The HDF Group owns HDF4 and HDF5 • HDF4 & HDF5 formats, libraries, and tools are open source and freely available with BSD-style license • Currently employ 37 FTEs • Looking for more developers now!

11. The HDF Group Mission To ensure long-term accessibility of HDF data through sustainable development and support of HDF technologies.

12. Goals of The HDF Group Maintain and evolve HDF for sponsors and communities that depend on it Provide support to the HDF communities through consulting, training, tuning, development, research Sustain the company for the long term to assure data access over time

13. The HDF Group Services • Helpdesk and Mailing Lists • Available to all users as a first level of support: help@hdfgroup.org, hdf-forum@lists.hdfgroup.org • Priority Support • Rapid issue resolution and advice • Consulting • Needs assessment, troubleshooting, design reviews, etc. • Training • Tutorials and hands-on practical experience • Enterprise Support • Coordinating HDF activities across departments • Special Projects • Adapting customer applications to HDF • New features and tools • Research and Development

14. Members of the HDF support community • NASA – Earth Observing System • NASA – CGNS Project • DOE – ExascaleFastForwardw/Intel • DOE – projects w/LBNL, ANL, LLNL, SNL • NSF – projects w/CH2M Hill, Consortium for Ocean Leadership, Arizona Geological Survey • ITER – project with General Atomics • A leading U.S. aerospace company • European Light Sources - Dectris, Diamond Light Source and ESRF • Priority Support Customers • “In kind” Support

15. Income Profile – 2013/2014 Total Income: $4.1 million

16. Heavily used at Supercomputer Centers

17. Recent Success Story • Performance of VPIC-IO on Bluewaters • I/O Kernel of a Plasma Physics application • 56 GB/s I/O rate in writing 5TB data using 5K cores with multi-dataset write optimization • VPIC-IO kernel running on 298,048 cores • ~10 Trillion particles • 291 TB, single file • 1 GB stripe size and 160 Lustre OSTs • 52 GB/s • 53% of the peak performance

18. Topics

19. Where We’ve Been • Release 1.0 • First “prototype” release in Oct, 1997 • Incorporated core data model: datatypes, dataspaces & datasets and groups • Parallel support added in r1.0.1, in Jan, 1999 • Release 1.2.0 - Oct, 1999 • Added support for bitfield, opaque, enumeration, variable-length and reference datatypes. • Added new ‘h5toh4’ tool • Lots of polishing • Performance optimizations

20. Where We’ve Been • Release 1.4.0 - Feb, 2001 • Added Virtual File Driver (VFD) API layer, with many drivers • Added ‘h4toh5’, h5cc tools, XML output to h5dump • Added array datatype • F90 & C++ API wrappers • Performance optimizations • Release 1.6.0 - July, 2003 • Generic Property API • Compact dataset storage • Added ‘h5diff’, ‘h5repack’, ‘h5jam’, ‘h5import’ tools • Performance optimizations

21. Where We’re At Now • Release 1.8.0 - Feb, 2008 • Features to support netCDF-4 • Creation order indexing on links and attributes • Integer-to-floating point conversion support • NULL dataspace • More efficient group storage • External Links • New H5L (links) and H5O (objects) APIs • Shared Object Header Messages • Unicode-8 Support • Anonymous object creation • New tools: ‘h5mkgrp’, ‘h5stat’, ‘h5copy’ • CMake build support • Performance optimizations

22. Repository Statistics HDF5 Library Source Code, Lines of Code by Date

23. Repository Statistics HDF5 Library Source Code, Lines of Code by Release

24. Software Engineering in HDF5 • We spend considerable effort in ensuring we produce very high quality code for HDF5. • Current efforts: • Correctness regression testing • Nightly testing of >60 configurations on >20 machines • Performance regression testing • Applying static code analysis – Coverity, Klocwork • Memory leak detection – valgrind • Code coverage – coming soon

25. Performance Regression Test Suite

26. Where We’ll Be Soon • Release 1.10 - Overview • Beta release ~ mid 2015 • Stopped adding major features, fleshing out our current efforts now • Major Efforts: • Improved scalability of chunked dataset access • Single-Writer/Multiple Reader (SWMR) Access • Improved fault tolerance • Initial support for asynchronous I/O • Virtual Object Layer

27. Where We’ll Be Soon • Release 1.10 - Details • New chunked dataset indexing methods • Single-Writer/Multiple-Reader (SWMR) Access • Improved Fault Tolerance • Journaled Metadata Writing • Ordered Updates • Page-aligned and buffered metadata access • Persistent file free space tracking • F2003 Support • Compressed group information • High-level “HPC” API • Collective I/O on multiple datasets • Metadata broadcast between processes • Performance optimizations • Avoid file truncation

28. Where We Might Get To • Release 1.10 - Maybe? • Full C99 type support (long double, complex, boolean types, etc) • Support for runtime file format limits • Improved variable-length datatype storage • Basic support for asynchronous I/O • Expanded Virtual File Driver (VFD) interface • Lazy metadata writes (in serial) • Virtual Object Layer • Abstraction layer to allow HDF5 objects to be stored in any container • Allows application to break “all collective” metadata modification limit in current HDF5

29. Where We’re Not Going • We’re not changing multi-threaded concurrency support • Keep “global lock” on library • Will use asynchronous I/O heavily • Will be using threads internally though

30. Topics

31. Tool activities in the works • New tool – h5watch • Display changes to a dataset, metadata and raw data • Improved code quality and testing • Tools library: general purpose APIs for tools • Tools library currently only for our developers • Want to make it public so that people can use it in their products

32. Topics

33. HDF5 in the Future “The best way to predict the future is to invent it.” – Alan Kay

34. Plans, Guesses & Speculations: HPC • Improve Parallel I/O Performance: • Continue to improve our use of MPI and parallel file system features • Reduce # of I/O accesses for metadata access • Collective Read/Write of metadata • Support for compression in parallel • Collective access mode only • Support asynchronous parallel I/O • Support Single-Write/Multiple-Reader (SWMR) access in parallel

35. Plans, Guesses & Speculations: Safety • Improve Journaled HDF5 File Access: • Journal raw data operations • Allow multi-operation journal transactions to be created by applications • Support fully asynchronous journal operations • Enable journaling for Parallel HDF5

36. Plans, Guesses & Speculations: Threadsafety • Focus on asynchronous I/O, instead of improving multi-threaded concurrency of HDF5 library: • Library currently thread-safe, but not concurrent • Instead of improving concurrency, focus on heavily leveraging asynchronous I/O • Use internal multi-threading where possible

37. Plans, Guesses & Speculations: Grab-bag • Improve data model: • Shared dataspaces • Attributes on dataspaces and datatypes • Improve raw data chunk cache implementation • More efficient storage and I/O of variable-length data, including compression

38. Topics

39. Virtual Object Layer • The VFL is implemented below the HDF5 abstract model: • deals with blocks of bytes in the storage container • does not recognize HDF5 objects nor abstract operations on those objects. • The VOL would be layered right below the API to capture the HDF5 model:

40. ExascaleFastForwardResearch • Intel, EMC & The HDF Group recently completed a research project for exascale storage research and prototyping: • https://wiki.hpdd.intel.com/display/PUB/Fast+Forward+Storage+and+IO+Program+Documents • Using HDF5 data model and interface as top layer of next generation storage system for future exascale systems • Laundry list of new features to prototype in HDF5 • Asynchronous I/O, transactions, end-to-end consistency checking, query/index features, Analysis Shipping, python wrappers, etc.

41. Autotuning and Performance Tracing • Why? • Because the dominant I/O support request at NERSC is poor I/O performance, many/most of which can be solved by enabling Lustre striping, or tuning another I/O parameter • Scientists shouldn’t have to figure this stuff out! • Three Areas of Focus: • Evaluate techniques for autotuning HPC application I/O • File system, MPI, HDF5 • I/O Modeling to narrow down the autotuning space. • Record and Replay HDF5 I/O operations

42. Autotuning HPC I/O • Goal: Avoid tuning each application to each machine and file system • Create I/O autotuner library that can inject “optimal” parameters for I/O operations on a given system • Apply to precompiled application binaries • Application can be dynamically linked with I/O autotuning library • No changes to application or HDF5 library • Tested with several HPC applications already: • VPIC, GCRM, Vorpal • Up to 100x performance improvement, compared to system default settings

43. References [1] "Taming Parallel I/O Complexity with Auto-Tuning", Babak Behzad, Huong Vu ThanhLuu, Joseph Huchette, SurendraByna, Prabhat, Ruth Aydt, Quincey Koziol, Marc Snir. Supercomputing (SC) 2013 [2] "Improving parallel I/O autotuning with performance modeling", Babak Behzad, Suren Byna, Stefan Wild, Prabhat, and Marc Snir. International Symposium on High-Performance Parallel and Distributed Computing (HPDC) 2014

44. Recording and Replaying HDF5 • Goal: • Extract an “I/O Kernel” from application, without examining application code • Method: • Dynamically link library at run-time to record all HDF5 calls and parameters in “HDF5 I/O trace file” • Create parallel replay tool that uses recording to replay HDF5 operations

45. Recording and Replaying HDF5 • Benefits: • Easy to create I/O kernel from any application, even one with no source code • Can use replay file as accurate application I/O benchmark • Can move replay file to another system for comparison • Can autotune from replay, instead of application • Challenges: • Accurate parallel replay is difficult • Reference: • "Automatic Generation of I/O Kernels for HPC Applications", Babak Behzad, Hoang-Vu Dang, Farah Hariri, Weizhe Zhang, and Marc Snir. Parallel Data Storage Workshop (PDSW) 2014

46. Codename “HEXAD” * help@hdfgroup.org • Excel is a great frontend with a not so great rear ;-) • We’ve fixed that with an HDF5 Excel Add-in • Let’s you do the usual things including: • Display content (file structure, detailed object info) • Create/read/write datasets • Create/read/update attributes • Plenty of ideas for bells an whistles, e.g., HDF5 image & PyTables support • Send in* your Must Have/Nice To Have feature list! • Stay tuned for the beta program

47. Restless About HDF5/REST

48. HDF Server REST-based service for HDF5 data Reference Implementation for REST API Developed in Python using Tornado Framework Supports Read/Write operations Clients can be Python/C/Fortran or Web Page Let us know what specific features you’d like to see. E.g. VOL REST Client Plugin

49. HDF Server Architecture

50. HDF Compass “Simple” HDF5 Viewer application Cross platform (Windows/Mac/Linux) Native look and feel Can display extremely large HDF5 files View HDF5 files and OpenDAP resources Plugin model enables different file formats/remote resources to be supported Community-based development model