1 / 53

HDF5: State of the Union

HDF5: State of the Union. Mohamad Chaarawi chaarawi@hdfgroup.org The HDF Group. What is HDF5?. A versatile data model that can represent very complex data objects and a wide variety of metadata. A completely portable file format with no limit on the number or size of data objects stored.

lani
Download Presentation

HDF5: State of the Union

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. HDF5: State of the Union Mohamad Chaarawi chaarawi@hdfgroup.org The HDF Group

  2. What is HDF5? A versatile data model that can represent very complex data objects and a wide variety of metadata. A completely portable file format with no limit on the number or size of data objects stored. An open source software library that runs on a wide range of computational platforms, from cell phones to massively parallel systems, and implements a high-level API with C, C++, Fortran 90, and Java interfaces. A rich set of integrated performance features that allow for access time and storage space optimizations. Tools and applications for managing, manipulating, viewing, and analyzing the data in the collection.

  3. HDF5 Technology Platform • HDF5 Abstract Data Model • Defines the “building blocks” for data organization and specification • Files, Groups, Links, Datasets, Attributes, Datatypes, Dataspaces • HDF5 Software • Tools • Language Interfaces • HDF5 Library • HDF5 Binary File Format • Bit-level organization of HDF5 file • Defined by HDF5 File Format Specification

  4. HDF5 Data Model • Groups – provide structure among objects • Datasets – where the primary data goes • Data arrays • Rich set of datatype options • Flexible, efficient storage and I/O • Attributes, for metadata Everything else is built essentially from these parts.

  5. “Groups” 3-D array lat | lon | temp ----|-----|----- 12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6 Table palette Raster image Raster image 2-D array “Datasets” Structures to organize objects “/”(root) “/TestData”

  6. Why use HDF5? • Challenging data: • Application data that pushes the limits of what can be addressed by traditional database systems, XML documents, or in-house data formats. • Software solutions: • For very large datasets, very fast access requirements, or very complex datasets. • To easily share data across a wide variety of computational platforms using applications written in different programming languages. • That take advantage of the many open-source and commercial tools that understand HDF5. • Enabling long-term preservation of data.

  7. Who uses HDF5? • Examples of HDF5 user communities • Astrophysics • Astronomers • NASA Earth Science Enterprise • Dept. of Energy Labs • Supercomputing centers in US, Europe and Asia • Financial Institutions • NOAA • Manufacturing industries • Many others • For a more detailed list, visit • http://www.hdfgroup.org/HDF5/users5.html

  8. Topics

  9. Brief History of HDF 1987 At NCSA (University of Illinois), a task force formed to create an architecture-independent format and library: AEHOO (All Encompassing Hierarchical Object Oriented format) Became HDF Early NASA adopted HDF for Earth Observing System project 1990’s 1996 DOE’s ASC (Advanced Simulation and Computing) Project began collaborating with the HDF group (NCSA) to create “Big HDF” (Increase in computing power of DOE systems at LLNL, LANL and Sandia National labs, required bigger, more complex data files). “Big HDF” became HDF5. 1998 HDF5 was released with support from DOE Labs, NASA, NCSA 2006 The HDF Group spun off from University of Illinois as non-profit corporation

  10. The HDF Group • Established in 1988 • 18 years at University of Illinois’ National Center for Supercomputing Applications • 8 years as independent non-profit company, “The HDF Group” • The HDF Group owns HDF4 and HDF5 • HDF4 & HDF5 formats, libraries, and tools are open source and freely available with BSD-style license • Currently employ 37 FTEs • Looking for more developers now!

  11. The HDF Group Mission To ensure long-term accessibility of HDF data through sustainable development and support of HDF technologies.

  12. Goals of The HDF Group Maintain and evolve HDF for sponsors and communities that depend on it Provide support to the HDF communities through consulting, training, tuning, development, research Sustain the company for the long term to assure data access over time

  13. The HDF Group Services • Helpdesk and Mailing Lists • Available to all users as a first level of support: help@hdfgroup.org, hdf-forum@lists.hdfgroup.org • Priority Support • Rapid issue resolution and advice • Consulting • Needs assessment, troubleshooting, design reviews, etc. • Training • Tutorials and hands-on practical experience • Enterprise Support • Coordinating HDF activities across departments • Special Projects • Adapting customer applications to HDF • New features and tools • Research and Development

  14. Members of the HDF support community • NASA – Earth Observing System • NASA – CGNS Project • DOE – ExascaleFastForwardw/Intel • DOE – projects w/LBNL, ANL, LLNL, SNL • NSF – projects w/CH2M Hill, Consortium for Ocean Leadership, Arizona Geological Survey • ITER – project with General Atomics • A leading U.S. aerospace company • European Light Sources - Dectris, Diamond Light Source and ESRF • Priority Support Customers • “In kind” Support

  15. Income Profile – 2013/2014 Total Income: $4.1 million

  16. Heavily used at Supercomputer Centers

  17. Recent Success Story • Performance of VPIC-IO on Bluewaters • I/O Kernel of a Plasma Physics application • 56 GB/s I/O rate in writing 5TB data using 5K cores with multi-dataset write optimization • VPIC-IO kernel running on 298,048 cores • ~10 Trillion particles • 291 TB, single file • 1 GB stripe size and 160 Lustre OSTs • 52 GB/s • 53% of the peak performance

  18. Topics

  19. Where We’ve Been • Release 1.0 • First “prototype” release in Oct, 1997 • Incorporated core data model: datatypes, dataspaces & datasets and groups • Parallel support added in r1.0.1, in Jan, 1999 • Release 1.2.0 - Oct, 1999 • Added support for bitfield, opaque, enumeration, variable-length and reference datatypes. • Added new ‘h5toh4’ tool • Lots of polishing • Performance optimizations

  20. Where We’ve Been • Release 1.4.0 - Feb, 2001 • Added Virtual File Driver (VFD) API layer, with many drivers • Added ‘h4toh5’, h5cc tools, XML output to h5dump • Added array datatype • F90 & C++ API wrappers • Performance optimizations • Release 1.6.0 - July, 2003 • Generic Property API • Compact dataset storage • Added ‘h5diff’, ‘h5repack’, ‘h5jam’, ‘h5import’ tools • Performance optimizations

  21. Where We’re At Now • Release 1.8.0 - Feb, 2008 • Features to support netCDF-4 • Creation order indexing on links and attributes • Integer-to-floating point conversion support • NULL dataspace • More efficient group storage • External Links • New H5L (links) and H5O (objects) APIs • Shared Object Header Messages • Unicode-8 Support • Anonymous object creation • New tools: ‘h5mkgrp’, ‘h5stat’, ‘h5copy’ • CMake build support • Performance optimizations

  22. Repository Statistics HDF5 Library Source Code, Lines of Code by Date

  23. Repository Statistics HDF5 Library Source Code, Lines of Code by Release

  24. Software Engineering in HDF5 • We spend considerable effort in ensuring we produce very high quality code for HDF5. • Current efforts: • Correctness regression testing • Nightly testing of >60 configurations on >20 machines • Performance regression testing • Applying static code analysis – Coverity, Klocwork • Memory leak detection – valgrind • Code coverage – coming soon

  25. Performance Regression Test Suite

  26. Where We’ll Be Soon • Release 1.10 - Overview • Beta release ~ mid 2015 • Stopped adding major features, fleshing out our current efforts now • Major Efforts: • Improved scalability of chunked dataset access • Single-Writer/Multiple Reader (SWMR) Access • Improved fault tolerance • Initial support for asynchronous I/O • Virtual Object Layer

  27. Where We’ll Be Soon • Release 1.10 - Details • New chunked dataset indexing methods • Single-Writer/Multiple-Reader (SWMR) Access • Improved Fault Tolerance • Journaled Metadata Writing • Ordered Updates • Page-aligned and buffered metadata access • Persistent file free space tracking • F2003 Support • Compressed group information • High-level “HPC” API • Collective I/O on multiple datasets • Metadata broadcast between processes • Performance optimizations • Avoid file truncation

  28. Where We Might Get To • Release 1.10 - Maybe? • Full C99 type support (long double, complex, boolean types, etc) • Support for runtime file format limits • Improved variable-length datatype storage • Basic support for asynchronous I/O • Expanded Virtual File Driver (VFD) interface • Lazy metadata writes (in serial) • Virtual Object Layer • Abstraction layer to allow HDF5 objects to be stored in any container • Allows application to break “all collective” metadata modification limit in current HDF5

  29. Where We’re Not Going • We’re not changing multi-threaded concurrency support • Keep “global lock” on library • Will use asynchronous I/O heavily • Will be using threads internally though

  30. Topics

  31. Tool activities in the works • New tool – h5watch • Display changes to a dataset, metadata and raw data • Improved code quality and testing • Tools library: general purpose APIs for tools • Tools library currently only for our developers • Want to make it public so that people can use it in their products

  32. Topics

  33. HDF5 in the Future “The best way to predict the future is to invent it.” – Alan Kay

  34. Plans, Guesses & Speculations: HPC • Improve Parallel I/O Performance: • Continue to improve our use of MPI and parallel file system features • Reduce # of I/O accesses for metadata access • Collective Read/Write of metadata • Support for compression in parallel • Collective access mode only • Support asynchronous parallel I/O • Support Single-Write/Multiple-Reader (SWMR) access in parallel

  35. Plans, Guesses & Speculations: Safety • Improve Journaled HDF5 File Access: • Journal raw data operations • Allow multi-operation journal transactions to be created by applications • Support fully asynchronous journal operations • Enable journaling for Parallel HDF5

  36. Plans, Guesses & Speculations: Threadsafety • Focus on asynchronous I/O, instead of improving multi-threaded concurrency of HDF5 library: • Library currently thread-safe, but not concurrent • Instead of improving concurrency, focus on heavily leveraging asynchronous I/O • Use internal multi-threading where possible

  37. Plans, Guesses & Speculations: Grab-bag • Improve data model: • Shared dataspaces • Attributes on dataspaces and datatypes • Improve raw data chunk cache implementation • More efficient storage and I/O of variable-length data, including compression

  38. Topics

  39. Virtual Object Layer • The VFL is implemented below the HDF5 abstract model: • deals with blocks of bytes in the storage container • does not recognize HDF5 objects nor abstract operations on those objects. • The VOL would be layered right below the API to capture the HDF5 model:

  40. ExascaleFastForwardResearch • Intel, EMC & The HDF Group recently completed a research project for exascale storage research and prototyping: • https://wiki.hpdd.intel.com/display/PUB/Fast+Forward+Storage+and+IO+Program+Documents • Using HDF5 data model and interface as top layer of next generation storage system for future exascale systems • Laundry list of new features to prototype in HDF5 • Asynchronous I/O, transactions, end-to-end consistency checking, query/index features, Analysis Shipping, python wrappers, etc.

  41. Autotuning and Performance Tracing • Why? • Because the dominant I/O support request at NERSC is poor I/O performance, many/most of which can be solved by enabling Lustre striping, or tuning another I/O parameter • Scientists shouldn’t have to figure this stuff out! • Three Areas of Focus: • Evaluate techniques for autotuning HPC application I/O • File system, MPI, HDF5 • I/O Modeling to narrow down the autotuning space. • Record and Replay HDF5 I/O operations

  42. Autotuning HPC I/O • Goal: Avoid tuning each application to each machine and file system • Create I/O autotuner library that can inject “optimal” parameters for I/O operations on a given system • Apply to precompiled application binaries • Application can be dynamically linked with I/O autotuning library • No changes to application or HDF5 library • Tested with several HPC applications already: • VPIC, GCRM, Vorpal • Up to 100x performance improvement, compared to system default settings

  43. References [1] "Taming Parallel I/O Complexity with Auto-Tuning", Babak Behzad, Huong Vu ThanhLuu, Joseph Huchette, SurendraByna, Prabhat, Ruth Aydt, Quincey Koziol, Marc Snir. Supercomputing (SC) 2013 [2] "Improving parallel I/O autotuning with performance modeling", Babak Behzad, Suren Byna, Stefan Wild, Prabhat, and Marc Snir. International Symposium on High-Performance Parallel and Distributed Computing (HPDC) 2014

  44. Recording and Replaying HDF5 • Goal: • Extract an “I/O Kernel” from application, without examining application code • Method: • Dynamically link library at run-time to record all HDF5 calls and parameters in “HDF5 I/O trace file” • Create parallel replay tool that uses recording to replay HDF5 operations

  45. Recording and Replaying HDF5 • Benefits: • Easy to create I/O kernel from any application, even one with no source code • Can use replay file as accurate application I/O benchmark • Can move replay file to another system for comparison • Can autotune from replay, instead of application • Challenges: • Accurate parallel replay is difficult • Reference: • "Automatic Generation of I/O Kernels for HPC Applications", Babak Behzad, Hoang-Vu Dang, Farah Hariri, Weizhe Zhang, and Marc Snir. Parallel Data Storage Workshop (PDSW) 2014

  46. Codename “HEXAD” * help@hdfgroup.org • Excel is a great frontend with a not so great rear ;-) • We’ve fixed that with an HDF5 Excel Add-in • Let’s you do the usual things including: • Display content (file structure, detailed object info) • Create/read/write datasets • Create/read/update attributes • Plenty of ideas for bells an whistles, e.g., HDF5 image & PyTables support • Send in* your Must Have/Nice To Have feature list! • Stay tuned for the beta program

  47. Restless About HDF5/REST

  48. HDF Server REST-based service for HDF5 data Reference Implementation for REST API Developed in Python using Tornado Framework Supports Read/Write operations Clients can be Python/C/Fortran or Web Page Let us know what specific features you’d like to see. E.g. VOL REST Client Plugin

  49. HDF Server Architecture

  50. HDF Compass “Simple” HDF5 Viewer application Cross platform (Windows/Mac/Linux) Native look and feel Can display extremely large HDF5 files View HDF5 files and OpenDAP resources Plugin model enables different file formats/remote resources to be supported Community-based development model

More Related