1 / 34

Moving applications to HDF5 1.8.0

Moving applications to HDF5 1.8.0. Outline. Quick overview of new features Move or not to move? HDF5 APIs versioning HDF5 library configuration HDF5 application configuration Examples. Goal of the tutorial. Raise awareness about forward/backward compatibility issues with the new release

onam
Download Presentation

Moving applications to HDF5 1.8.0

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Moving applications to HDF5 1.8.0 HDF and HDF-EOS Workshop XI

  2. Outline • Quick overview of new features • Move or not to move? • HDF5 APIs versioning • HDF5 library configuration • HDF5 application configuration • Examples HDF and HDF-EOS Workshop XI

  3. Goal of the tutorial • Raise awareness about forward/backward compatibility issues with the new release • Help with transition to the new release • Get feedback from you before official release is done HDF and HDF-EOS Workshop XI

  4. Current status of the HDF5 1.8.0 release • HDF5 1.8.0-beta4 was released on October 22, 2007 • Stable file format and APIs • Has been successfully tested with NetCDF-4 and HDF-EOS5 on many supported platforms • Tasks left before the full release • Bugs fixing • Incorporating contributed features • Performance tuning (more will follow) • Documentation • Thorough backward/forward compatibility testing for APIs and files format HDF and HDF-EOS Workshop XI

  5. Quick overview of new features in HDF5 1.8.0 and their affect on current applications HDF and HDF-EOS Workshop XI

  6. HDF5 1.8.0 themes • Better handling of HDF5 metadata • Reduced HDF5 file overhead • Efficient metadata caching • Improved internal structures for better handling of variable-length data • Features to support HDF and NetCDF harmonization • New and improved tools • Numerous bug fixes and improvements • Result: More than 200 new APIs and revised file format HDF and HDF-EOS Workshop XI

  7. Quick overview of the new features • Complete list can be found at http://www.hdfgroup.uiuc.edu/HDF5/doc_1.8pre/WhatsNew180.html • Two types of features • Features that did not require file format and APIs changes • Features that did require file format changes HDF and HDF-EOS Workshop XI

  8. Quick overview of the new features • Features that did not require file format changes and changes to existing APIs • Improved metadata caching • New filters (scale+offset, n-bit, arithmetic transformation) • Improved error handling mechanism • Support for dimension scales, packet tables and other high-level APIs • Affect on applications: • Applications can be extended (calls added) to use new features • There is no need to modify existing calls • Created files are compatible with the earlier versions of software HDF and HDF-EOS Workshop XI

  9. Quick overview of the new features • Features that did require file format changes and changes to existing APIs • Creation order of the objects (groups, datasets, attributes) • Support for UTF-8 encoding for strings that describe links to the objects and names of the attributes • Configurable compact or indexed storage for groups • Shared object headers, external links • New file format features to reduce files overhead • Affect on applications: • Applications have to be modified to use new features • Created files are NOT compatible with the earlier versions of software HDF and HDF-EOS Workshop XI

  10. To move or not to move? HDF and HDF-EOS Workshop XI

  11. Why applications should move to HDF5 1.8.0? • My application doesn’t need any new features, why should I consider moving to HDF5 1.8.0? • Take advantage of improved library • Bug fixes (e.g. corrupted files) • Better metadata cache performance • Smaller memory footprint • Support for big attributes, new compression filters, datatype conversion, etc. • Availability of new platforms like Windows 64-bit • Better support for different Fortran compilers • Improved tools • First step in taking advantage of the new features in the future HDF and HDF-EOS Workshop XI

  12. Why applications may not move to HDF5 1.8.0? • May be expensive • Moving to the new release may require • Source code modification • Building scripts modification • No new features are desired HDF and HDF-EOS Workshop XI

  13. Making decision…. • I would like to use new features, how much work it would be? • I would like to use new library but I cannot modify my application, can I do it? • The rest of the talk is about the ways to move applications to the new release without doing much work HDF and HDF-EOS Workshop XI

  14. Introduction to HDF5 APIs versioning HDF and HDF-EOS Workshop XI

  15. Example: Four ways to create a group • Old way H5Gcreate ( loc_id, “New/My old group”, 0 ) H5Gcreate1( loc_id, “New/My old group”, 0 ) • No new features can be invoked • New way H5Gcreate ( loc_id, “New/My new group”, lcpl_id, gcpl_id, gapl_id) H5Gcreate2( loc_id, “New/My new group”, lcpl_id, gcpl_id, gapl_id) • New features can be invoked • Creation order • Unicode names • Compact storage • Intermediate group creation HDF and HDF-EOS Workshop XI

  16. New APIs in HDF5 1.8.0 • Examples of the new APIs • H5Acreate1, H5Aopen1 • H5Acreate2, H5Aopen2 • H5Gcreate1, H5Gopen1 • H5Gcreate2, H5Gopen2, H5G_link_hard • H5Rget_obj_type1 • H5Rget_obj_type2 • New APIs have more parameters to set up creation and access properties for the objects • Default values for new parameters H5P_DEFAULT will emulate old behavior H5Gcreate ( loc_id, “New/My old group”, 0 ) H5Gcreate ( loc_id, “New/My new group”, H5P_DEFAULT,H5P_DEFAULT, H5P_DEFAULT) HDF and HDF-EOS Workshop XI

  17. Removed APIs in HDF5 1.8.0 • Examples of the removed APIs • H5Acreate, H5Aopen • H5Gcreate, H5Gopen • H5Rget_obj_type • Still can be called by using API mapping and macros H5Gcreate ( loc_id, “New/My old group”, 0 ) HDF and HDF-EOS Workshop XI

  18. Deprecated APIs in HDF5 1.8.0 • Include both public and private APIs • Functionality is replaced by new functions • See files in the src directory • H5Adeprec.c H5Ddeprec.c H5Edeprec.c H5Gdeprec.c H5Rdeprec.c H5Tdeprec.c • Examples of deprecated APIs • H5Acreate1, H5Aopen_name • H5Gcreate1, H5Gopen1, H5G_link_hard • H5Rget_obj_type1 • For complete list see Changes.html file, “API Compatibility Macros in HDF5” • Deprecated APIs will be removed with caution (in some cases will not be removed at all) HDF and HDF-EOS Workshop XI

  19. HDF5 Macros for API compatibility Name: H5Gcreate Signature: hid_t H5Gcreate( hid_t loc_id, const char *name, size_t size_hint ) hid_t H5Gcreate( hid_t loc_id, const char *name, hid_t lcpl_id, hid_t gcpl_id, hid_t gapl_id Purpose: Creates a new empty group and links it to a location in the file. Description: H5Gcreate is a macro that is mapped to either H5Gcreate1 or H5Gcreate2, depending on the needs ofthe application. HDF and HDF-EOS Workshop XI

  20. HDF5 Macros for API compatibility • Defined in the H5version.h file Example of the H5Gcreate macro #ifdef H5_USE_16_API … #if !defined(H5Gcreate_vers) #define H5Gcreate_vers 1 #endif /* !defined(H5Gcreate_vers) */ … #if !defined(H5Gcreate_vers) || H5Gcreate_vers == 2 #ifndef H5Gcreate_vers #define H5Gcreate_vers 2 #endif /* H5Gcreate_vers */ #define H5Gcreate H5Gcreate2 #elif H5Gcreate_vers == 1 #define H5Gcreate H5Gcreate1 #else /* H5Gcreate_vers */ #error "H5Gcreate_vers set to invalid value" #endif /* H5Gcreate_vers */ … HDF and HDF-EOS Workshop XI

  21. HDF5 Library configuration • Mapping of APIs is set at configuration time • Several configuration flags are provided to enable different mappings • Macro name is mapped to old API, e.g. H5Gcreate is mapped to H5Gcreate1 • Macro name is mapped to new API, e.g. H5Gcreate is mapped to H5Gcreate2 (will have different parameters from 1.6 H5Gcreate) • Disable old APIs completely, e.g. H5Gcreate1 is not available in the library HDF and HDF-EOS Workshop XI

  22. HDF5 Library configuration HDF and HDF-EOS Workshop XI

  23. HDF5 Library configuration • Library configuration information is available in the libhdf5.settings file • Examples ……. Default Version of Public Symbols: v18 With Deprecated Public Symbols: Yes ……. Default Version of Public Symbols: v16 With Deprecated Public Symbols: Yes ……. Default Version of Public Symbols: v18 With Deprecated Public Symbols: No HDF and HDF-EOS Workshop XI

  24. HDF5 Application configuration options Global mapping • HDF5 APIs mapping can be done by application • Assuming both deprecated and new symbols are available in the library: h5cc my_program.c • Both H5Gcreate1, H5Gcreate2 and H5Gcreate may be used h5cc -DH5_NO_DEPRECATED_SYMBOLS my_program.c • Only new symbols are available for application; H5Gcreate is mapped to H5Gcreate2; application may use both H5Gcreate2 and H5Gcreate; cannot use H5Gcreate1 h5cc -DH5_USE_16_API my_program.c • H5Gcreate is mapped to H5Gcreate1; all three H5Gcreate1, H5Gcreate2 and H5Gcreate can be used HDF and HDF-EOS Workshop XI

  25. HDF5 Application configuration options Per-function scale • Version and mapping can be set per function • Assuming both deprecated and new symbols are available in the library: h5cc -D H5Gcreate_vers=1 -D H5Acreate_vers=2 my_program.c • Maps H5Gcreate to H5Gcreate1 • Maps H5Acreate to H5Acreate2 • both H5Gcreate1 and H5Gcreate2 may be used; the same for H5Acreate1 and H5Acreate2 h5cc -D H5Gcreate_vers=2 my_program.c • Maps H5Gcreate to H5Gcreate2 • Both H5Gcreate1 and H5Gcreate2 may be used HDF and HDF-EOS Workshop XI

  26. HDF5 Fortran and C++ applications • HDF5 Fortran and C++ libraries use new APIs H5<name>2 using default parameters for new arguments • Backward/forward compatible • New features are not available yet HDF and HDF-EOS Workshop XI

  27. Example: --with-default-api-version=v18 hid_t file_id, group_id; /* identifiers */ ... /* Open “file.h5” */ file_id = H5Fopen(“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT, H5P_DEFAULT); /* Create several groups in a file */ grp1_id = H5Gcreate (file_id, ”New/larry", H5P_DEAFULT, gcpt, gapt); grp2_id = H5Gcreate1(file_id,"/larry",0); … grp3_id = H5Gcreate2(file_id,”New/larry", H5P_DEAFULT, gcpt, gapt); HDF and HDF-EOS Workshop XI

  28. Example: --with-default-api-version=v16 hid_t file_id, group_id; /* identifiers */ ... /* Open “file.h5” */ file_id = H5Fopen(“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT, H5P_DEFAULT); /* Create several groups in a file */ grp1_id = H5Gcreate (file_id, "/larry",0); grp2_id = H5Gcreate1(file_id,"/larry1",0); grp3_id = H5Gcreate2(file_id,”New/larry", H5P_DEAFULT, gcpt, gapt); HDF and HDF-EOS Workshop XI

  29. Example: --disable-deprecated-symbols hid_t file_id, group_id; /* identifiers */ ... /* Open “file.h5” */ file_id = H5Fopen(“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT, H5P_DEFAULT); /* Create several groups in a file */ grp1_id = H5Gcreate (file_id, ”New/larry", H5P_DEAFULT, gcpt, gapt); /* Compilation will fail */ grp2_id = H5Gcreate1(file_id,"/larry",0); grp3_id = H5Gcreate2(file_id,”New/larry", H5P_DEAFULT, gcpt, gapt); HDF and HDF-EOS Workshop XI

  30. Example: new version of the file format hid_t file_id, group_id; /* identifiers */ ... /* Open “file.h5” and create objects using new format*/ fapl = H5Pcreate(H5P_FILE_ACCESS); ret = H5Pset_latest_format(fapl, 1); fid = H5Fcreate(“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT, fapl); /* Compose group creation property list */ gcpl = H5Pcreate(H5P_GROUP_CREATE); ret = H5Pset_link_creation_order(gcpl, (H5P_CRT_ORDER_TRACKED | H5P_CRT_ORDER_INDEXED)); /* Create a group */ gid = H5Gcreate(fid, GNAME, H5P_DEFAULT, gcpl, H5P_DEFAULT); HDF and HDF-EOS Workshop XI

  31. Example: h5dump --group=1 tordergr.h5 HDF5 "tordergr.h5" { GROUP "1" { GROUP "a" { GROUP "a1" { } GROUP "a2" { GROUP "a21" { } GROUP "a22" { } } } GROUP "b" { } GROUP "c" { } } } HDF and HDF-EOS Workshop XI

  32. Example: h5dump … --sort_by=creation_order … tordergr.h5 HDF5 "tordergr.h5" { GROUP "1" { GROUP "c" { } GROUP "b" { } GROUP "a" { GROUP "a1" { } GROUP "a2" { GROUP "a22" { } GROUP "a21" { } } } } } HDF and HDF-EOS Workshop XI

  33. Example: creation of intermediate groups /* Create link creation property list */ lcr_id = H5Pcreate(H5P_LINK_CREATE); /* Set flag for intermediate group creation */ status = H5Pset_create_intermediate_group(lcr_id, TRUE); grp_id = H5Gcreate (file_id, "/A/B/C", lcr_id, H5P_DEFAULT, access_id); Groups A and B will be created automatically HDF and HDF-EOS Workshop XI

  34. Thank you! Questions ? HDF and HDF-EOS Workshop XI

More Related