1 / 37

HDF5 Tools Update

HDF5 Tools Update. Peter Cao - The HDF Group xcao@hdfgroup.org November 6, 2007.

cahil
Download Presentation

HDF5 Tools Update

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HDF5ToolsUpdate Peter Cao - The HDF Group xcao@hdfgroup.org November 6, 2007 This report is based upon work supported in part by a Cooperative Agreement with NASA under NASA NNX06AC83A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration.

  2. Outline Overview of current tools New tools in HDF5 HDF-Java 2.4 release

  3. HDF5 Command Line Tools Readers • h5dump, h5diff, h5ls • new tools: h5check, h5stat • Writers • h5repack, h5repart, h5import, h5jam/h5unjam • new tool: h5copy, h5mkgrp • Converters • h4toh5, h5toh4, gif2h5, h52gif

  4. Where to start Tools located at bin/ directory • Use “-h” option for help • Online help http://www.hdfgroup.org/hdf5tools.html • Email to THG help desk help@hdfgroup.org

  5. h5dump Dumps file content to stdout in • ASCII text • XML • Binary 4

  6. h5dump -H SDS.h5 HDF5 "SDS.h5" { GROUP "/" { GROUP "Floats" { DATASET "FloatArray" { DATATYPE H5T_IEEE_F32LE DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) } } } DATASET "IntArray" { DATATYPE H5T_STD_I32LE DATASPACE SIMPLE { ( 5, 6 ) / ( 5, 6 ) } } } }

  7. h5dump -d /Floats/FloatArray SDS.h5 HDF5 "SDS.h5" { DATASET "/Floats/FloatArray" { DATATYPE H5T_IEEE_F32LE DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) } DATA { (0,0): 0.01, 0.02, 0.03, (1,0): 0.1, 0.2, 0.3, (2,0): 1, 2, 3, (3,0): 10, 20, 30 } } }

  8. h5dump -x SDS.h5

  9. h5dump Binary Output -b F, --binary=[MEMORY|FILE|LE|BE] h5dump -d /IntArray -o out_le.bin -b LE SDS.h5 000000000000000 00000001 00000002 00000003 00000004 00000005 0000030 0000000a 0000000b 0000000c 0000000d 0000000e 0000000f 0000060 00000014 00000015 00000016 00000017 00000018 00000019 0000110 0000001e 0000001f 00000020 00000021 00000022 00000023 0000140 00000028 00000029 0000002a 0000002b 0000002c 0000002d

  10. h5diff Show Differences Between two objects or two files

  11. h5diff SDS.h5 SDS2.h5 Dataset: </IntArray> and </IntArray> 5 differences found

  12. Report differences Dataset: </IntArray> and </IntArray> position IntArray IntArray difference ------------------------------------------------------ [ 0 0 ] 0 10 10 [ 1 0 ] 10 100 90 [ 2 0 ] 20 200 180 [ 3 0 ] 30 300 270 [ 4 0 ] 40 400 360 5 differences found h5diff SDS.h5 SDS2.h5 -r /IntArray

  13. h5repack Copies a file to a new file with • Compression filter • Data layout • Other user options

  14. h5repack: new filters -f{GZIP|SZIP|SHUF|FLET|NBIT|SOFF|NONE} For example h5repack -i SDS.h5 -o out.h5 -f GZIP=6

  15. h5repack: data layout -l {CHUNK| COMPA| CONTI} For example h5repack -i SDS.h5 -o out.h5 -l CHUNK=2x3

  16. h5repack: latest version of file format The testing file was created with HDF5 1.6.5. The root group of the file contains 100 groups. Each group at the root contains 9 sub groups. Each sub group contains one compound dataset. The compound dataset has 40 members of 32 bit integer

  17. h5repart Repartitions a file into a family of files For example h5repart -m 200m int16kx16k.h5 part200m%d.h5 part200m0.h5 200 MB 977 MB 200 MB part200m1.h5 200 MB part200m2.h5 200 MB part200m3.h5 177 MB part200m4.h5

  18. h5import Imports binary/ASCII data into an HDF5 file • h5import infile -c config_file [infile -c config_file2 ...] -outfile outfile For example h5import float5x4x2.txt -c First_set.conf -o First_set.h5 GROUP "/" { GROUP "work" { DATASET "First-set" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 5, 2, 4 ) / ( 8, 8, H5S_UNLIMITED ) } DATA { (0,0,0): 1.01, 1.02, 1.03, 1.04, (0,1,0): 1.11, 1.12, 1.13, 1.14, (1,0,0): 1.21, 1.22, 1.23, 1.24, (1,1,0): 1.31, 1.32, 1.33, 1.34, (2,0,0): 1.41, 1.42, 1.43, 1.44, … } } } }} PATH work/First-set INPUT-CLASS TEXTFP RANK 3 DIMENSION-SIZES 5 2 4 OUTPUT-CLASS FP OUTPUT-SIZE 64 OUTPUT-ARCHITECTURE IEEE OUTPUT-BYTE-ORDER LE CHUNKED-DIMENSION-SIZES 2 2 2 MAXIMUM-DIMENSIONS 8 8 -1

  19. h5jam/h5unjam • h5jam h5jam -u test_ub.txt -i test_ub.h5 • h5unjam h5unjam -i test_ub.h5 -o out_ub.txt -o out_ub.h5 User Block HDF5 Jam unjam User Block HDF5

  20. h5ls • Lists selected information about file objects in the specified format For Example, h5ls -r SDS2.h5 • /Floats Group • /Floats/DoubleArray Dataset {10, 5} • /Floats/FloatArray Dataset {4, 3} • /Floats/subs Group • /IntArray Dataset {5, 6}

  21. gif2h5 / h52gif gif2h5 – converts a GIF file into HDF5 gif2h5apollo17_earth.gif apollo17_earth.h5 h52gif – converts an HDF5 file into GIF h52gifapollo17_earth.h5 apollo17_earth2.gif -i /apollo17_earth.gif/Image0 -p "/apollo17_earth.gif/Global Palette"

  22. h5toh4 / h4toh5 h5toh4 -- Converts an HDF5 file to an HDF4 file h4toh5 -- Converts an HDF4 file to an HDF5 file http://www.hdfgroup.org/h4toh5/

  23. New tools h5copy h5mkgrp h5stat h5check

  24. h5copy Copies an object within a file/cross files / / Floats IntArray FloatArray FloatArray

  25. h5copy-f shallow / floats -f shallow / f32 64-bit floats integers / i1 i2 f32 64-bit floats f1 f2 f32 64-bit f1 f2

  26. h5copy-f soft / / f1 dset_SL /f1 -f soft f1 dset_SL /f1 / dset_SL /f1

  27. h5copy-f ref / d1 d2 / dset_ref -f ref d2 d1 / dset_ref dset_ref

  28. h5mkgrp Creates groups h5mkgrp test.h5 /info /data /data/test1 /data/test2 / info data test1 test2

  29. h5stat Prints statistics about HDF5 file • Header size • Raw data size • Attributes, datatypes, and etc. Helps • To troubleshoot overhead in HDF5 files • To choose specific object’s properties and storage strategies

  30. h5check A validation tool that verifies if an HDF5 file is encoded according to the HDF5 File Format Specification

  31. Why h5check? • Verify if the file is compliant with the File Format • As a verification tool required by the application of HDF5 File Format to be an ANSI standard • Serves as a watch dog that the HDF5 library implementation is compliant with the File Format

  32. Example: a compliant file • % h5check example1.h5 • VALIDATING example1.h5 • FOUND super block signature • VALIDATING the super block at 0... • VALIDATING the object header at 928... • VALIDATING the btree at 384... • FOUND btree signature. • VALIDATING the local heap at 96... • FOUND local heap signature. • … • Result: File is in compliance.

  33. Example: a non-compliant file h5check invalid2.h5 • FOUND super block signature • VALIDATING the super block at 0... • VALIDATING the object header at 928... • VALIDATING the btree at 384... • FOUND btree signature. • VALIDATING the SNOD at 1248... • FOUND SNOD signature. • VALIDATING the object header at 976... • check_sym(at 1248): Errors from check_obj_header() • decode_validate_messages(): Failure in type->decode(). • H5O_sdspace_decode(): Bad version number in simple dataspace message. • VALIDATING the local heap at 96... • FOUND local heap signature. • Main(): Errors from check_obj_header(). • decode_validate_messages(): Failure in type->decode(). • H5O_attr_decode(): Can't decode attribute dataspace. • H5O_sdspace_decode(): Bad version number in simple dataspace message. • … • Result: File is not in compliance.

  34. Implementation Status Basic file format components 1.6 - DONE 1.8 -IN PROGRESS

  35. HDF-Java 2.4 http://www.hdfgroup.org/hdf-java-html ftp://ftp.hdfgroup.org/HDF5/hdf-java HDF-Java 2.4 Released !!

  36. HDF-Java 2.4 Release • Build with HDF4 2r2 and HDF5 1.6.6 • New platforms Mac intel Linux 64-bit AMD Solaris 64-bit • 36 new enhancements and 44 bugs fixed • Fixed memory leak • Test suite • Enhanced documentation

  37. Memory leak in old version while (true) { H5File f = new H5File (fname, H5File.READ); f.open(); f.close(); }

More Related