1 / 14

HDF5 and casacore

HDF5 and casacore. Ger van Diepen ASTRON. casacore. C++ library for astronomical data handling Array templated N-dim arrays (STL-conforming) Record dict-like container Table RDBMS-like storage mechanism; TaQL query language

jscofield
Download Presentation

HDF5 and casacore

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HDF5 and casacore Ger van Diepen ASTRON

  2. casacore C++ library for astronomical data handling • Array templated N-dim arrays (STL-conforming) • Record dict-like container • Table RDBMS-like storage mechanism; TaQL query language • MeasurementSet visibility data storage and access (using Tables) • Measures values in frame (direction, position, epoch, ...) • Coordinates world coordinates for images • Images N-dimensional image cubes with 0 or more masks (also supports HDF5, FITS, Miriad, expressions (LEL)) • Used by LOFAR, ASKAP, ALMA, eVLA, MeqTrees, pyrap, pydal • See Download casacore.googlecode.com Classes www.astron.nl/casacore/trunk/casacore/doc/html TaQL www.astron.nl/casacore/trunk/casacore/doc/notes/199.html LEL www.astron.nl/casacore/trunk/casacore/doc/notes/223.html MS definition www.astron.nl/casacore/trunk/casacore/doc/notes/229.html

  3. HDF5 in casacore Simple C++ classes on top of subset of HDF5 API http://www.astron.nl/casacore/trunk/casacore/doc/html/group__HDF5__module.html • Core • HDF5Hid classes • Resource handling (automatically close a hid) • HDF5File • HDF5DataType • Create HDF5 data types for C++ data types • HDF5DataSet • HDF5 dataset of various types • Uses chunking • HDF5Record • Map a Record (dict) to HDF5 groups

  4. HDF5Lattice • http://www.astron.nl/casacore/trunk/casacore/doc/html/classcasa_1_1HDF5Lattice.html • N-dim array in a HDF5DataSet • Part of casacore Lattice/Image framework • Handles iteration, subsetting, etc. • Automatically sizes HDF5 chunk cache given an access pattern • HDF5Image • http://www.astron.nl/casacore/trunk/casacore/doc/html/classcasa_1_1HDF5Image.html • HDF5Lattice with world coordinates • Coordinates stored using HDF5Record • Part of casacore Image framework • PagedImage, FITSImage, MiriadImage, ImageExpression

  5. Performance • tLatticePerf • Create a 3-dim float lattice using casacore Tables or HDF5 on MacBook 2 GB RAM • Iterate by chunk, plane, and line line is done tile-wise to reduce required cache size • Various array and chunk sizes • 128^3 16^3 8 MB • 256^3 16^3 64 MB • 512^3 32^3 0.5 GB • 1024,1024,128 64,64,16 0.5 GB • 2048,2048,64 128,128,4 1 GB • 4096,4096,32 128,128,4 2 GB • 128^3 array with chunk sizes 2, 4, 8, 16, 32, 64, 128 (^3) • Also compared file sizes

  6. 128^3 float; chunk 2^x HDF5CCTS cache HDF5CCTS cache File size (KB) getTile (sec) HDF5CCTS cache more time for very large chunk

  7. getplanexy,xz,yz, getlinex,y,z HDF5CCTS cacheCCTS mmap

  8. 8 MB Creating tLatticePerf_tmp.tab with shape [128, 128, 128] and tile shape [16, 16, 16] CCTS create 0.23 real 0.02 user 0.05 system Creating tLatticePerf_tmp.hdf with shape [128, 128, 128] and tile shape [16, 16, 16] HDF5 create 0.37 real 0.02 user 0.03 system CCTS getTiles 0.03 real 0.01 user 0.02 system HDF5 getTiles 0.02 real 0.02 user 0.01 system CCTS xy getPlane 0.03 real 0.01 user 0.02 system HDF5 xy getPlane 0.06 real 0.06 user 0.01 system CCTS xz getPlane 0.04 real 0.01 user 0.02 system HDF5 xz getPlane 0.08 real 0.07 user 0.01 system CCTS yz getPlane 0.06 real 0.04 user 0.02 system HDF5 yz getPlane 0.17 real 0.16 user 0.01 system CCTS x getLine 0.09 real 0.06 user 0.02 system HDF5 x getLine 0.85 real 0.83 user 0.01 system CCTS y getLine 0.08 real 0.06 user 0.02 system HDF5 y getLine 0.98 real 0.97 user 0.01 system CCTS z getLine 0.08 real 0.07 user 0.02 system HDF5 z getLine 1.16 real 1.03 user 0.01 system

  9. 64 MB Creating tLatticePerf_tmp.tab with shape [256, 256, 256] and tile shape [16, 16, 16] CCTS create 2.2 real 0.13 user 0.33 system Creating tLatticePerf_tmp.hdf with shape [256, 256, 256] and tile shape [16, 16, 16] HDF5 create 2.37 real 0.12 user 0.22 system CCTS getTiles 0.24 real 0.09 user 0.15 system HDF5 getTiles 0.15 real 0.1 user 0.05 system CCTS xy getPlane 0.31 real 0.11 user 0.19 system HDF5 xy getPlane 0.61 real 0.52 user 0.08 system CCTS xz getPlane 0.36 real 0.17 user 0.19 system HDF5 xz getPlane 0.76 real 0.68 user 0.08 system CCTS yz getPlane 0.56 real 0.37 user 0.19 system HDF5 yz getPlane 1.55 real 1.45 user 0.09 system CCTS x getLine 0.5 real 0.34 user 0.16 system HDF5 x getLine 6.61 real 6.53 user 0.06 system CCTS y getLine 0.48 real 0.32 user 0.16 system HDF5 y getLine 7.82 real 7.73 user 0.07 system CCTS z getLine 0.48 real 0.31 user 0.16 system HDF5 z getLine 8.31 real 8.19 user 0.08 system

  10. 0.5 GB Creating tLatticePerf_tmp.tab with shape [512, 512, 512] and tile shape [32, 32, 32] CCTS create 21.31 real 0.75 user 1.46 system Creating tLatticePerf_tmp.hdf with shape [512, 512, 512] and tile shape [32, 32, 32] HDF5 create 27.15 real 0.67 user 0.98 system CCTS getTiles 20.74 real 0.62 user 1.52 system HDF5 getTiles 19.89 real 0.36 user 0.82 system CCTS xy getPlane 19.17 real 0.74 user 1.66 system HDF5 xy getPlane 29.32 real 1.46 user 1.13 system CCTS xz getPlane 22.93 real 1.25 user 1.74 system HDF5 xz getPlane 26.51 real 1.94 user 1.12 system CCTS yz getPlane 62.63 real 6.73 user 2.27 system HDF5 yz getPlane 74.89 real 19.67 user 1.45 system CCTS x getLine 18.58 real 1.79 user 1.57 system HDF5 x getLine 43.91 real 27.01 user 0.92 system CCTS y getLine 46 real 2.41 user 2.18 system HDF5 y getLine 83.2 real 36.41 user 1.52 system CCTS z getLine 42.6 real 2.91 user 2.15 system HDF5 z getLine 84.69 real 41.87 user 1.41 system

  11. 0.5 GB Creating tLatticePerf_tmp.tab with shape [1024, 1024, 128] and tile shape [64, 64, 16] CCTS create 24.28 real 0.77 user 1.44 system Creating tLatticePerf_tmp.hdf with shape [1024, 1024, 128] and tile shape [64, 64, 16] HDF5 create 22.94 real 0.64 user 0.94 system CCTS getTiles 20.3 real 0.55 user 1.42 system HDF5 getTiles 19.9 real 0.29 user 0.78 system CCTS xy getPlane 19.06 real 0.83 user 1.57 system HDF5 xy getPlane 21.11 real 0.86 user 1 system CCTS xz getPlane 22.04 real 0.93 user 1.61 system HDF5 xz getPlane 28.69 real 1.51 user 1.02 system CCTS yz getPlane 48.88 real 7.08 user 2.08 system HDF5 yz getPlane 55.24 real 19.79 user 1.22 system CCTS x getLine 20.12 real 1.11 user 1.51 system HDF5 x getLine 30.67 real 13.22 user 0.88 system CCTS y getLine 41.32 real 1.72 user 1.97 system HDF5 y getLine 55.62 real 22.01 user 1.19 system CCTS z getLine 36.12 real 5.15 user 1.84 system HDF5 z getLine 100.49 real 67.79 user 1.25 system

  12. 1 GB Creating tLatticePerf_tmp.tab with shape [2048, 2048, 64] and tile shape [128, 128, 8] CCTS create 39.39 real 2.54 user 3.06 system Creating tLatticePerf_tmp.hdf with shape [2048, 2048, 64] and tile shape [128, 128, 8] HDF5 create 49.96 real 1.21 user 2.21 system CCTS getTiles 59.18 real 2.11 user 3.45 system HDF5 getTiles 73.55 real 0.68 user 2.18 system CCTS xy getPlane 45.98 real 2.17 user 3.34 system HDF5 xy getPlane 49.96 real 1.14 user 2.11 system CCTS xz getPlane 51.63 real 1.97 user 3.31 system HDF5 xz getPlane 43.11 real 2.58 user 1.9 system CCTS yz getPlane 73.26 real 15.21 user 3.64 system HDF5 yz getPlane 98.87 real 39.88 user 2.35 system CCTS x getLine 37.92 real 2.19 user 3.02 system HDF5 x getLine 51.17 real 13.35 user 1.73 system CCTS y getLine 61.6 real 3.19 user 3.53 system HDF5 y getLine 100.33 real 30.68 user 2.32 system CCTS z getLine 70.48 real 16.63 user 3.59 system HDF5 z getLine 300.85 real 241.81 user 2.63 system

  13. 2 GB Creating tLatticePerf_tmp.tab with shape [4096, 4096, 32] and tile shape [128, 128, 4] CCTS create 85.27 real 3.14 user 5.79 system Creating tLatticePerf_tmp.hdf with shape [4096, 4096, 32] and tile shape [128, 128, 4] HDF5 create 83.29 real 2.59 user 3.7 system CCTS getTiles 75.12 real 2.2 user 5.64 system HDF5 getTiles 84.17 real 1.21 user 3.31 system CCTS xy getPlane 76.88 real 3.41 user 7.21 system HDF5 xy getPlane 129.02 real 2.36 user 4.65 system CCTS xz getPlane 110.61 real 3.37 user 7.96 system HDF5 xz getPlane 154.15 real 9.12 user 5.03 system CCTS yz getPlane 179.2 real 29.51 user 8.04 system HDF5 yz getPlane 257.18 real 85.12 user 5.04 system CCTS x getLine 74.77 real 3.12 user 6.12 system HDF5 x getLine 105.67 real 26.36 user 3.49 system CCTS y getLine 170.27 real 5.56 user 7.92 system HDF5 y getLine 223.01 real 63.88 user 4.82 system CCTS z getLine 182.7 real 60.03 user 7.67 system HDF5 z getLine 1089.83 real 940.97 user 6.02 system

  14. Results • HDF5 line by line uses a lot of user time • Verified by independent tHDF5 program • submitted as bug • Due to B-tree lookups? • HDF5 uses less system time, but usually slower • Files are about equal in size • HDF5 slightly larger (more if tiles are smaller) • due to B-tree? Note: • writes faster because no fsync done • for larger data sets kernel’s file cache not used

More Related