100 likes | 184 Views
HDF5 Q4 Demo. Architecture. Full VOL Plugin Support. Skeletal IOD Artificial Data Read at server New Features/Functionality: Attributes (H5A*) Links (H5L*) Objects (H5O*) Datatype Conversion: Done at server, avoids extra buffer allocation at client
E N D
Architecture Friday Seminar
Full VOL Plugin Support • Skeletal IOD • Artificial Data Read at server • New Features/Functionality: • Attributes (H5A*) • Links (H5L*) • Objects (H5O*) • Datatype Conversion: • Done at server, avoids extra buffer allocation at client • Examples (BE to LE, 16 to 32 bit, etc…) • Non-contiguous memory selections: • Hyperslabs, point selections • Now supported with Mercury non-contiguous transfers
Asynchronous Execution • Nearly all HDF5 API routines are completely asynchronous now • Dependencies built at the client, executed at the server • No wait at client for previous operations to complete to be able to forward a child operation • Few API routines still cause partly asynchronous behavior • Child operations need to wait for them to complete at the client. • A small number of routines have to be synchronous: • Iterate/visit • H5Oopen • Using event queue (EQ) objects now to manage requests instead of individual requests: • Very cumbersome to manage a request for every operation issued. • Event Queue can be though of as “bags” to wait/test a bunch of requests. • There is no effect on dependencies or influence on execution order for requests in an EQ. • User can still pop off the last inserted request in EQ to manage it separately.
Unimplemented Functionality • Deferred • Iterate/visit (Does not make sense without a storage backend – Too much work to generate artificial metadata for them) • A few metadata query (H5*get_*) routines (same reason) • Variable Length Datatypes • Probably or Definitely will not support: • External Links - probably • References (H5R*) – probably • H5*_by_indexoperations – probably • File Mounting – definitely • H5Oopen_by_addr – definitely • Some Query operations (H5Dget_offset) – definitely
Testing Status • Testing is improved: • Verifying new VOL routines are shipped and call the correct IOD routines at server. • Regression testing framework is setup (Cmake only now, autotools later). • Need to beef up regression testing, but won’t be able to do much until we get something for a storage backend. • Temporary Native HDF5 backend: • Call in to the native HDF5 storage format to verify routines and their parameters are shipped from the client and executed correctly at the server. • Very limited: • 1 client & 1 server possible for this now • Not asynchronous
Data Integrity • Support in our level of the stack. • Added a public routine for applications to generate checksums: H5checksum(). • Added properties and routines to set properties for data integrity: • herr_tH5Pset_dxpl_checksum(hid_tdxpl_id, uint32_t value); • herr_tH5Pset_dxpl_checksum_ptr(hid_tdxpl_id, uint32_t *value); • herr_tH5Pset_dxpl_inject_corruption(hid_tdxpl_id, hbool_tflag);
ACG Dynamic Data Structures • Added 4 new routines and their FF version: • herr_tH5DOappend(hid_tdataset_id, hid_tdxpl_id, unsigned axis, size_t extension, hid_tmemtype, constvoid *buffer); • herr_tH5DOsequence(hid_tdataset_id, hid_tdxpl_id, unsigned axis, hsize_tstart, size_tsequence, hid_tmemtype, void *buffer); • herr_tH5DOset(hid_tdataset_id, hid_tdxpl_id, consthsize_tcoord[], hid_tmemtype, • constvoid *buffer); • herr_tH5DOget(hid_tdataset_id, hid_tdxpl_id, consthsize_tcoord[],hid_tmemtype, void *buffer);
H5DOappend() 3x5 Dataset H5DOappend(did, dxpl, 0, 3 , memtype, buf); H5DOappend(did, dxpl, 1, 2 , memtype, buf2); After 1: 6x5 Dataset After 2: 6x7 Dataset • H5DOsequence is very similar: • Read instead of write (no extension). • Accepts a starting position to read from.
ACG support for Next Quarter • Support for VL Datatypes: • Each array element in the dataset corresponds to an IOD BLOB object • Add an Append only hint to the Dataset Creation property list: • Optimizes access and storage • Only 1 BLOB object needed • Each array element stores an offset into the BLOB object • Map Objects: • KV stores at the HDF5 API level.