1 / 24

The HDF Group Q5 Demo

The HDF Group Q5 Demo. 5.6 HDF5 Transaction API 5.7 Full HDF5 Dynamic Data Structure. Q5 Highlights (I). New features added since Q4 (other than 5.6 & 5.7): Support for variable length datatypes Some deferred items from Q4 (mostly metadata get routines)

morwen
Download Presentation

The HDF Group Q5 Demo

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. The HDF Group Q5 Demo 5.6 HDF5 Transaction API 5.7 Full HDF5 Dynamic Data Structure

  2. Q5 Highlights (I) • New features added since Q4 (other than 5.6 & 5.7): • Support for variable length datatypes • Some deferred items from Q4 (mostly metadata get routines) • Data Integrity for metadata; although not helpful at the moment without a real storage backend. • Enable/Disable Data integrity checks through a property. • Can be done for metadata and raw data. • Still working with skeletal IOD on a local Linux box: • EMC has their first code demo this quarter. • Many tests (especially reads) are “faked” for now. • This is the main limitation for adding automated regression testing framework and benchmarks. • Accounts for new Cray system at LANL pending.

  3. Q5 Highlights (II) • Testing: • More tests are always added with new features. • Tests verify correctness of asynchronous execution, axe dependencies, function shipping, and HDF5 to IOD translation (to a simple extent). • Try to cover all routines added. • Automated with a simple script to run tests on a local machine. • Still working more on a larger automated regression testing framework. • (should be available in Q6). • This Demo will focus on 2 main and new features added: • Transactions and Read Contexts • Dynamic Data Structure support (Map objects)

  4. Outline • 5.6 HDF5 Transaction API: • Transactions and Versions recap from earlier technical presentation. • Diving more into semantics and usage. • Go through some pseudo code & actual code. • Run Demo. • 5.7 Full HDF5 Dynamic Data Structure: • Changes from Q4 • New Map objects and routines • Code example and Run Demo.

  5. FastForward Transactions (I) • A transaction consists of a set of updates to a container. • container ≈ file • Updates are added to a transaction, not made directly to a container. • Updates include additions, deletions, and modifications. • Any number and size of updates may be included in a single transaction. • Tiny transactions may have high overhead. • Large transactions will amortize the overhead. • Multiple processes can add updates to a single transaction.

  6. FastForward Transactions (II) • A transaction is finishedwhen no more updates will be added to the transaction. • Transactions can finish in any order. • The updates for a finished transaction are not visible in the container. • A finished transaction must be committed in order for its updates to become visible in the container. • Transactions are committed in strict numerical order. • When a transaction is committed, all updates in the transaction are applied atomically to the container and become visible. • If all updates cannot be applied, none are applied, and the transaction is discarded.

  7. FastForward Container Versions • A version identifies a container at a given state. • The version number equals the number of the committed transaction that transitioned the container into the state. • A read context can be created for a container version. • The read context allows access to the contents of the container version. • A given container version is guaranteed to remain readable until all associated read contexts are closed.

  8. Acquiring a Container Version • This is important, so that IOD will not flatten this version, and it is available to read from. • Only one process is required to acquire, but any number can. IOD will ref. count the acquired context. • If one process acquires a version, it can communicate the acquired version to other processes so that they do need to acquire it themselves. • The same number of release calls need to be issued as acquire calls. • [although different processes can release than acquired]

  9. Transactions and Read Contexts in Q5 • Two new set of routines added to the HDF5 FastForward API: • H5TR for transactions • H5RC for Read Contexts • All new APIs are asynchronous. • All HDF5 read/get routines take Read Context IDs. • All HDF5 write/update routines take Transaction IDs. • Note that transactions specify a read context, so writes/updates happen within one. • H5Fcreate_ff() will always use up transaction 0, i.e. application starts using transaction 1. • H5Fopen_ff() will return an acquired read context with the latest readable version of the container.

  10. Container Version Acquire/Release semantics /* Any Leader Process */ version = 15; /* acquire container version 15 */ rid = H5RCacquire(file_id, &version, H5P_DEFAULT, event_q); /* wait for the acquire to complete. This is not necessary, but user must live with the consequences that the acquire on a flattened version will fail and so all subsequent reads and the release call on rid will fail . */ /* If Leader has delegates that it wants to tell that it has acquired the container version, it has to wait for the acquire to complete before informing them */ H5EQwait(event_q, &num_requests, &status); MPI_Ibcast ()/ MPI_Isend() … /* Read from Container */ … /* If other processes were informed to use this version, must wait to hear from them before releasing */ MPI_Barrier()/MPI_Bcast()/MPI_Recv() … /* release container version 15. This is async. */ H5RCrelease(rid, event_q); /* Close RC ID. This is a local operation that just frees resources. */ H5RCclose(rid); … /* Wait on all Events, Everything was asynchronous thus far */ H5EQwait(event_q, &num_requests, &status); /* Any Delegate process */ MPI_Ibcast()/MPI_Irecv() … MPI_Wait(); /* client received a version – x = 15 ; create a read context ID. This is a local immediate operation. */ rid = H5RCcreate(file_id, x); /* Read from Container */ … /* Wait for all reads to complete */ H5EQwait(event_q, &num_requests, &status); /* Tell leader I am done with my reads.*/ MPI_Barrier()/MPI_Bcast()/MPI_Secv() … /* Close RC ID. This is a local operation that just frees resources. */ H5RCclose(rid1); …

  11. Hints to H5RCacquire • It is possible that the user is not interested with one Exact version to acquire; i.e. does not want the acquire to fail if the version specified is not valid. • Through the Read Context Acquire property list, the user can specify the following hints to acquire: • H5RC_EXACT: Fail if the current version is not available (default). • H5RC_PREV: Acquire the highest version smaller than the one specified if it is not available. • H5RC_NEXT: Acquire the lowest version greater than the one specified if it is not available. • H5RC_LAST: Acquire the last readable version; this will ignore the version specified in acquire.

  12. Other ways to Acquire a Version • When opening a container, the user can optionally ask to also acquire the last readable version of the container: • When finishing a transaction, the user can also optionally ask to acquire it into a read context: file_id= H5Fopen_ff(file_name, H5F_ACC_RDONLY, fapl_id, &rid, H5_EVENT_QUEUE_NULL); H5TRfinish(tid, H5P_DEFAULT, &rid, H5_EVENT_QUEUE_NULL);

  13. Creating a Transaction • All operations that write/update the container must be part of a started transaction. • All transactions must be created within a read context, because some updates require reading from the container. • Transactions may be started by x leader processes and communicated to other delegate processes, or started by all processes. • Those are two different models of operation. • In the former case, it is not required that the processes who started the transaction be the one who finishes or aborts it.

  14. Operating Models • Leaders (Red) • Will start transaction. • Will communicate that transaction has started and is available to do updates on to the delegates (blue). • Will hear back from delegates that they are done updating. • Will finish transaction; or designate a delegate to do it. • Same number of start and finish calls. • Delegates (Blue) • Will hear from leaders that transactions can be used. • Can write to transactions. • Have to inform leaders when done with updates to transactions.

  15. Leaders/Delegates Model /* create transaction object with an already acquired read context 15 – this is a local immediate operation */ tid= H5TRcreate(file_id, rid, (uint64_t)20); /* start transaction 20 with default model, i.e. Leader Model. */ if( I am a leader process) /* This is asynchronous, but here we make it synchronous so we can tell the delegates that it has been started */ ret = H5TRstart(tid, H5P_DEFAULT, H5_EVENT_QUEUE_NULL); trans_num = 20; /* Tell other procs that transaction 20 is started */ MPI_Ibcast(&trans_num, …); /* Leader processes can continue writing to transaction 20, while others wait for the ibcastto complete */ if(I am a delegate process) MPI_Wait(&mpi_req, MPI_STATUS_IGNORE); /* Write to container */ …. /* Delegate processes have to complete operations before notifying the leader */ if(I am a delegate process) H5EQwait(event_q, &num_requests, &status); /* Each Leader synchronizes with its delegates that they are done writing */ MPI_Barrier()/Bcast()/… if( I am a leader process) /* Finish the started transaction. */ ret = H5TRfinish(tid, H5P_DEFAULT, NULL, event_q); /* Note that leader does not have to wait for its updates to complete before issuing the finish. We track this internally. */ /* Wait on all Events*/ H5EQwait(event_q, &num_requests, &status);

  16. Multiple Leaders/No Delegates Model /* All processes are considered equal participants in the transaction semantics; no inter-process communication is required More communication with the IONs will be done though. */ /* create & start transaction 20 with num_peers = n */ tid2 = H5TRcreate(file_id, rid2, (uint64_t)20); trspl_id= H5Pcreate (H5P_TR_START); H5Pset_trspl_num_peers(trspl_id, n); H5TRstart(tid2, trspl_id, event_q); H5Pclose(trspl_id); /* Update/write to the container */ … /* finish transaction 20. */ H5TRfinish(tid2, H5P_DEFAULT, NULL, event_q); /* Wait on all Events; everything was asynchronous thus far. */ H5EQwait(event_q, &num_requests, &status);

  17. Operating Within Transactions • HDF5 has metadata and raw data operations. • Operations that occur inside a transaction are typically to update the contents of the container, i.e. create a group, create a dataset within a group, write to the dataset, etc… • Is it required, for example, if I create a dataset to commit the transaction that the dataset was created in to be able to write to the dataset? • No! But there are limitations/rules that must be followed

  18. Operating Inside a Transaction • If I start transaction 1: • Create a group G1: • Create a group G2 in G1: • This is possible because a read is not required to write to G1. • If we happen to do something like this : the operation will fail because it will require a read to get to G1 from the root group. But since G1 is not readable in the transaction it has been created in, the result will be a failure. • Using a path in an access operation requires all objects in the path to be readable from the read context that is used by the transaction. • gid1 = H5Gcreate_ff(file_id, "G1", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT, tid1, event_q); • gid2 = H5Gcreate_ff(gid1, "G2", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT, tid1, event_q); • gid2 = H5Gcreate_ff(file_id, “G1/G2", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT, tid1, event_q);

  19. Access to Objects created by Leader • But what if 1 process created the object, and other processes need to write to the object in the same transaction? • The leader process will need to retrieve a token representing the object when it created it. This is a local operation: • H5Oget_token(object_id, &token); • Send the token to the other processes. • The other processes will open the object using the token they received: • obj_id = H5Oopen_by_token(token); • This is commonly referred to as local-to-global/global-to-local operation. • This is not currently supported. Deferred to Q6.

  20. Quick Summary • Transactions provide a mechanism for making atomic updates to a container. • Committed transactions result in container versions. • Writes are done to the future. • To uncommitted transactions • Reads are made from the past. • From container versions

  21. Demo • Look at some example code. • Run Tests.

  22. Dynamic Data Structures • The main purpose is to support ACG’s need to access the FastForward stack. • We added a new HDF5 object called a Map object with a new set of routines that should fully support the Dynamic Data Structure use case. • H5DO routines to conveniently append to datasets and fast-append mechanism: • Realized that implementation with IOD will not be possible without an atomic append feature. • Will drop from current project as it wasn’t high priority as other features. • Support for variable length data is done with some limitations: • No nested VL types, or VL types as fields of compound types. • Neither are very commonly seen.

  23. Map objects • A Map object is a direct mapping to a KV store. • We wanted to expose that type of access to the application. • New routines added: hid_tH5Mcreate_ff(hid_tloc_id, const char *name, hid_tkeytype, hid_tvaltype, hid_tlcpl_id, hid_tmcpl_id, hid_tmapl_id, hid_ttrans_id, hid_teq_id); hid_tH5Mopen_ff(hid_tloc_id, const char *name, hid_tmapl_id, hid_trcxt_id, hid_teq_id); herr_tH5Mset_ff(hid_tmap_id, hid_tkey_mem_type_id, const void *key, hid_tval_mem_type_id, const void *value, hid_tdxpl_id, hid_ttrans_id, hid_teq_id); herr_tH5Mget_ff(hid_tmap_id, hid_tkey_mem_type_id, const void *key, hid_tval_mem_type_id, void *value, hid_tdxpl_id, hid_trcxt_id, hid_teq_id); herr_tH5Mget_types_ff(hid_tmap_id, hid_t *key_type_id, hid_t *val_type_id, hid_trcxt_id, hid_teq_id); herr_tH5Mget_count_ff(hid_tmap_id, hsize_t *count, hid_trcxt_id, hid_teq_id); herr_tH5Mexists_ff(hid_tmap_id, hid_tkey_mem_type_id, const void *key, hbool_t*exists, hid_trcxt_id, hid_teq_id); herr_tH5Miterate_ff(hid_tmap_id, hid_tkey_mem_type_id, hid_tvalue_mem_type_id, H5M_iterate_func_t callback_func, void *context, hid_trcxt_id); herr_tH5Mdelete_ff(hid_tmap_id, hid_tkey_mem_type_id, const void *key, hid_ttrans_id, hid_teq_id); herr_tH5Mclose_ff(hid_tmap_id, hid_teq_id);

  24. Demo • Look at some example code. • Run Tests.

More Related