1 / 20

Metadata Journaling

Metadata Journaling. Dana Robinson The HDF Group. Efficient Use of HDF5 With High Data Rate X-Ray Detectors Paul Scherrer Institut. Journaling. Metadata. File. Suppose we have some interrelated metadata that we would like to write into a file. Journaling. Metadata. Corrupted File. X.

keon
Download Presentation

Metadata Journaling

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Metadata Journaling Dana Robinson The HDF Group Efficient Use of HDF5 With High Data Rate X-Ray Detectors Paul ScherrerInstitut

  2. Journaling Metadata File Suppose we have some interrelated metadata that we would like to write into a file

  3. Journaling Metadata Corrupted File X ? If the write is interrupted (process killed, etc.), then we will have an invalid/corrupt file.

  4. Journaling Metadata Transaction File 2 1 Journal Journaling avoids the corrupt file problem by recording the set of writes (a transaction) in a journal.

  5. Journaling Metadata Transaction File X Journal h5recover When a transaction is interrupted, a recovery tool can repair the file.

  6. HDF5 Implementation HDF5 1.10.0 feature. Prevents loss of an entire file due to a crashed writer. Covers file metadata only! We make no guarantees about data. Works with parallel. Currently uses an external journal file. Journaling slows performance.

  7. Superblock Additions Journaling Flag Internal/External Flag Journal Location (path or address) Journal Version Number

  8. Journal File Format Binary file <journal> --> <header> <body> <header> --> <header_start_tag> <journal_version_number> <target_file_name_len> <target_file_name> <creation_date> <header_end_tag> <body> --> (<begin_transaction> | <entry> | <end_transaction> | <comment>) <entry> --> <begin_entry_tag> <transaction_number> <entry_base_addr> <entry_length> <entry_body> <end_entry_tag> <end_transaction> --> <end_trans_start_tag> <transaction_number> <end_trans_end_tag> <comment> --> <begin_comment_tag> <comment_length> <comment_string> <end_comment_tag>

  9. New API Call herr_t H5Pset_jnl_config(hid_tplist_id, const H5AC_jnl_config_t *config_ptr); Takes a struct (H5AC_jnl_config_t) which contains journaling parameters.

  10. H5AC_jnl_config_t typedefstructH5AC_jnl_config_t { int version; /* metadata journaling configuration fields: */ hbool_tenable_journaling; charjournal_file_path[H5AC__MAX_JOURNAL_FILE_NAME_LEN + 1]; hbool_tjournal_recovered; size_tjbrb_buf_size; intjbrb_num_bufs; hbool_tjbrb_use_aio; hbool_tjbrb_human_readable; } H5AC_jnl_config_t; Defined and documented in H5ACpublic.h Will be documented in the reference manual in HDF5 1.10.0.

  11. Transaction Start/End Transaction start/end calls are added to the beginning and end of all API functions that modify metadata. H5Xdo_something() { start_transaction(); /* Do things which modify metadata */ end_transaction(); }

  12. Ring Buffer Journal Buffer 4 Journal Buffer 0 Journal Buffer 3 Journal Buffer 1 Journal Buffer 2

  13. Journal Buffers Entry 0 Contains raw/binary journal entries to be later streamed to the journal location. These journal entries vary in size. Entry 1 Entry 2 Entry 3

  14. Start Transaction Assign the next transaction ID number. Insert a "begin transaction"message in the journal buffer with that transaction ID. Return the ID to the caller.

  15. Insert a Journal Entry Check for space in the current journal buffer. If no space… Start an asynchronous write of the current journal buffer. Test to see if the next buffer has an uncompleted write If there is, stall until it completes Switch to the next journal buffer Make an entry in the journal buffer

  16. End Transaction Insert an "end transaction" entry into the journal buffer. Increment the transaction ID number.

  17. Flush and Close Flush Write current journal buffer to disk Flush journal entries Truncate the journal Close Flush (as above) Load superblock and set journaling tag to FALSE Sync superblock

  18. Parallel Requires relatively few changes Transaction entries must be serialized at sync points and end of transaction. Process 0 really handles the transaction I/O. Journal I/O only happens at synch points for better I/O efficiency.

  19. h5recover h5recover [OPTIONS] [OBJECTS] [HDF5_FILE] OBJECTS -j, --journal [JOURNAL_FILE] journal file name OPTIONS -b, --backup [BACKUP_NAME] Specify a name for the backup copy of the HDF5 file. default = '[HDF5_FILE].backup' -f --force Recover without confirmation if the journal file is empty. -n, --nocopyDo not create a backup copy. -h, --help Print a usage message and exit -v, --verbose Generate more verbose output (repeat for increased verbosity) -V, --version Print version number and exit -x, --examine Examine the supplied file(s), report, and exit without action.

  20. h5recover Algorithm Try to find the superblock in the target file. Check to see if the journaling flag is set. Try to find the journal. Open the journal and validate it. Apply all metadata writes specified in the journal up to the last transaction. Reset the journaling flag and flush the file to disk.

More Related