1 / 30

Journaled Component Files

Learn how to protect component files from damage and recover from unexpected system failures using journaling techniques. Available in APL version 12.0.3 and later.

hilldaniel
Download Presentation

Journaled Component Files

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Journaled Component Files Or – How to never see FILE DAMAGED again! John Scholes and Richard Smith 13 October, 2008

  2. Component files

  3. Purely linear file layout 1 2 3 • Free space • Component data (APL arrays) • Global file information (root)

  4. Updating a linear file 1 2 2 2 3 3 3 • Replacing a component with a smaller one wastes space • Replacing a component with a larger one is not possible ... • ... unless you move potentially large amounts of data first

  5. Actual file layout 3 1 2 • Free space • Global file information (root) • Component index blocks • Component data (APL arrays) • Free space nodes

  6. Updating a component 1 1’ • Write the new data to free space • (Note that the free space node is overwritten) • Update the component index blocks • Update the free space nodes • Update the root

  7. Adding a component 2 1’ • Write the new data in free space • (Note that a free space node is overwritten) • Update the component index blocks • Update the free space nodes • Update the root

  8. Adding – and causing damage 2 ! 1’ • Write the new data in free space • (Note that a free space node is overwritten) • ** APL process is killed ** • The free space node is still referenced but has been corrupted

  9. The solution - journaling • The free space in a file can be safely updated • The majority of an update occurs in this free space • Updates to existing data are first written to a journal • The update is then completed

  10. Adding - journaled 2 1’ • The free space can be updated • The journal is put in free space • Most of the component is written • (The free space node was left intact) • All remaining updates are journaled • The journal is activated

  11. Adding - journaled 2 1’ • Only free space updated so far • Entire update recorded in file

  12. Adding - journaled 2 2 1’ • The journal is executed • The journal is removed • The update is complete

  13. Accessing the file – example 1 2 2 1’ • Normal case - there is no journal • Nothing needs to be done

  14. Accessing the file – example 2 2 1’ • Process killed before journal complete • The updates were all in free space • The file has been safely rolled back

  15. Accessing the file – example 3 2 2 1’ • Process killed after journal complete but before update finished • The journal is (re-)executed • The journal is removed • The update has been completed and damage repaired

  16. Journaled files • Are supported now in 12.0.3 • Have very little impact on performance and file size • May be enabled on a per-file basis • ⎕FPROPS converts a file to/from journaled

  17. Journaled files • Can only be accessed by 12.0.3 or later (but journaling can be switched off) • Are not enabled by default • Protect from file damage if APL is killed • Do not currently always protect from file damage if the OS is killed

  18. Disk caching Disk O/S Kernel APL Process 1 1 3 3 2 1 2 3 2 3 3 2 1 2 3 1 1 • Disk writes are held in memory and flushed efficiently (out of sequence) • Data still flushed if APL killed • But if the O/S is killed, out of sequence data may be lost

  19. Why this matters - example 2 1’ • 1. Write to free space (inc journal) • 2. Mark journal as present • O/S dies; update 1 incomplete • Executing this broken journal would corrupt the file • There are 4 such points in an update

  20. Critical update sequence 2 2 1’ These must be done atomically: 1. Write to free space (inc journal) 2. Mark journal as present 3. Execute the journal 4. Remove the journal

  21. fsync solution Disk O/S Kernel APL Process 1 2 3 1 2 3 1 3 • fsync causes APL to wait for the data to be committed to disk • Could issue 4 fsyncs per update

  22. fsync solution • Slows the application considerably • So we should reduce the number of fsyncs if possible • Good news is that we can

  23. First fsync elimination 2 1’ • 1. Write to free space (inc journal) • 2. Mark journal as present • O/S dies; update 1 incomplete • Executing this broken journal would corrupt the file • Solution: add checksums to detect

  24. Second fsync elimination 2 2 1’ • 2. Mark journal as present • 3. Start executing the journal • O/S dies; journal no longer present • No journal for recovery • Solution: use the checksumming and redundancy to rebuild indices

  25. Second fsync elimination • Note: omitting this fsync does not prevent damage • But we are able to fix it

  26. Third fsync elimination 2 2 1’ • 3. Execute the journal • 4. Remove the journal • O/S dies; earlier updates lost • No journal for recovery • Rebuild indices

  27. Fourth fsync elimination 2 2 1’ 3 • 4. Remove the journal • O/S dies; update lost • If the journal is still present we may re-execute it on recovery • Otherwise it will fail its checksum validation

  28. Additional journaling options • Two fsyncs eliminated by checksumming • One further fsync eliminated if recovery tool used • Last fsync eliminated if recovery tool used ... • ... potential loss of more data

  29. Additional journaling options • Are planned for a future release • Will have a greater impact on performance and file size • Will offer a variety of options so that security and performance may be balanced • Will be configured on a per-file basis

  30. Journaled Component Files John Scholes and Richard Smith 13 October, 2008

More Related