Preservation DataStores (PDS): A demonstration of Preservation Aware Storage

1. Preservation DataStores (PDS):A demonstration of Preservation Aware Storage Contacts: Simona Cohen, Michael Factor, Dalit Naor http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

2. Preservation Descriptive Information Content Information Content Data Object Representation Information Reference Provenance Context Fixity PDS Overview • OAIS–based preservation-aware storage, media-agnostic, and generic storage to support logical preservation • Manage preservation specific metadata • Compute fixity • Update technical provenance • Manage PDI RepInfo • Ensure referential integrity • Storlet container • Module container that can execute restricted modules with predefined interfaces for data intensive functions, e.g., transformations, fixity calculations • Optimal scheduling • Update PDS modules, e.g., fixity algorithm, packaging format • Managing availability/ data loss • Physically co-locate data and metadata • Cluster related AIPs on the same media unit, based upon their relative importance • AIP identifier generation – globally unique identifiers http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

3. PDS Architecture ingestAIP, accessAIP, transformAIP, getPreservationPolicy • Layered approach is based on open standards: OAIS, XAM, OSD • In CASPAR, all layers are utilized. In other embodiments, only some layers can be used • Utilizes XAM to provide logical abstraction of containers (XSets) • Offloads preservation functionality to storage http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

4. AIP Representation in the Preservation Engine • The basic building-block of an AIP is the Information Object • A single instance of data is a byte stream • Zero or more instances of RepInfo to interpret the data • RepInfo is also an AIP • Shared resource with unique ID • RepInfo recursive network • Keep the design simple http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

5. Preservation of an AIP • Ingest AIP • Storing an AIP in PDS • Bit and logical preservation of an AIP • Bit migrations • Format migrations - data transformation • Transformation modules are packed as AIPs and preserved • Transformation result is a new version for the original AIP • Migrations are documented as Provenance records • During migrations PDS performs operations on AIP • Update PDI (e.g., fixity calculation, additional provenance events) • Execute previously loaded storlets • Access AIP • Retrieval of an AIP • By retrieval time, the original AIP may have several versions and copies http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

6. Demo

7. MST DataNatural Environment Research Council (NERC) Mesosphere-Stratosphere-Troposphere (MST) Radar at Aberystwyth • Provides measurements of vertical and horizontal components of the wind • Highly complex data • Preserved for the atmospheric scientific community • Needs long-term preservation for research purposes • Community is highly dependent on interoperability http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

8. British Atmospheric Data Centre and Self-Describing Data Formats • BADC strongly advocate the use of self-describing files to capture provenance and semantic information • Two self-describing file formats compete in the Atmospheric Science Community • NASA AMES (ASCII) • NetCDF (binary) • The current user community finds NetCDF more suitable http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

9. Demo Flow View AIP Ingest AIP 10 years later Load and apply transformation 20 years later Access and view transformed data http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

10. pds_view Video: http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

11. Ingest AIP flow in PDS • store (XUID, AIP ID) mapping persistently PDS web service AIP ID ingest (AIP) Preservation Engine • store (OID, XUID) mapping persistently XUID create XSet write fields (AIP data) commit XAM • Unpack AIP • Generate AIP ID • Compute fixity • Add “ingest” provenance event XAM to OSD (VIM) create OSD objects write (XSet data) OSD http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

12. Demo Flow View AIP Ingest AIP 10 years later Load and apply transformation 20 years later Access and view transformed data http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

13. pds_ingest Video: http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

14. 10 Years Later... http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

15. Format Change • Converting NetCDF to NASA-AMES • The MST NetCDF file version is no longer supported by the NetCDF foundations libraries, and files are not backwards compatible with this NetCDF version • The skills base of the atmospheric scientist in the future is heavily biased towards the use of ASCII-based file formats • The NASA-AMES format has become the preferred data format of the atmospheric scientists’ community • Converting NetCDF to PNG • Provide additional quick image view of the data for users with limited access permissions http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

16. Load transformation flow in PDS PDS web service AIP ID loadTransformation (AIP) Preservation Engine • Register transformation • Ingest transformation AIP http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

17. Transform AIP flow in PDS PDS web service AIP ID transformAip (targetAIPID, transformationAIPID) Preservation Engine • Access transformation AIP • Access target AIP • Execute transformation • Ingest result as new AIP, a version of the target AIP http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

18. Demo Flow View AIP Ingest AIP 10 years later Load and apply transformation 20 years later Access and view transformed data http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

19. pds_xform Video: http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

20. 20 Years Later... http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

21. Access AIP flow in PDS • Validate AIP ID • Validate fixity • Add “access” provenance event • Package AIP PDS web service AIP access (AIP ID) Preservation Engine AIP data open XSet (XUID) read fields XAM • Lookup XUID by AIP ID XAM to OSD (VIM) XSet data read OSD objects • Lookup OSD OID by XUID OSD http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

22. Demo Flow View AIP Ingest AIP 10 years later Load and apply transformation 20 years later Access and view transformed data http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

23. pds_access Video: http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

24. Thank you! Haifa preservation team: Simona Cohen, Michael Factor, Aner Hamama, Ealan Henis, Dalit Naor, Petra Reshef, Shahar Ronen