DLM Forum Härnösand November 5, 2009

1. Flexible and automated ingest of records from EDRM systemsLauri LehtNational Archives of Estonia DLM Forum Härnösand November 5, 2009

2. What happens when a central digital archive starts archiving digitally born documents with their descriptive metadata from different sources and different systems?

3. The archive finds out that: • Several EDRM systems used • Data created in ERDMSs are different • No interoperability! • Difficult to archive data into a central archive

4. What may be a solution? • Design a central software for standardisation of exported metadata • Connect all EDRMSs to the central software • Assumption: EDRMS can export metadata and records in XML • Normalise metadata XML in the central software • Ingest only normalised metadata XML into the central digital archive

5. Estonian implementation • The National Archives of Estonia designed and created a central software: Universal Archiving Module (UAM) • 2007-2009

6. Universal Archiving Module • Desktop software used by the agency’s archivist • Connected to EDRMSs with XML/XPath/XSLT-converters • Based on the NAE requirements for archiving digital documents • In Estonian, soon in English, multi-language option

8. UAM functionality • Create description of agency and archives • Import descriptions of Functions, Series and Dossiers that are extracted from EDRMSs • Validate these descriptions according to each level • If needed, supplement these descriptions with additional data • If needed, change the archival schema • Import documents and their descriptions that are extracted from EDRMSs

9. UAM functionality • Connect documents to necessary parts (Dossiers) of the archival schema • Check if file formats are archiving friendly • If needed, extract digital signatures and convert files into archiving formats • Form technical metadata of the files • Export archival schema and documents / files from UAM into XML capsules • Send these XML capsules to the NAE digital archive via internet or on offline media

10. UAM essentials • Possible to set rules for mandatory metadata and file formats • Manual input of missing metadata • Automated creation of technical metadata (JHove) • Some hardcoded validation rules • Automatic validation of NAE requirements and generation of SIPs (based on two different XML Schemas) • Transfer of SIPs using internet or off-line media

12. Purposes of UAM • Main purposes • Ensure unity and quality of documents and metadata sent to NAE • Simplify fulfilling of technical requirements (technical metadata, digital signature, converting etc) • Secondary purposes • Recommend an archival format for digital documents (XML document capsule) • Enable preparation of delivery of digital documents for other long-term digital archives • Let the agency’s archivist do the work!

13. UAM does not … support appraisal and extermination procedures. … directly manage the workflow between the agency and the public archives. … manage the entire digital archive of the agency as a whole.

14. Current status of project • UAM tested in one agency, a pilot project • UAM converter for one EDRMS, others coming • Requirements for connecting EDRMS to UAM in public tenders by agencies • 2nd development phase ending beginning of 2010

15. Conclusions • Flexible and automated ingest of records from different EDRMSs is possible! • NAE is interested in cooperation and further development of UAM. • Contact NAE for further details!

16. Thank you! The National Archives of Estonia Digital Preservation Bureau www.ra.ee Lauri.Leht@ra.ee (UAM project manager: Tarvo.Karberg@ra.ee)