Evaluation of a Danish Digital Migration Project Danish National Archives (DNA)

1. Evaluation of a Danish Digital Migration ProjectDanish National Archives (DNA)

2. Goals of the evaluation • Report to the Ministry of Culture • Spread the word internationally (are we the first ones?) • Lessons learnt for future migration projects

3. BackgroundLegal and preservation perspectives • The Danish National Archives decides in which formats and structures public institutions submit their data for ingest • It hasn’t always been like that, and data submitted before 1998 had very heterogeneous datastructures and -formats • A standardisation of these data was therefore necessary in order to comply with today’s preservation standard

4. The digital migration projectChallenge and purpose of the Migration Project -Standardise records to make future operations automatic Records from before1998 Preservation standard (Execution Order no. 342) Access AUTOMATION Migration

5. The digital migration projectThe big numbers • 2005-2008; 30 person years • 10-15 specialised project members • Investment costs:190.000 $ • Total costs: 2.6 million $ • 1.7 GB data from 1960-90’ies in over • 200 different database structures totaling over • 2.200 Information Packages

6. The digital migration project The Object of the Migration

7. The digital migration projectPrerequisites #1 • Media Migration project

8. The digital migration projectPrerequisites #2 • Development of preservation standard

9. The digital migration projectKnowledge management • The knowledge challenge • Need of disappearing knowledge (ancient databases, code pages, COBOL) • Need of particular manpower capable of executing long, repetitive manual tasks in a structured and disciplined manner • The knowledge management setup • A favourable framework for easy access to knowledge via online databases and dynamic intranet • Two major documents dedicated to migration principles and working procedures. Updated at all times. Immense importance.

10. The digital migration projectMigration method and procedure • Establishment of Software developmentspecificationrequirements • What to do - Requirements for the underpinning of the manual work: The Migration Principles • How to do it - Underpinning of the manual work: The Working Procedures

11. The digital migration projectPreparation of data and documentation • Preparation of data • Restructuring of data and documentation into new IP structure • Registration of the IPs metadata into the preservation databases • Transfer to an intermediate storage, waiting for migration • Preparation of documentation • Documenting the IPs • Establish a documentation standard • Gather, create and systematise documentation • Scanning and registration of documentation • Documenting the migration • Logfiles

12. The digital migration projectSoftware development • In-house software development for reasons of control and specialisation of task • 1 flexible migration tool was developed for all the pre 1998 archives, and 1 for the post 1998 ones • Enabling automatic migration for the older archives required the development of special pre-migration software that could handle the myriads of different formats and structures.

13. The digital migration projectThe Migration - Illustration

14. The digital migration projectConclusions Migration (of old data) is expensive! • The transformation of 1.7 GB of data (1,825,361,100 bytes) and its prerequisites took  50 person years (65,000 hours) or  27.4 KB per hour to migrate

15. The digital migration projectConclusions Meeting preservation principles is expensive! • The Danish National Archives do not dispose of archives once they’ve been appraised and deemed worthy of preservation • The salvage operation of the magnetic tapes cost $ 9,000 per decayed tape and $ 62 per intact tape • It was 253,5 times as long to migrate 1 GB of a non standardised data as 1 GB of standardised data (63,36 hours <-> 0,25 hours)

16. The digital migration projectConclusions Standardisation is the key! • The goal of every migrating preservation institution must be to standardise data so that it can be migrated with standardised software • The DNA had to develop 175 applications in order to migrate 165 non standardised IPs • The development of migration tools for non standardised IPs represented 80 % of the expenses pertaining to software development

17. The digital migration projectConclusions Quality control is expensive but worth it! • A retrospectively unsatisfactory random check was performed on the magnetic tapes in 1995 and resulted in postponing the media migration project • Testing that data match your preservation standard on ingest is very important  influences preservation cycle • In an ideal world, there is thus 100% compliance between the preservation standard and all other preservation tools – from Ingest to Access

18. The digital migration projectConclusions Project management at service for everyday work! • Rigorous management, loose method • The project was meticulously planned, but there was room for changing methods subsequently to lessons learnt • Knowledge management priority • The exchange of and access to knowledge was facilitated by direct online access, weekly meetings and dynamic documents • Project management as instigator for an intense and enthousiastic project process • Teamwork, enthousiasm, importance of task, responsibility and social commitment were succesfully implemented by the project management

19. The digital migration projectConclusions Make sure your ancillary activitiesand preliminary examinations are adequate! • Hardware • The hardware used to process the migrations wasn’t powerful enough • Data and development • Examination of the data wasn’t thorough enough, and resulted in inadequate specification requirements for the software • The project could have examined the market for open source or commercial software solutions • Organisation • The agreements and coordination within the organisation weren’t in place from the beginning

20. The digital migration projectConclusion The migration project was a succes But is automation a possibility today? Almost, but… Anyways, we saved the data!

21. THANK YOU! MigreringsprojektetKonklusioner Köszönöm szépen! Evaluation Report  http://www.sa.dk/content/dk/forskning_og_udvikling/udviklingsprojekter/format-_og_strukturkonverteringsprojektet