1 / 37

Implementing an Integrated DAMS FEDORA / VITAL at the National Library of Wales

Implementing an Integrated DAMS FEDORA / VITAL at the National Library of Wales. OR08 3 rd April 2008 Paul Bevan paul.bevan@llgc.org.uk Glen Robson glen.robson@llgc.org.uk. Outline. Introduction Single Point of Access – how and why Objects Structure Content Streams Ingest

oscar-cantu
Download Presentation

Implementing an Integrated DAMS FEDORA / VITAL at the National Library of Wales

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Implementing an Integrated DAMSFEDORA / VITAL at the National Library of Wales OR08 3rd April 2008 Paul Bevan paul.bevan@llgc.org.uk Glen Robson glen.robson@llgc.org.uk

  2. Outline • Introduction • Single Point of Access – how and why • Objects Structure • Content Streams • Ingest • Architecture • Implementation of Single Sign On • Lessons Learnt

  3. Background • FEDORA Users since 2005 • Early Pilot Project • What do we have? • How can we deal with it? • What should come first? • Steep Learning Curve • Planning for a repository infrastructure

  4. 2006: A ‘New System’ For NLW • Diverse Collections • No Single Metadata Standard/Format • Variance in the application of Standards • Many Legacy Systems • Improved Efficiency • Increasing Digital Resources & Assets VTLS (Virtua & VITAL)

  5. Single Metadata Repository • Existing Data Formats Migrated to MARC21 • Integrated Information Management System controls accession, circulation, statistics, preservation, etc. • Facilitates: • Combined (& Efficient) Workflows • Reduced Support Requirements • Single Point of Access…

  6. Single Point of Access • Why? • Varied Collections • Compartmentalised Users • Why Not? • Information Overload • Data Specific Functionality • Solution: Many & One

  7. Multiple Resource Providers • Single Point of Access, Single Metadata Repository, Single Catalogue. • Resources from many back-end systems: • Subscribed Electronic Resources • Free Online Resources • Digitised/Digital Resources (VITAL) • Stacks (Virtua)

  8. Full Skin eResources Family History Catalogue FEDORA/ VITAL Subscribed Resources ATHENS Search Engines OAI-PMH Digital Exhibitions IngentaConnect (etc) APIs Handle Resolution

  9. DAMS Project • 18 Month implementation process • Development requirements from VTLS • Substantial metadata discussions • Lots of institutional self-discovery

  10. VITAL Disseminators OAIS MARC MARC Virtua METS METS METS Digital Archive METS

  11. Virtua / Vital Integration • NLW Spec, VTLS Development • Primary access through catalogue • Less confusion for the user • Retain existing skills • Initially DAMS will hold digitisation images so catalogue record already exists • ‘Single Sign On’ • Existing patron information used for authentication • User & Group details passed directly from Catalogue to FEDORA as part of HTTP request

  12. Would we do the same again? Or… in the age of cross-searching is it worth integrating?

  13. Object Rules & Content Model • Every object will have a MARC record (at some level) within the IMS • METS will be our SIP, AIP and DIP • METS will be the policy, FEDORA services will enact the policy. • eg. DC Section in METS populates the DC datastream • eg. Structural Map held in METS, structure in RELS-EXT datastream • Every object in the repository will have a METS document • Every object will have DC for OAI-PMH • Object Rules evolve into a content model

  14. Content Model (AIP): Still Image http://hdl.handle.net/10107/1 Object (PID/Handle) DS: METS http://hdl.handle.net/10107/1-0 http://hdl.handle.net/10107/1-1 http://hdl.handle.net/10107/1-2 http://hdl.handle.net/10107/1-10 http://hdl.handle.net/10107/1-11 http://hdl.handle.net/10107/1-12 http://hdl.handle.net/10107/1-13 DS: DC (OAI-PMH) DS: RELS-EXT DS: Object (eg. TIFF File) DS: Object (eg. JPEG File) DS: Object (eg. Thumbnail File) DS: Object (eg. Zoomify File)

  15. METS Header: • History of METS document • Descriptive Meta Data • mods on digital copy • link to dc for original • premis details on original file name • relationship information to other objects in Fedora • Administrative Meta Data • mix technical meta data on images 1 • dc legacy technical data • premis admin data including checksums, and relationships between data streams • link to MARC record in Virtua • Digital Providence • List of actions that have occurred to the object • List of agents who ran the software / versions of the software used • File Sec • List of Data streams linking above sections to particular data streams • Structural Map • First one allows display via a METS viewer type application • Subsequent ones are linked to Disseminators • Behaviour Section • Links objects to their dissemination

  16. Object Structure Digitisation Projects (Object) Geoff Charles Collection (Object) … Wrexham Young Farmers' Club ploughing competition (Object) MARC Catalogue Record isPartOf

  17. Content Streams • The library’s mission is to: “collect, preserve and give access to recorded knowledge, in all documentary forms, with an especial emphasis on the intellectual record of Wales, for the benefit of all engaged in research and learning, or with other information needs” • In reality this means a LOT of stuff: • Born digital • Digitised • Physical Media • Digital Deposits • Archived websites • Floppy disks in archival files • And on… • And on… All of which needs prioritising

  18. Digitisation Projects (Still Images)  • Geoff Charles “Geoff Charles (1909 – 2002) was a newspaper photographer who dedicated 50 years of his life to portraying Wales through the lens of his camera. The contribution by Geoff Charles to Wales is unique and today his archive of 120,000 photographs is one of the treasures of the National Library of Wales.” • 2,294 Records • 14,038 Images • 16,322 Fedora Objects

  19. Welsh Journals Online • JISC Funded • 90 Journals since 1900 • Titles in Welsh and English • 400,000 scanned pages • OCR TEI & Coordinates metadata • ‘Complex’ rights issues.

  20. Physical Digital Deposits • Variety of unexpected and unknown deposits as part of archives • Large collection of deposited CDs • No selection process yet • CD Accessioning System (CDAS) • In-House development • Remove data from CD (to ISO) & Checksum • Create METS manifest • Determine filetypes with DROID • Selection Interface • Navigation of CDs over web interface • View files to aid in selection

  21. Offair Recording • NSSAW Archive • Every Programme (DVB-T & Radio) of Welsh Interest • All S4C • News & Current Affairs • Welsh Productions • Proprietary BOB system integrated with Virtua & VITAL • No Networked Access

  22. Wills • All Welsh probate records • 182, 404 Records • 816, 325 Images • 998, 729 Fedora Objects • 27 Days ingest • Concerns over Triplestore • Upgrade?

  23. Ingest • All automated • Wills data is catalogued in MARC 21 with links to an image • 1 Record points to a number of pages

  24. Process – Step 1Create SIP MARC Record Catalogue Parent METS Marc To METS Default METS Image METS Image METS Image METS Jhove Output Image METS

  25. Process – Step 2Ingest FOXML METS METS To FOXML Ingest XML Reserve PID Fedora Web Services Time to Ingest 16,322 Objects 10 Hours 40mins Fedora

  26. Architecture: Servers • Using Fedora 2.1.1 (No Journaling) • Redundancy rather than Performance. Server 1 Server 2 Handle Server Handle Server Vital Instance Vital Instance File System (FOXML) File System (FOXML) Oracle Database Oracle Database • SAN Connected through iSCUSI (Holds Reference Images) • Tape / Optical library (Stores Archive copies)

  27. DataGuard • Allows failover of database • Keeps both databases in sync • Currently we use rsync to sync fedora's data files • Data more important than Database!!

  28. Disseminators • Disseminator for each object that returns METS • Use cocoon to turn METS XML into web page • With no authentication you can build the view functionality as a disseminator • Once you turn on authentication it asks for username and password every time • Solution is to create a guest account and a web application which sits in between the user and fedora • Could be in the same tomcat instance as Fedora

  29. System Architecture

  30. Rights Issues for Wills • Free access internally • Paid for external access • External Group paid access • Printing services

  31. Implementing ‘Single Sign On’ Issue 1 - Authentication • Username + Password and group in catalogue • Web service on catalogue to authenticate • Need to develop servlet filter for Fedora to authenticate against this • web service Issue 2 - Authorization • Use Fedora XACML

  32. Implementing ‘Single Sign On’ Issue 3 - Single Sign on Use Case 1 • User goes to library catalogue • Logs on to catalogue • finds an item which is in Vital • shouldn't have to login again Use Case 2 • User in the library goes to catalogue • finds an item which is in Vital • Solution to Use Case 1 - Surrogates • Solution to Use Case 2 - Surrogates or local username? • Fedora sees IP address as catalogue • Can we pass a surrogate IP address

  33. Lessons Learnt • Keep your options open • 80% Soft, 20% Dev • Managing Expectations is Difficult • And that’s not even talking about format migration…

  34. Questions? Paul Bevan – paul.bevan@llgc.org.uk Glen Robson – glen.robson@llgc.org.uk http://dev.llgc.org.uk

More Related