1 / 20

Preserving email

Preserving email. The PeDALS approach. About us. Pete Watters Arizona State Library, project coordinator pwatters@lib.az.us Richard Pearce-Moses Clayton State University, Georgia, principal investigator rpearcemoses@clayton.edu Brian Schnackel Arizona State Library, lead developer

toby
Download Presentation

Preserving email

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Preserving email The PeDALS approach

  2. About us • Pete Watters Arizona State Library, project coordinator pwatters@lib.az.us • Richard Pearce-Moses Clayton State University, Georgia, principal investigator rpearcemoses@clayton.edu • Brian Schnackel Arizona State Library, lead developer bschnackel@lib.az.us

  3. PeDALS methodology • PeDALS strives for OAIS compliance • Archivists focus on process, not individual records • Business rules… • generate normalized metadata • transform SIPs into standardized AIPs • create DIPs for each record

  4. Why email? • Suited to the PeDALS methodology • Born digital • Potential for historical value • Message transmission information provides a rich source of metadata • All partners had Outlook PST files

  5. Project goals • Atomize individual messages • To store as individual AIPs • To disseminate as browser-friendly DIPs • Create a database of rich metadata • From the process: to support administration • From the email headers: to support discovery • From BagIt, New Zealand Metadata Extractor, other sources: to support preservation

  6. Keep in mind PeDALS is intended for permanent records PeDALS is not a records management system Deleting files is difficult at best

  7. Why curate the PST? • When negotiating with the originating office, archivists encourage weeding PSTs of non-permanent records • Archivists work with rules rather than records – they don’t have time to weed the collections • If you give us junk, we’ll archive junk. • PSTs plucked from hard drives can work, but more likely to generate errors during processing.

  8. Metadata hiccups • Metadata taken from headers was surprisingly messy • One response is to learn to cope with a complete lack of authority control • Or possibly correct by “data wrangling” from within the database

  9. For example • Senders and recipients can be an email address or display name from one or more contact lists “Janet Napolitano” or “governor@az.gov” or “jnapolitano@az.gov” or “Napolitano, Janet “ or “Janet” or “J Napolitano”? • Subject line not reliable source for titles or abstracts – often blank, repetitive, or a remnant from an unrelated message

  10. Privacy tightrope • Email (and other records) may be open to the public by statute, but some content may be sensitive • Personally identifying information • Private information (intimate, of no public interest) • Repositories must develop procedures and policies for aggregates that may have some records with sensitive information

  11. Privacy storm clouds • Boucher/Stearns draft legislation for online privacy would require “notice to and consent of an individual prior to the collection and disclosure of certain personal information” such as street and email addresses, phone numbers, aliases, and other common information. • Excludes government agencies, but may include academic libraries. • Possible chilling effect on archives: Keeping such information confidential would effectively block access to email and many other records

  12. First attempts • PST file structure was proprietary • Considered third-party Outlook plug-ins • Smithsonian Institution had done research http://siarchives.si.edu/cerp/RAC_SIA_CERP_tools_V2_CC.pdf • Adopted open-source PST export utility • No longer supported • Written in Visual Basic http://www.genusa.com/utils/pmseu.htm

  13. From within Outlook • Could generate human-readable XML of email messages • Was based on code open to public • Did not require understanding of PST structure

  14. Looking at the PST It’s more than just email • What to do with tasks, calendar items, contacts? • Need to give the archivist the ability to decide what to keep • What about viruses, corrupt attachments?

  15. More than just the “msg” What is the record? What are we authenticating? • PST as database; messages are constructs of fields in tables tied together by keys and other tables • XML is best way to preserve these relations and dependencies

  16. Export utility’s drawbacks • Did not use the full record • Had almost no way to handle errors • Tended to break when dealing with large PST files that had not been curated • Required a copy of Outlook • Ran very slowly

  17. Microsoft PST spec • In late February, Microsoft released the PST specification http://msdn.microsoft.com/en-us/library/ff385210.aspx • 203 pages of techspeak with some errors and inaccuracies • Based on the spec, we’ve been developing a file-based tool that doesn’t require Outlook.

  18. File-based processor Generates XML from the entire PST file Much improved exception handling Does not require Outlook Runs much more quickly

  19. Other lessons learned File-based processor was slow to develop because of some errors in Microsoft’s documentation. Test on as many PST samples as possible. Don’t rely on small curated samples. Discovered differences between Unicode PST files and earlier ANSI-encoded files.

  20. Whither the PST? • PSTs are not an automatic occurrence in Outlook 2010 • But they can be generated manually and can remain part of a scheduled retention routine

More Related