Advisory Committee on the Electronic Records Archives April 29-30, 2009 Program Director’s Update

1. Advisory Committee on the Electronic Records ArchivesApril 29-30, 2009Program Director’s Update

2. Topics • Development and deployment of the ERA instance for the G.W. Bush presidential records • Plans for further development

3. Where is ERA?

4. Rocket Center, WV

5. Erma Ora Byrd Conference & Learning Center

7. The Search & Access ERA Instance for G. W. Bush Electronic Presidential Records

8. Focus: Functions: Federal Records Nationwide records management program National Archives Creation, review and approval of records schedules Manage transfer of physical and legal custody of all types of records Systematically collect, create, and manage lifecycle data about records Actual transfer, inspection, and archival storage of electronic records What Does the Base ERA Do?

9. Focus: Functions: Presidential Electronic Records George W. Bush Presidential Library Rapid ingest of very large volumes of electronic records Automatic indexing on ingest Immediate searchability, based on index Creation of different versions to support structured search of priority records Basic case management for review and redaction of sensitive content. What Does the Search & Access ERA Do?

10. Search and Access Instance Development • Achieved Initial Operating Capability December 8, 2008 • LMC proposed and received NARA and EOP agreement on an expedited method for transfer of electronic records. • NARA has enjoyed excellent collaboration from the EOP. • NARA implemented a contingency plan for access to high priority e-records, the finding aid for WH paper records and the database of digital photography, pending completion of processing into ERA.

11. EOP Transfer & Ingest Overview Exchange = 57 TB ARMS (FRA) = 5.1 TB SAN A2 RMS (Update) Merlin One2 = 36 TB ? 5/16? RMS Exchange SAN A SAN B1 Snap Server SAN B2 Storage Arrays ARMS (PRA) = 1.9 TB Merlin One = 36 TB RMS = 1.0 TB WARDS = .018 TB WARDS (delta) = 0.001 TB PDS = 0.0005 TB PDS (delta) = 0.0005 TB Non-Pri Types = 0.2 TB Non-Pri Types = 20TB 6.0 Exchange 7.0 ARMS (SAN) Data Type SW Drops PDS WARDS 2/11 7.1 Merlin One RMS 7.2 1/15 1/26 SAN B Returns 12/12 12/15 12/8 12/5 (IOC) 1/20 1/30 ARMS (PRA) SASS Operations (Ingest) PDS PDS (delta) WARDS WARDS (delta) Merlin One 11 January 3, 2020

12. G.W. Bush Presidential Electronic Records

13. Processing Status - 1 • All Bush e-records have been transferred to NARA’s custody. • Not all have been transferred to the ERA Data Center in WV. • EOP is maintaining copies until NARA successfully completes ingest. • Archives Operational Issues • Several sets of records were not transferred in the formats previously agreed by NARA and EOP • NARA required retransmission • Some records exhibited anomalies • Some ARMS email records had binary data in the “To” field • Some metadata in the digital photography system did not have corresponding images. • Some entries in the Records Management System are missing some fields. • MS Exchange email was not divided presidential from federal records or associated with EOP component, and contained numerous duplicates. • EOP is addressing these problems prior to transfer to ABL. • EOP has converted from proprietary to standard format. • NARA will preserve both the original files and the output of the EOP processing. • Encoding of date of birth in the Access system impeded searches on that field. • Viruses have been found in a small percentage of files. • Infected files have been successfully quarantined. LMC & NARA are working to produce clean copies.

14. Processing Status - 2 • Technical Issues • Issues with COTS products: • Automatic indexing of a batch of records stops when errors are found in any of the records; e.g., binary data in headers of email. • Erroneous results returned in certain conditions • Incomplete search results returned in other cases. • LMC underestimated storage space needed for the index. Additional hardware has been ordered. • Unanticipated software development needed to ensure complete and accurate mapping between ‘.eml’ email produced by the EOP and the original MS ‘.pst’ files • NARA directed LMC to hire a subcontractor to perform actual ingest of records.

15. Status of Requests for Bush Records • 28 Requests for access as of March 17, 2009 • Primarily for paper records • NARA has responded using data about the paper records in the Records Management System • A few requests were for digital photographs. • Most requests were addressed using the two systems NARA set up under the Contingency Plan because processing of the records had not been completed at the time the requests were received. • Three requests fulfilled using records on temporary ERA storage.

16. Plans for Further Development

17. What’s in Store for the Future? • Increment 2 • Preservation Framework • Introduction and use of a variety of tools for different preservation needs • Public access • Information about all types of records • Online access to electronic records • Initial system evolution • Increments 3 - 5 • Incremental enhancements in capability & capacity • Continuing system evolution • Governmentwide expansion • Full Lifecycle Management Plans • Appraisal case management and workflow • Search Framework supporting different tools • FOIA and other access case management • Review and redaction of sensitive content

18. Base Instance EOP Instance System Management Help Desk Network ERA Functional View: Current Status Agencies White House Enterprise Service Bus Shared Services Data Management

19. Records Center Instance Base Instance EOP Instance Congressional Instance System Management Network Help Desk Preservation Framework Public Access Data Management Current capability: solid fill Future capability: hashed fill ERA Functional View: Planned Agencies Agencies White House Committees Enterprise Service Bus Shared Services Public

20. ERA Instances • Base Instance (June 2008) • Used by NARA and federal agencies • For management of all federal records • For transfer, inspection and management of federal electronic records • EOP instance (December 2008) • Used by NARA and Presidential Administrations • For transfer, inspection, and management of presidential electronic records • Congressional Instance (future) • Used by NARA for Congressional Committees • For transfer, inspection, and management of presidential electronic records • Federal Records Center Instance (future) • Used by NARA and other federal agencies • For transfer and storage of temporary and permanent federal electronic records that remain under the control of the originating agency

21. ERA Shared Services • System Management (current) • System operation and maintenance • Security • User account management • Deployment of new & updated software • Backup & other common services • Help Desk (current) • Respond to technical questions and issues from users • Network • Link to the Internet, NARANET (current) • Interfaces with other systems (future) • Data Management • Data about records and transactions related to them (current) • Description of NARA holdings (Increment 2) • Review and redaction of records with restricted content (future) • Preservation Framework (Increment 2) • Tools to overcome obsolescence of different digital formats (future) • Public Access (Inc. 2 +) • Search and retrieval of information about records, regardless of custody • Search and access to electronic records in NARA’s custody • Search and access to digitized records from NARA’s holdings • Freedom of Information Act for restricted records in NARA’s custody

22. Advantages of the Instances & Shared Services Approach • Instances enable different business rules and processes for different mission requirements: • Base Instance: Federal Records Act provisions on governmentwide records management and on the National Archives • EOP instance: Presidential Records Act • Congressional instance: House and Senate rules. • Federal Records Center Instance: Federal Records Act provisions on storage of temporary and permanent records under originating agencies’ authority.

23. Advantages of the Instances & Shared Services Approach • Shared services maximize utilization of resources, reduce redundancy and provide a stable foundation for system growth and evolution over time. • Shared services deliver capabilities and capacity wherever needed, regardless of differences in mission and business needs • E.g. the Preservation Framework can be used to preserve any electronic records, regardless of whether they came from Congress, the White House or a federal agency. • E.g., a citizen seeking access to information will be able to find it using a single web portal, regardless of whether • It is information about records or in the records, • the records are in NARA’s physical custody, • the records are electronic or hard copy, • they originated in the White House, Congress or an agency.

24. Record Identity Record Integrity Original Order Preservation Electronic Record1 Electronic Record1’ Preservation Framework Electronic Record2 Electronic Record2’ … Electronic Recordn Electronic Recordn’ … Tool1 Tool2 Tooln The Preservation Framework supports the introduction and use of an arbitrary number and variety of processes under the control of archival requirements for authenticity.

25. Public Access • Information about all records • From Records Schedules • Archival Descriptions • Other NARA information • Online access to electronic records • Online access to scanned versions of hard copy records • Requests for copies of records • Freedom of Information Act requests for restricted records • Assistance from NARA staff

26. Increment 3 Work Status • Authority to Proceed Issued for Early Analysis • Architectural Framework • Preservation examination and prototyping • Search Engine examination and selection • Open Access examination and selection • Enhancements to address authorized user defined changes and software defects not addressed at IOC • Discussions begun on scope of work and technical details for full proposal • Target date for award: 7/09

27. Governmentwide Expansion • Initial Implementation • June 2008 – June 2009 • Four collaborating agencies • NARA staff proxy for other agencies • Invitational Phase • June 2009 – February 2010 • Additional agencies by invitation • Voluntary Phase • February 2010 – December 2010 • Additional agencies who volunteer and meet critera • Mandatory Phase • January 2011 • All agencies

28. 9/05 9/06 9/07 9/08 9/09 9/10 9/11 ERA Base Search & Access ERA Public Access & Preservation Framework Enhancement Enhancement The Development Timeline 6/08 Initial Operating Capability) Full Operating Capability Operation & Maintenance