Public Records in the Digital Age

1. Public Records in the Digital Age Salvador Barragan Curator of Government Records Nebraska State Historical Society

2. A ‘record’ is the complete set of documentation required to provide evidence of a business transaction. What is a record?

3. Before Paper we stored valuable historical data on stone and papyrus. In our current period records of historical permanency were stored on paper and kept in filing cabinets When the cabinet was full, records were sent to file room. Now records are stored electronically on computers When the computer is ‘full’ – add more hard drives or servers. Basic skills to manage and maintain records has been lost, replaced by infinite storage Shifting Media

4. Electronic Records Management Goals • Bring the record to the forefront of system design activities. • Identify electronic records functionality as part of system design. • Create electronic records that support legal, fiscal and evidentiary needs. • Create long term archival storage for both retention schedule and historical purposes.

5. Goals con’t 5.Create electronic records that are accessible and usable over time (non-proprietary formats). 6.Integrate diverse document forms and formats into records. 7.Identify need for internal and external primary and secondary access to records.

6. Three Functional Requirements for Electronic Records Management & Preservation

7. 1.Records Capture – Records are created or captured and identified to support the business process and meet all records management requirements related to the process.

8. 2.Records Maintenance and Accessibility – Electronic records are maintained so that they are accessible and retain their integrity for as long as they are needed.

9. 3.System Reliability – A system is administrated in accordance with best practices in the information resource management field to ensure the reliability of the records it produces.

10. What happens when you do not have a RM system?

12. As electronic records become more integrated into society, producers of those records will be held to higher standards of conduct HIPPA Sarbanes Oxley Federal and State Mandates Case Law Higher Standards

13. NE Public Records Laws 002.01 Record. The Records Management Act (Revised Statutes of Nebraska, Chapter 84, Article 12) defines a record as: "any book, document, paper, photograph, microfilm, sound recording, magnetic storage medium, optical storage medium, or other material regardless of physical form or characteristics created or received pursuant to law, charter, or ordinance or in connection with any other activity relating to or having an effect upon the transaction of public business." A record is information that is inscribed on a tangible medium or that is stored in an electronic or other medium and is retrievable in perceivable form. http://statutes.unicam.state.ne.us/Corpus/statutes/chap84/R8412013.html

14. The foundation of democracy in America is government accountability to the people and permanency of our culture and heritage. Records Retention

15. So the question becomes… who takes care of the records, and do they have the knowledge understanding of the new technology?

16. Historically records sent to file room, staff maintained access to records and managed lifecycle based on need and legal requirements Now records are managed by users and IT staff, based on capacity and cost. Caretakers of Information

17. Taking into account the goals of records management and the function of Records, what are we to do?

18. Or what is the solution? • Best Practice Models • Standards • Systems • Digital Archive?

19. Best Practice Models • OAIS Model • Washington State Archives • http://www.digitalarchives.wa.gov/default.aspx

20. OAIS Model

21. www.digitalarchives.wa.gov

22. Standards When ever possible follow the prevailing best practices and standards.

23. Standards for E-Records…. • Hardware • Software • Formats • Management • Authenticity

24. File Room of the 21st century Capacity and Speed double every 18 months Many choices Tape Optical Spinning Disc First Immutable Law of Digital Archiving “What hardware you use today will be obsolete within four years” Hardware

26. Network – Cisco Backbone end to end LAN and SAN EMC – SAN storage 5 TB now, 20TB by end of Year HP – Servers and desktops ADIC – Tape Library for offsite, disaster recovery (nightly or weekly back up, remember Katrina and 9-11) Microsoft – Software and Development Digital Archives Hardware

27. Native ASCII TIF PDF/A (Used by the Federal Courts) XML http://www.thexmltoolkit.org/guides.asp(metadata and interoperability) DoD 5015.2-STD compliant system Nebraska State Records Guidelines Whenever possible seek the Open, documented solution! Remember WordStar and DBase II ??? Archival Software and File Format Standards

28. Cross cultural and contextual boundaries Interoperability Interoperability & Metadata schema Metadata & Interoperability

29. Interoperability and XML

30. Essential to maintain control of the information explosion Allows hard coded rules and information exchange BUT still requires a strong knowledge, understanding and implementation of basic records management Second Immutable Law of Digital Archiving: “Data is Data, a Record is a Record, It is the content that drives retention, not the media” Content Management

31. DoD 5015.2-STD compliant system http://www.dtic.mil/whs/directives/corres/pdf/50152std_061902/p50152s.pdf Wrap original file in native format Wrap XML copy Apply metadata & XML for indexing, searching & retrieval Provide chain of custody & authenticity ‘Content Management’

32. Microsoft Solution SQL Server back end BizTalk translation utility SSH Tectia for secure transport http://www.ssh.com/products/ Washington State Archives Case Study ‘Content Management’

33. Maintain Chain of Custody In the care of trusted 3rd party Received from trusted, known source Authenticity

34. Encrypted SSH FTP transmission Issue Digital Certificate Verify IP and computer information MD5 Hash on all original files Copy of FTP on tape prior to ingestion DB backups on tape Record Level Security for confidential Info Data Security

35. Restrict records at item, field or series level Restrict to individual, dept, office or global Uses authenticated login to reveal fields Anonymous users see ‘Restricted’ Record Level Security

36. Deep Storage XML

37. Deep Storage XML Schema • Record Common • Who • What • When • Where • Original File • ‘web’ file • Security • Fixity • Vital Records • Type • Birth • Date of • Father, Mother • Hospital

38. MUST be flexible Microsoft BizTalk 2004 Transforms, adds metadata based on business rules Creates ‘deep storage’ copy wrapping original file in XML, with Hash Creates ‘web’ version of original file Ingestion Process

39. Designed around latest industry standards Open source, non-proprietary file storage Applies metadata ‘tags’ to save information about record creator, date, agency, subject, etc. Provides chain of custody & authenticity of record Allow search and retrieval of archival records through a web page Archive Database

40. Distributed, non-standardized environment Limited technology expertise in some agencies Unpredictable data growth rate Few business models Emerging technologies Limited internal expertise Risks

41. Authenticity of record Metadata File naming conventions Corporate Culture Start small with e-mail, web page Use existing retention schedules Educate Shift AWAY from desktops… …And move to central servers Management Software is a must! Privacy of sensitive data Management Issues

42. “Anything that you do today, will need major overhaul in two years or sooner” Technology and industry changing at unprecedented rates… But, more records are ‘lost’ every day! Key is to be flexible and address with systematic forethought Third Immutable Law

43. How to handle Records over the Web.

44. Open Record

45. Restricted Record Confidential

46. E-Commerce

47. Ecommerce Functionality Add to Shopping cart Add to Shopping Cart

48. Shopping Cart

49. Billing Information

50. View and Submit Order