Living Collections, Ambient Data

1. Micah Altman Harvard University Archival Director, Henry A. Murray Research Archive Associate Director, Harvard-MIT Data Center Senior Research Scientist, Institute for Quantitative Social Sciences E: micah_altman@harvard.eduW: http://maltman.hmdc.harvard.edu/ Living Collections, Ambient Data

2. Virtual Archives, “Living Collections”, Data Sharing, Community Data …data is not curated by single organization with single set of standards • What are the external curator’s collection development goals? How do they align with yours? • Who is responsible for adding, deleting, or modifying the content? Can they delegate this to others? • How will your archive be notified when content is added or modified? • What is the procedure for review of new content? Is it ongoing, or retrospective? • What agreements are in-place to support archiving this content? • Can and does the current depositor make commitments with respect to future content? • Does archive have clear rights to preserve and disseminate third-party curated content? • Are rights to content permanent? Can archive preserve content that curator has removed from their collection? • Is agreement itself persistent, subject to renewal, etc? • See: “Appraisal and Acquisition of Actively Curated Collections “ Micah Altman Considerations for Preserving Collections Curated by Third Parties

3. Paper -> digital Ad-hoc metadata -> Federated Catalog Idiosyncratic usage terms -> standard terms+ standard extensions + click-through Curator-focused acquisitions -> depositor focused Single copies -> distributed replication Micah Altman Evolution at the Murray Archive

4. Micah Altman Federation and Virtual Hosting

5. Micah Altman http://dvn.iq.harvard.edu/dvn

6. Examples • Harvesting and analysis of blogs for virtual political opinion surveys • Continuous collection of CSPAN, real-time subject coding, continuous dissemination • Cell phone data stream collection: movement, proximity to others, social network analysis • Participative goals-based redistricting • Agent-based models of emerging institutions • FMRI analyses of reaction to political and social scenarios • Modal Features** • Analyses emerge through exploration and interactions • Data collection from non-experimental, non instrumental, sources • Increasing scale of data • Compute limited • Data confidentiality • High-level analysis tools • Remote collaboration is part of projects Micah Altman Shifting Evidence Base? • Collective holdings of all U.S. numeric social science data in all major data archives, government repositories (estimated): • 50 TB < • “Ambient” data increasingly becoming subject of social science research (2002 data): • Web (surface): 167 TB • Radio: 3,500 TB • Television: 69,000 TB • Web (deep): 92000 TB • Email (originals): 441,000 TB • Telephone: 18,000,000 TB

7. Are we placing too much responsibility on the content creators to deposit data? Should the Partnerships shoulder even more of the burden to preserve content? What advice do you have regarding sustainability of projects after NDIIPP funding comes to an end? In particular, how do we continue to invest in collection development activities? How is your business model for collection and preservation likely to change in the next five years? What can we do to ensure that growth does not outpace the capabilities of young preservation services? Micah Altman Discussion Questions

8. Contact me: http://maltman.hmdc.harvard.edu/ <Micah_Altman@harvard.edu> Data-PASS Alliance: http://www.icpsr.umich.edu/DATAPASS/ Micah Altman More Information