180 likes | 277 Views
A Platform for Auditable, Distributed, Asymmetric Archival Replication. Micah Altman Associate Director, Harvard-MIT Data Center Institute for Quantitative Social Science, Harvard University
E N D
A Platform for Auditable, Distributed, Asymmetric Archival Replication Micah AltmanAssociate Director, Harvard-MIT Data CenterInstitute for Quantitative Social Science, Harvard University Bryan BeecherDirector of Computing and Network ServicesInter-university Consortium of Political and Social Research, University of Michigan Marc MaynardDirector of Technical ServicesThe Roper Center for Public Opinion Research, University of Connecticut Jonathan CrabtreeAssistant Director for Archives and Information TechnologyHW Odum Institute for Research in Social Science, University of North Carolina CNI 2008 Fall Task Force Meeting
Our Story • Who are you guys? • What problem are you trying to solve? • What have you done? • Why do we care? CNI 2008 Fall Task Force Meeting
Data-PASS • Partnership devoted to identifying, acquiring and preserving data at-risk of being lost to the social science research community • Partners • ICPSR • Odum Institute • Harvard MIT Data Center • Roper Center • National Archives http://flickr.com/photos/phauly/35555985/ CNI 2008 Fall Task Force Meeting
Data-PASS CNI 2008 Fall Task Force Meeting
Data-PASS • Lots of little files (social science data) • ASCII data files • PDF technical documentation (codebooks) • Millions of ‘em • Archival storage • Was tape • Now disk CNI 2008 Fall Task Force Meeting
Before CNI 2008 Fall Task Force Meeting
After CNI 2008 Fall Task Force Meeting
Archival storage? http://failblog.org/2008/02/08/floppy-fail/ CNI 2008 Fall Task Force Meeting
Archival storage? • Remote disks • Grids • Clouds • With partners? CNI 2008 Fall Task Force Meeting
Why roll your own? • Policy-driven • Auditable • Asymmetric • Independence of each location CNI 2008 Fall Task Force Meeting
Syndicated Storage Platform (SSP) • Start with LOCKSS • Lots of Copies Keep Stuff Safe • But used in a closed network • Private LOCKSS Network (PLN) • A few of them out there • MetaArchive perhaps the best known • Biggest selling point was independence of each node in the PLN CNI 2008 Fall Task Force Meeting
PLNs • LOCKSS is really easy to setup • PLNs are more difficult • Other differences between traditional PLN and our needs • Our content isn’t harvestable via HTTP • Our PLN nodes are different sizes • Our trust model requirement prevents a centralized authority controlling the network CNI 2008 Fall Task Force Meeting
SSP = Stone Soup Platform? • ICPSR and Odum setup a small PLN • HDMC provided a harvester and designed the schema • Odum built the Comparator • Roper is building the Invitor CNI 2008 Fall Task Force Meeting
PLN CNI 2008 Fall Task Force Meeting
Schema • Nodes • IP address • Storage commitment • AUs • Max size • # in the PLN • Lots more CNI 2008 Fall Task Force Meeting
Comparator • diff for our SSP • Compares • Contents of the LOCKSS Cache Manager [sic] • Schema • Produces • List of differences between “what is” and “what should be” • Feeds into another tool for “fixing the PLN” • Machine-actionable output (XML) CNI 2008 Fall Task Force Meeting
Invitor • Reads the report from the Comparator • Issues requests to PLN nodes to ADD or DROP an AU • Expectation is that PLN nodes always accept an ADD if they can • An offer they cannot refuse • Requests may be reviewed/approved by a human administrator (or not) • USENET news technology? CNI 2008 Fall Task Force Meeting
Summary • Data-PASS is a group ofarchives committed to preserving social science data • Exploring various technology options • One avenue is a custom LOCKSS deployment • Network schema • OAI data harvester • Comparison tool • Network update tool CNI 2008 Fall Task Force Meeting