120 likes | 234 Views
Curation Micro-Services “It’s a Series of Tubes”. Curation Micro-Services “It’s a Series of Tubes”. The Unix philosophy. “Make each program do one thing well” “To do a new job, build afresh rather than complicate old programs by adding new features”
E N D
Curation Micro-Services “It’s a Series of Tubes” Curation Micro-Services“It’s a Series of Tubes”
The Unix philosophy “Make each program do one thing well” “To do a new job, build afresh rather than complicate old programs by adding new features” “Expect the output of every program to become the input to another, as yet unknown, program” “Design and build software … to be tried early” “Don't hesitate to throw away the clumsy parts and rebuild them” — D. L. McIlroy et al., “Unix time-sharing system forward,” Bell System Technical Journal 57:6, part 2 (1978): 1902
The micro-services “philosophy” CC http://www.flickr.com/photos/oskay/265899811/ http://www.flickr.com/photos/elsie/8229790/ CC
Design goals Principle of least surprise Multiple interface modalities • RESTful HTTP • Command line • Procedural (Java, Perl, Ruby, …) Linked data Stable URL references The file system is the database State or content Storage node Object Version File Storage service http://example-store/ default/ 1234/ 3/ xyz state/
“You say micro, I say macro…” Service Tool Convention Access ANVL ARK bagit.pl BagIt CAN checkm.pl Checkm Dflat ERC EZID GhOST Ingest Inventory LockIt N2T namaste.pl Namaste Noid Pairtree ReDD RUU Storage Access ANVL ARK BagIt CAN Checkm Dflat ERC EZID GhOST Ingest Inventory LockIt N2T Namaste Noid Pairtree ReDD RUU Storage
Ingest process flow Create identifier Identity Identifier Submit Submitting user agent Ingest Node Addversion Notification Version metadata Getversion metadata Addversion Storage Node Notification Version metadata Getversion metadata Inventory Node Version metadata
Ingest implementation Submitting user agent Ingest notification HTML form Submission notification Batch or single object Job metadata Submitter Queue Consumer Ingester Storage Servlet Implicitly multi-threaded Zookeeper dæmon Dæmon Explicitly multi-threaded Servlet Implicitly multi-threaded Job payload
Questions? silverpipes.jog / firstpresmacomb.org
More information UC Curation Center (UC3) http://www.cdlib.org/uc3 Micro-service specifications https://confluence.ucop.edu/display/Curation Digital curation group http://groups.google.com/group/digital-curation UC3 Stephen Abrams Erik Hetzner Margaret Low Mark Reyes Perry Willett Patricia Cruse Greg Janée David Loy Tracy Seneca Scott Fisher John Kunze Isaac Rabinovitch Marisa Strong