OAI Protocol for Metadata Harvesting & Its Usefulness to STM Publishers

1. OAI Protocol for Metadata Harvesting& Its Usefulness to STM Publishers Timothy W. Cole (t-cole3@uiuc.edu)Mathematics Librarian & Professor of Library Administration University of Illinois at Urbana-Champaign 2005 Allen Press Emerging Trends SeminarNational Press Club, Washington, D.C. 13 April 2005

2. OAI Protocol for Metadata Harvesting • ‘Harvesting’ approachto interoperabilityat metadata level • Divides world intoMetadata Providers& Service Providers • Builds on HTTP,XML, & community metadata standards http://www.openarchives.org/ Timothy W. Cole (t-cole3@uiuc.edu)University of Illinois at UC

3. Metadata Harvesting Model End-Users OAI Service Provider Metadata & Content Repositories [Retrieval] Content Metadata(e.g. XML) OAIProvider [Search] OAI Harvester AggregatedMetadata OAIProvider Metadata(e.g. SQL) Content Timothy W. Cole (t-cole3@uiuc.edu)University of Illinois at UC

4. Metadata Harvesting Model (cont.) • OAI Service Provider (harvester) is middleman between content provider and end-user for selected metadata-based transactions – e.g., • Resource discovery • Value-added link mediation • Transactions involving full content still conducted directly between end-users and content provider – e.g., • Delivery of complete article in desired format • OAI-PMH is not synonymous with Open Access Timothy W. Cole (t-cole3@uiuc.edu)University of Illinois at UC

5. How OAI-PMH Works • OAI “VERBS” Identify ListMetadataFormats ListSets ListIdentifiers ListRecords GetRecord Timothy W. Cole (t-cole3@uiuc.edu)University of Illinois at UC

6. Protocol Details • OAI Transaction = OAI request (HTTP) & corresponding OAI response (XML) • Transactions initiated by harvester • Optional flow control mechanisms to manage provider load • OAI Item Identifiers – persistent & unique • Item (Metadata) Date Stamps – support selective harvesting • OAI supports multiple metadata formats • Distinguishes between an ITEM (complete metadata) &a RECORD (disseminated item of metadata in given format) Timothy W. Cole (t-cole3@uiuc.edu)University of Illinois at UC

7. Reliance on HTTP & XML • OAI-PMH is a REpresentational State Transfer (REST) protocol (unlike RPC, SOAP) • OAI requests and responses are sent via the HTTP protocol • OAI requests encoded as HTTP GET or POST operations • OAI responses are valid XML documents • Consistency and data “quality” is ensured by using XML Schema Definitions (XSD) for all responses • XML Namespaces used to identify which parts of response are metadata and which parts support the Protocol Timothy W. Cole (t-cole3@uiuc.edu)University of Illinois at UC

8. What it takes to implement OAI • Dynamic Web server functionality (e.g., CGI) • Capacity to respond with XML • Descriptive metadata in a standard format • OAI persistent identifiers & date stamps may require changes to metadata creation workflow • Open source implementations available (starting points) • OAI-PMH included in turnkey publishing solutions: • Public Knowledge Project (UBC) • Open Repository (BioMed Central), ... • Eprints.org, DSpace, Fedora, ARNO, CDSware, ... Timothy W. Cole (t-cole3@uiuc.edu)University of Illinois at UC

9. Provider Performance Issues • Database design biggest impact on performance • e.g., load to dynamically map to DC, other formats • Webserver performance load can be kept quite low • Use resumptionTokens, other flow control mechanisms to improve performance • Fetch only records needed to satisfy current request • resumptionTokens should retain state information for best performance and for idempotency Scale example: OCLC repository with 4+ million records Timothy W. Cole (t-cole3@uiuc.edu)University of Illinois at UC

10. OAI Implementation Guidelines for Repositories • Tools Required • Basic program strategies (incl. object-oriented approaches) • Guidance for use of • optional container elements • Metadata generation / mapping, data cleaning • Use of OAI Sets • resumptionToken, flow control, load-balancing • Denial-of-service prevention • Error handling • Strategies for deleted metadata records http://www.openarchives.org/OAI/2.0/guidelines-repository.htm Timothy W. Cole (t-cole3@uiuc.edu)University of Illinois at UC

11. Why OAI? • OAI is not synonymous with open access -- content provider maintains access control over full content • Implement once, provide metadata to multiple services • Less performance impact than robotic Web harvesting • Simpler than z39.50 • Puts your metadata in additional portals • But, less control over • How your metadata is presented to end-user • What your metadata is put next to by service providers • How valuable a commodity is your metadata? Timothy W. Cole (t-cole3@uiuc.edu)University of Illinois at UC

12. Who’s Using OAI to Expose Metadata OAI Data Provider Registry (http://oai.grainger.uiuc.edu/registry) • As of 1 March 2005: 607 active OAI metadata provider repositories • Range in size from millions of items, to less than 100 items • More than half are institutional repositories or eprint archives • Handful of publisher / publisher-aggregators, e.g.: • PubMed Central; BioOne; BioMed Central (partial); Project Euclid; Africa Journals Online; Institute of Physics (user id & password); American Physical Society (restricted access); ... • Individual journals, e.g.: • J. of STEM Education; Electronic J. of Probability; J. of Cognitive Affective Learning; Canadian J. of Communication; ... Timothy W. Cole (t-cole3@uiuc.edu)University of Illinois at UC

13. Who’s Harvesting Metadata Using OAI-PMH • Portals encouraging Open Access, e.g.: • OAIster; Public Knowledge Project; Citebase; Cyclades; ... • NSDL (STEM Education); NCSTRL (computer science);SAIL (physical science e-prints); ... • Local harvesting projects • As way to share data internally • As a collation service to their users – e.g., Grainger Search Service;OAI harvesting supported by some Library meta-search utilities • Web search engines that use OAI as one input stream • Yahoo! ingests from OAIster; Google looking to harvest DSpace sites; Scirus includes OAI metadata; ... • mod_OAI (Apache Web servers) as an alternative to Web robotic harvesting? Timothy W. Cole (t-cole3@uiuc.edu)University of Illinois at UC

14. Indirect Benefits from OAI-PMH • From Bibusages study (French National Library): • Digital Libraries are used in conjunction with Web search engines, generalist portals, commercial sites • Mix of intensive & casual users • DL users seeking answer for specific information need; most time spent discovering, viewing, & downloading documents “Digital Libraries … are now attracting a new type of public, bringing about new, unique and original ways for reading and understanding texts.”Houssem Assadi, et al. “Users & Uses of Online Digital Libraries in France,” ECDL 2003 Timothy W. Cole (t-cole3@uiuc.edu)University of Illinois at UC

15. Evolution of Scholarly Communication • Ubiquitous nature of electronic pre-prints & post-prints • Extensive linking to supporting content on the Web • Mixing of author-paid publication with traditional subscription based business models (e.g., AIP, Springer trials) • Citation frequency up for articles also available in arXiv: Demographic and citation trends in Astrophysical Journal papers and preprints / Greg J. Schwarz and Robert C. Kennicutt, Jr. BAAS 36:1654-1663, 2004 [also: http://arxiv.org/abs/astro-ph/0411275] • Some publishers encouraging self-archiving of pre-prints • IMS; APS; AIP; ... [see http://www.sherpa.ac.uk/romeo.php] OAI-PMH underpins these kinds of self-archiving services Timothy W. Cole (t-cole3@uiuc.edu)University of Illinois at UC

16. A Librarian’s Perspective • The information landscape can be seen as a contour map in which there are mountains, hillocks, valleys, plains and plateaus…. A specialized collection of particular importance is like a sharp peak. Upon a plateau there might be undulations representing strengths and weaknesses…. The landscape is, however, multidimensional. Where one scholar may see a peak another may see a trough. The task is to devise mapping conventions which enable scholars to read the map of the landscape fruitfully, at the appropriate level of generality or specificity. Michael Heaney (2000), “An Analytical Model of Collections and their Catalogues.” Timothy W. Cole (t-cole3@uiuc.edu)University of Illinois at UC