Search to Discovery: Finding Global Scholarly Resources with Primo

1. Search to Discovery: Finding Global Scholarly Resources with Primo Pascal Calarco & Alison Hitchens, Library December 6, 2011

2. Agenda • The state of search in libraries (Pascal) • Expanding Primo beyond the local catalogue (Alison) • Questions

3. Library Information Systems: Milestones Discovery Metasearch Citation Linking ILS 3rd gen (Client-server; 1990s) ILS 2nd gen (Mainframe; 1980s) OCLC (library network; 1972) Early systems MARC 1960 1980 1990 2000 2010

4. In the beginning, there was the card catalog (1901+) Indexes: • Subject • Author • Title • Interfiled cards, call number access

5. Library of Congress National Union Catalog (pre-1956)

6. HenrietteAvram, Developer of MARC • Programmer/analyst at Library Of Congress • Developed system for printing card catalog information (MARC) • ISO certification 1973

7. Later, there was the Online Public Access Catalog (OPAC) • Machine Readable Cataloging (MARC) • Inventory of the print/physical holdings of a library • Better than the card catalog; keyword searching & boolean functionality • Non-intuitive; required training or intermediation (information professional) • Limited generally to single library

8. Library networks & resource sharing

9. Print to Electronic

10. Now: Electronic Almost Ubiquitous • 85%+ of journal literature digital • Hundreds of specialized scholarly databases • Mass print book digitization efforts • Electronic books going mainstream • Aggregated meta-indexes: 750 million metadata for journal/newspaper articles

11. Goal: improve user experience • Users want to FIND not search • Source required information to user regardless of format or location • Leverage our knowledge of academic community @ uWaterloo • Integrate into key services: LMS, CMS, other library services

12. Catalog ILL Meta-search eReserve Website Science-Direct Web of Science ETDs EEBO JSTOR Database Content Silos Content Silos System Silos

13. Metasearch: an interim step • aka Federated Search; emerged 2003 • Distributed search from one interface via web services, SOAP/XML gateways • Idiosyncratic and slow; vendors implemented variously • Relevancy of merged results problematic

14. Problems with catalog searching & evolution to discovery • UCLA & Berkeley: information retrieval & user behavior (1986-1996) • Google Books: “digitize the world’s knowledge” (2002) • Karen Schneider, Andrew Pace, Roy Tennant: “The OPAC ‘Sucks’”(2002) • Next generation catalogs -> Discovery (2008+)

15. Catalogs: Information Science Research • Christine L. Borgman (1986) “Why are online catalogs hard to use? Lessons learned from information retrieval studies” Journal of the American Society for Information Science • Ray R. Larsen (1991) “The decline of subject searching: Long-term trends and patterns of index use in an online catalog” Journal of the American Society for Information Science • Ray R. Larsen (1992) “Evaluation of advanced retrieval techniques in an experimental online catalog” Journal of the American Society for Information Science • Ray R. Larsen (1996) “Cheshire II: designing a next-generation online catalog” Journal of the American Society for Information Science • Christine L. Borgman (1996) “Why are online catalogs still hard to use?” Journal of the American Society for Information Science

16. How Users Search: What We’ve Learned • Most people make typos at least some of the time • Most searches are 2, 3, 4 words with no Boolean operators • Most searches use keyword • Search is hesitant, iterative, often random process of discovery • Most people start elsewhere • Few read help screens • Few use advanced search – this is true even in Google

17. The Google Effect • Expectations for web search tools now: • Radically simplified UI, fast results • Aggregated content • Relevant results on first page • Natural Language queries • Spelling correction/adaptation

18. The OPAC “Sucks” • The OPAC lacks common features of most search engines • Relevance ranking vs. last in, first out • Spell checking (related - did you mean?) • Popular query operators like + and – • Refine search • Sort flexibility • Faceting • Citation indexing vs full text • Developed for print materials, limitations with electronic materials or atomized items (like articles) • Difficult for certain known item search

19. Industry Trends • Decouple the front end (search and discovery) from the back end (inventory and cataloguing) • Service Oriented Architecture – many programs loosely coupled • Cloud services -- SaaS • The 5th generation of library business systems emerging now – hosted, cloud solutions

20. Discovery Characteristics • Enhanced Search Functionality • Faceted browse • Relevance ranking • “Did you mean?” / Spell Checking • auto-correction, resubmit search • Content aggregation • Integrating search for books, articles, etc. • Single, Simple Search Box • FRBR – functional requirements for bibliographic record, grouping editions

21. Discovery Characteristics, cont. • Enhanced Experience • Sometimes fun and engaging • Interactive/Collaborative • User centered design • Enhanced Services • Find it / Get it for me • Book Covers / Synopsis • Full text • Availability on same page as results

22. Discovery Characteristics, cont. • Enhanced Content • Article Searching • Commercial Data • Merging Special Collections • Harvesting Online Collections • Grey Literature • Free Content • Enhanced Access • Syndication - Getting into users tools • Course Management Systems • Browser and Desktop Tool Bars • Portals

23. OPAC ILS Discovery Components • Next Generation Catalog • Next Generation “Unified Search” Aid Vendor Data Vendor Data Full Text OAI User Interface Normalization & Apache SOLR/Lucene MetaSearch MARC Circ Data

24. Content Components Phase I Phase II Future Primo Central TUG Others OCUL Archives Geospatial HathiTrust RACER Primo

25. Evolution of Discovery

26. Options for Expanding Primo • Local ingestion of resources using FTP or OAI harvesting • Searching remote resources in Primo using the Primo DeepSearch API* • Subscribing to a large centralized index, such as Primo Central *Application Programming Interface

27. Local ingestion of records • Example: Hathi Trust Digital Library • Harvest the public domain records from Hathi Trust Digital Library • Normalize the records • Index the records in our local Primo database • Schedule updates from Hathi Trust into Primo

28. Normalization: creating local sort field (Date – Oldest)

29. Primo Normalized XML (PNX)

30. Open source & Open platform • Primo uses Lucene for its indexing • SOLR exposes Lucene as a web service and allows for faceting • APIs and web services allow flexibility and customization

31. We can’t index everything! • Trying out a subscription to Primo Central, a centralized index of scholarly journal articles, newspapers, conference proceedings etc. • User sees one interface; user is searching 2 indexes

32. What is Primo Central Index? • A centralized index • of free and restricted resources • primarily articles & e-books • based on metadata & full-text provided by publishers/aggregators • based on the collections selected by the library in the Primo Administration module • created & maintained by our vendor, Ex Libris

33. What is Primo Central Index? • A centralized index • of records harvested using the same process as our local Primo database • created using the same PNX record structure as our local Primo database • indexed using the same indexing tools as our local Primo database

34. Blending local and remote resources • Both local and remote results are represented in the facets • Blended relevance ranking • Can configure Primo to boost high ranking local results so that when Primo is doing relevance ranking on our 4 million records alongside 100s of millions of Primo Central records local results aren’t missed by the user

35. Search = local resources & Primo Central

36. How does it work? Ex Libris has created & indexed records for millions of items based on information from the publishers Primo searches Primo Central the same way it searches the local database Full text availability is determined in advance by our URL resolver SFX, i.e. Delivery of the resource uses menu for

37. New features: snippets give context If your search term is found in the full-text, Primo supplies a snippet highlighting the term

38. New features: expanding the search Defaults to our library’s electronic subscriptions but users can expand the search to all of Primo Central

39. New Facets & Facet Values

40. Added value: bX Recommender

41. Trouble-shooting remote resources • We can view the PNX records using web services but we have no control over the content or the normalization rules • Records have the same structure as our local records but are missing local fields and don’t reflect local policies

42. Assessing Primo Central • Over 65 hours of one-on-one usability testing and focus groups with undergraduate students, graduate students, faculty, staff and alumni • Library staff survey • Feedback form • Statistics from Cognos

43. Looking to the future • What other content should be added to Primo? • How can we improve/enhance the interface? • What is the right balance for boosting local physical resources? • How do we point users to resources that can’t be searched using Primo?

44. Questions? • Pascal Calarco • Associate University Librarian, Digital & Discovery Services • pvcalarco@uwaterloo.ca • Alison Hitchens • Cataloguing & Metadata Librarian • ahitchen@uwaterloo.ca