Discovery or Displacement? A Major Longitudinal Study of the Effect of Web-Scale Discovery Services on Online (Jou

1. Discovery or Displacement?A Major Longitudinal Study of the Effect of Web-Scale Discovery Services on Online (Journal) Usage SCELC Colloquium March 5, 2014 Michael Levine-Clark, University of Denver John McDonald, University of Southern California Jason Price, SCELC Consortium

2. “…a steep increase in full text downloads and link resolver click‐throughs suggests Summon had a dramatic impact on user behavior and the use of library collections during this time period.” The Impact of Web-scale Discovery on the Use of a Library Collection Doug Way (2010) http://scholarworks.gvsu.edu/library_sp/9/

3. Vendor marketing http://www.oclc.org/partnerships/econtent/solutions.en.html

4. Does implementation of a discovery service impact usage of publisher-hosted journal content?

5. What did we measure? • Whether there is an effect • NOT why that effect exists (that’s a future study!)

6. “Society will need to shed some of its obsession for causality in exchange for simple correlations: not knowing why, but only what” • (Cukier& Mayer-Schonberger. 2013. Big data: A revolution that will transform how we live, work, and think.)

9. Data collection • List of libraries with discovery services • Searched on lib-web-cats • Surveyed Libraries • Discovery service Implemented • Implementation Date (month/year) • Search box location • Marketing effort • 149 Libraries Gave Approval • 33 libraries selected for this phase • 6 for each of the 4 major discovery services and a group of 9 libraries with no service

10. Dataset • 33 Libraries • 28 US, 2 CA, 1 each from UK, AUS, NZ • WorldCat book holdings • Average: 1,114,193 ; Range: ~300k to ~2.6mil • Implementation dates (Discovery Libraries): • 2010 (3), 2011 (19), 2012 (2) • 6 Publishers • 9,206 Journals • 163,545 Usable Observations

11. Methodology Compared COUNTER JR1 total full text article views for the 12 months before vs 12 months after implementation date June 2010 May 2011 May 2012 Implementation Year 1 Year 2 End Start Included implementation month in Year 1 to ensure that both periods included an entire academic year

13. Examine Data for Outliers

14. Observations by Library & Service

15. Observations by Publisher

17. Average Usage Change By Discovery & Publisher

18. Analyzing Usage Change: % vs Total Which is the better measure? Is it the same for publisher- & journal-level data?

19. Reducing variation due to institution size Currently converting to change per FTE Values are shown as x 1,000 to bring the change metric back per journal-library combination to a minimum of 0.1 2013 JISC Discovery study took a similar approach

20. Average Usage Change By Discovery & Publisher Per Journal & Per 10,000 FTE

21. Full Model Including Discovery Service, Publisher, and Library Including Discovery Service, Publisher, and Library

22. Does the effect of discovery service differ across libraries? Library 1-9 Library 10-15 Library 16-21 Library 22-27 Library 28-33

23. Nested ANOVA Model [all three factors – preliminary results]

24. Does usage change vary across libraries? Institution (sorted by Mean Change)

25. Does usage change vary across publishers? Publisher (sorted by Mean Change)

26. Does usage change vary across discovery services?

27. Does the effect of discovery service differ across publishers? Publisher

28. Results Can we detect differences between Discovery Services, Publishers, and/or Libraries and/or their interactions? Library – Yes Publisher – No Discovery Service – Yes Differential discovery service effect by publisher – Yes

29. Next Steps • Design & test for effects of: • Aggregator full text availability • Publisher Size • Journal Subject • Overall usage trends (Requires Disc Srvc ‘control’) • Configuration options in Discovery services • Expand pool of libraries • Perhaps explore WHY

30. Sharing Data • With participating libraries • Customized reports for each library • With participating publishers • Customized reports for each publisher • Presentations as requested • With discovery vendors • Presentations as requested • In publications and presentations • Maintaining anonymity of data

31. Doing “Resarch”, SCELC Style! • Why SCELC? • SCELC Discovery Project • Partial funding provided by SCELC • SCELC Participation • Survey: http://bit.ly/DSparticipation

32. Past/Future Presentations • Ithaka Sustainable Scholarship Conference (October 2013) • Charleston Conference (November 2013) • ER&L/Library JournalWebinar (December 2013) • ShangaiJiao Tong Univ / Beijing Univ Forum (Jan 2014) • SCELC Colloquium (March 2014) • ER&L (March 2014) • UKSG (April 2014) • Presentations posted on slideshare; linked from: • http://visualcv.com/lpq4t1s michael.levine-clark@du.edu | johndmcd@usc.edu | jason@scelc.org

33. Sharing SCELC library experiences with Discovery Systems Lala Badal, California Lutheran University (WorldCat Local) Linda Wobbe, Saint Mary's College (Ebsco Discovery Service) Beth Namei, University of Southern California (Summon) Panel Moderator: Jason Price (SCELC) SCELC Colloquium March 5, 2014

34. Ten Minute Assessments • Two institutional goals each • All the same - Percent of content discoverable • All different - Choose one thatsinteresting and assessable • Impact of your discovery system on instruction and reference • What is the single biggest factor that would increase use of your institution’s discovery system?

35. WorldCat Local Discovery Service at CLU LalaBadal, California Lutheran University March 2014

36. CALIFORNIA LUTHERAN UNIVERSITY • 35 undergraduate majors and 34 minors. • Graduate programs – doctorates in educational leadership, • higher education leadership and clinical psychology. • Master’s degrees in education, psychology, computer science, • business administration, public policy and administration, • information systems and technology, and economics. • Total enrollment – 4,282 students. PEARSON LIBRARY Collection features about 132,000 physical volumes, 130 active print periodicals, 158 databases, 219,000 eBooks, 65,000 fulltext journals, and 10,500 streaming online e-videos. WorldShare Management Services & WorldCat Discovery System was implemented in September 2011.

37. Discovery System that • Provides single point access to many library research resources • - In addition to our print collection, 80% of the library online resources overall are • discoverable and searchable in our WorldCat Local. • Enablespositive end user experience • Increases usage of the subscribed electronic resources

38. Percentage of discoverable and searchable resources by material type

39. Full-text downloads: pre and post discovery system assessment

40. Online Catalog Usage Selected Year (January 2013 – December 2013) Total: 472,907 Selected Period (January 2014 - February 2014) 52 Weeks Prior Total: 391,151 Total: 64,159

41. Use at Reference Using WorldCat Local for reference assistance is the “default action” for our Reference/Information Commons “staff”. If the discovery service search does not yield relevant results, then they start using subject related databases and/or research guides.

42. Impact on Instruction The WorldCat Local introduced to all lower–division classes. All 4 librarians at CLU teach the discovery layer and if needed additional resources (specific databases, e-journals, websites, and etc.) introduced as well. History Subject Guide Based on the CLU librarians impute and available research on impact of discovery systems on information literacy sessions, our Information Literacy Manager outlined why, when and how we use DS in instruction

43. Impact on Instruction Based on the CLU librarians impute and available research on impact of discovery systems on information literacy sessions, our Information Literacy Manager outlined why, when and how we use DS in instruction: Instruction approaches Introduce WCL as a place to launch your research, as one-stop shopping. Metaphor:A discovery tool is similar to a large department store such as Wal-Mart where everything is available and the quality of the merchandise can vary, whereas a subject database is more comparable to a boutique store, smaller selection but higher quality. Buck & Steffy (2013) Class Activities Have students explore the discovery tool and a more conventional database to discover similarities and differences. Then have a class discussion about the value of each tool. Have students work in teams of two to work on different options for refining a search and share the experience with the class. Have students list the ways they might use a discovery tool in their future work. Quick Writes (See lesson plan on the Discovery Tool vs. the Web) WCL Lesson Plan - IL Session

44. What is the single biggest factor that would increase use of WCL discovery system at CLU? AWARENESS

45. Unique Features: • - Allows users to discover relevant content beyond the library specific holdings • Provides librarians with collection development tool and generates • more PDA through ILL

46. Unique Features: • Provides patrons with opportunity to save searches and results, create and share reading lists and bibliography.

47. Unique Features: • Provides patrons with opportunity to save searches and results, create and share reading lists and bibliography.

48. Discovery Service Assessment Linda Wobbe Saint Mary’s College of CA 2014

49. SMC Goals • Google-like searching • Access the entire Library collection •  Discovery of overlooked resources •  Improve cross-disciplinary access

50. Discovery: search everything?