Objective Collections Evaluation Using Statistics at the MIT Libraries

1. Objective Collections Evaluation Using Statistics at the MIT Libraries Mathew Willmott MIT Libraries ACS National Meeting and Exposition August 22, 2010

2. Overview • Introduction/Background • Data Gathering • Data Analysis • Decision Process • Applications • Future Work

3. Introduction: Statistics • “There are three kinds of lies…” • Shortcomings of anecdotal evidence • New technology for dissemination enables new technology for evaluation

4. Introduction: Financial Issues • In the world • At MIT • In the MIT Libraries

5. Introduction: Library Collection • Size of collection • Focus of collection • Cancellation project feasibility

6. Data Gathering: What data? • Cost • Usage • Impact Factor/Subject ranking • Papers published by MIT researchers • MIT-affiliated editors • Citations from MIT-authored papers

7. Data Gathering: From where? • Our budget commitments database • Publisher-distributed reports • Journal Citation Reports • Licensed databases • Journal web pages • Local Journal Utilization Report

8. Data Gathering: How? • Mostly manual • Some selective • Small team gathering for all librarians

9. Data Analysis Based analysis on four main data categories: • Cost per use • Average subject ranking • Papers published by MIT researchers • Presence of MIT-affiliated editors

10. Data Analysis • Ranked journals in each category of data • Assigned a “point” to the lowest performing journals in each category: • Lowest 50% by cost per use • Lowest 33% by subject ranking • Lowest 50% by papers published • No MIT-affiliated editors • Each journal ended up with a score of 0 (high-performing) to 4 (low-performing)

11. Data Analysis Data presented to librarian staff in Excel workbook: • All raw data • Sheets analyzing each category of data • Sheet assigning a score to each journal, with changeable criteria

12. Example of spreadsheet

13. Example of spreadsheet Lowest 50%: Cost per use > $20

14. Example of spreadsheet Lowest 50%: Cost per use > $20 Lowest 20%: Cost per use > $50

15. Example of spreadsheet Change the $20 per use criteria value…

16. Example of spreadsheet …to a $50 per use criteria value.

17. Decision Process • NOT used to make final cancellation decisions; important to note that there are other factors to be taken into account. • Used to identify candidates for cancellation that subject librarians would then examine more carefully.

18. Applications • Faculty and other stakeholders are very data-driven; this process allows for clearer explanations and communications • Process encourages a big picture view across all disciplines • There are some caveats: can’t cancel much from one publisher, society packages aren’t comparable…

19. Future Work: Other data • Trends from year to year • Eigenfactor/Article Influence Score • More LJUR data

20. Future Work Can be of use when not in cancellation mode: • Evaluate collections • Identify where money could be better spent • Identify which parts of the collection need better promotion

21. Thank you! Contact: willmott@mit.edu (photo credit: Flickr user neilio)