COUNTER: making statistics useful

1. COUNTER: making statistics useful Peter Shepherd Director COUNTER January 2007

2. Background • Understanding usage • Different approaches • Role of usage statistics • Usage statistics • Should enlighten rather than obscure • Should be practical • Should be reliable • Are only part of the story • Should be used in context • How can usage statistics help us measure… • Success? • Value? • Behaviour?

3. COUNTERCodes of Practice • Definitions of terms used • Specifications for Usage Reports • What they should include • What they should look like • How and when they should be delivered • Data processing guidelines • Auditing • Compliance • Maintenance and development of the Code of Practice • Governance of COUNTER

4. COUNTER: current Codes of Practice • 1) Journals and databases • Release 1 Code of Practice launched January 2003 • Release 2 published April 2005 replacing Release 1 in January 2006 • Now a widely adopted standard by publishers and librarians • 60%+ of Science Citation Index articles now covered • Librarians use it in collection development decisions • Publishers use it in marketing to prove ‘value’ • 2) Books and reference works • Draft Code of Practice published in February 2005 for comment • Final version incorporating feedback was launched March 2006 • Relevant usage metrics less clear than for journals • Different issues than for journals • Direct comparisons between books less relevant • Understanding how different categories of book are used is more relevant

5. Journal and Database Code of Practice Usage Reports • Journal Report 1 • Full text article requests by month and journal • Journal Report 2 • Turnaways by month and journal • Database Report 1 • Total searches and sessions by month and database • Database Report 2 • Turnaways by month and database • Database Report 3 • Searches and sessions by month and service

6. Code of Practice for books • Book Report 1 • Number of successful requests by month and title • Book Report 2 • Number of successful section requests by month and title • Book Report 3 • Turnaways by month and title • Book Report 4 • Turnaways by month and service • Book Report 5 • Total searches and sessions by month and title • Book Report 6 • Total searches and sessions by month and service

7. Journal Report 1Full text article requests by journal Html and PDF totals reported separately

8. COUNTER: deriving metrics from Journal Report 1 • Local metrics • For libraries and library consortia • At journal, collection and publisher level • To compare the cost-effectiveness of journal subscriptions • To assess the value of Big Deals • Global metrics • For authors, funding agencies, libraries and publishers • At article, journal, collection and publisher level • To compare quality and value

9. COUNTER: ‘local’ metrics • JISC (UK Joint Information Systems Committee) • Funded by UK higher education funding councils • Supports higher education in the use of information and communications technologies • Access to information and communication resources • Advice on creation and preservation of digital archives • Implications of using ICT • Network services and support • Research to develop innovative solutions • National overview of online journal usage • Develop a reliable, widely applicable methodology • Use COUNTER Journal Report 1 ‘article full-text requests’

10. Local metrics: JISC Project • COUNTER data was analysed in relation to: • usage range • Price band • Subject category • Metrics derived from this analysis • Trend in number of full-text article requests • Full text article requests per title • Full text article requests per publisher package • Full text article requests per FTE user • Most requested titles • Usage of subscribed vs.. unsubscribed titles • Cost per full-text article request • Cost per FTE user • Summary report available at: www.ebase.uce.ac.uk/projects/NESLi2.htm

11. Local metrics: JISC project • Growth in article downloads • Publisher A: 12%- 208% • Publisher B: 12%- 59% • Publisher C: 23%- 154% • Publisher D: 22%- 81% • Cost per download • Publisher A: £0.97- £5.26 • Publisher B: £0.70 - £2.91 • Publisher C: £0.80 - £3.29 • Publisher D: £0.45 - £2.26

12. COUNTER: ‘global’ metrics • Impact Factor • Well-established, easily understood and accepted • Endorsed by funding agencies and researchers • Does not cover all fields of scholarship • Reflects value of journals to researchers • Over-emphasis on IF distorts the behaviour of authors • Over-used, mis-used and over-interpreted • Usage Factor • Usage-based alternative perspective • Would cover all online journals • Would reflect value of journals to all categories of user • Would be easy to understood

13. Global metrics: UKSG Project • Assess the feasibility of developing and implementing journal Usage Factors • Level of support from author, librarian and publisher communities • Data from which UF would be derived • COUNTER Journal Report 1? • Article numbers • Process for consolidation, calculation and reporting of UFs • Factors in the calculation • Level of reporting • Total usage • Articles • Report in April 2007 • Just completed set of 29 interviews with industry leaders • Wider online survey will take place in February 2007

14. UKSG Project: feedback • Are the COUNTER usage statistics sufficiently robust? • Frustration at lack of comparable, quantitative data on journals • Should items covered by restricted to articles? • Many journals still have significant usage in print • Diversity of views on the factors in the calculation • Specified usage period • Specified publication period • Usage data is more susceptible to manipulation • Will the journal be a meaningful concept in the future? • Two measures with different limitations are better than one, and UF will be derived from a set of credible, understandable data • Usage data will be used as a measure of value, whether publishers like it or not

15. Current issues • Interface and user behaviour effects on usage statistics • E.g. downloading HTML and PDF of the same article in one session • COUNTER is testing data filter solutions, but what does the duplicate downloading signify? • Reporting separately purchasable digital archive usage • Currently all usage for a journal is usually reported together • Separately purchasable archives mean we need separate reports for archival content, or a year of publication breakdown of usage • Usage in Institutional Repositories • Growth in Institutional Repository (IR) content • Need for credible IR usage statistics • IR usage statistics already being collected, but no standards • What to count and what not to count:- • How to deal with partial open access journals • How to deal with journals whose content becomes free after a fixed time period

16. New Development - SUSHI • Standardized Usage Statistics Harvesting Initiative (SUSHI) • No mechanism yet for automatically retrieving, combining, and storing COUNTER usage data from different sources • NISO-sponsored XML-based SUSHI aims to provide a means to do just this, via a standard model for machine to machine automation of statistics harvesting. • COUNTER and NISO have signed an agreement to work together on the development of SUSHI. More details of SUSHI can be found at:- http://www.niso.org/committees/SUSHI/SUSHI_comm.html

17. Other technologies that may impact usage reporting

18. Metasearch • Also known as: • Federated search • Broadcast search • User presented with a single search interface • Searches multiple information sources at same time • Eliminates (or assists with) resource selection • Retrieve, consolidates and ranks results • A new challenge for statistics

19. Metasearch: a new challenge • What can be affected • Session counts • Search counts • Why? • “Search all” option or automatic selection/search of many resources • Perform simultaneous activities • Optimization techniques

20. Prefetching • Google, and other services are introducing prefetching of links to improve performance • How it works… • A user performs a search that results in a number of results. • Google will introduce <link> tags for the first few to cause the browser to fetch in the background • The prefetched pages are loaded into browser cache

21. Prefetching • If the results were links to publisher full text then: • Each prefetch is a full text request • When the user does click the link, another header request is made, that could be considered yet another full text request • Without some kind of control, this activity could result in significant over-counting

22. Prefetching • The prefetch action is identified in the HTTP header of the request • Vendors can change systems to recognize prefetch and do one of: • Simply return an error (don’t deliver full text), the end user does not see this error, or • Fulfill the request, but do not count the transaction

23. What do usage statistics tell us about… • Success? • Value? • Behaviour?

24. Usage statistics: measuring value • JISC Project has identified some basic metrics derived from COUNTER data • Trend in number of full-text article requests • Full text article requests per title • Full text article requests per publisher package • Full text article requests per FTE user • Usage of subscribed vs. unsubscribed titles • Cost per full-text article request • Differences between subject fields • But…. • Limited to data from COUNTER-compliant vendors • Does not distinguish between different types of usage

25. Usage statistics: measuring success • Impact Factor • Widely used as a measure of ‘success’ for a journal • But… • Citation habits vary across different scientific fields • Citation patterns depend on journal type • Do not cover all journals • Usage Factor? • An alternative measure • Relevant in applied fields, where citation levels are lower • But…. • Requires global usage statistics • Requires universal adoption of the same standards

26. Usage statistics: measuring behaviour • Usage of different components of the journal • TOC, abstract, full-text, references • Variations between fields • Physics, medicine • Variations between institutes • Academic • Teaching, research, etc • Between departments • Industrial

27. COUNTER Membership • Member Categories and Annual Fees (2007) • Publishers/intermediaries: £530 • Library Consortia: £355 • Libraries: £265 • Industry organization: £265 • Library affiliate: £106 (non-voting member) • Benefits of full membership • Owner of COUNTER with voting rights at annual general meeting, etc. • Regular bulletins on progress • Opportunity to receive advice on implementation

28. http://www.projectcounter.org Apply for COUNTER membership

29. For more information………. http://www.projectcounter.org Thank you! Peter Shepherd, COUNTER pshepherd@projectcounter.org