Coalition for Networked Information Task Force Meeting, April 16, 2004, Washington, DC

1. Coalition for Networked Information Task Force Meeting, April 16, 2004, Washington, DC Making Sense of Usage Statistics for Online DatabasesChallenges, Lessons, and Strategies in a Statewide Context William E. MoenSchool of Library and Information Sciences<wemoen@unt.edu> Texas Center for Digital Knowledge University of North Texas Denton, TX 72603 Charles R. McClureSchool of Library and Information Studies<cmcclure@lis.fsu.edu>Information InstituteFlorida State UniversityJohn Carlo BertotTallahassee, FL 32306 <bertot@lis.fsu.edu>

2. Overview • Why and what of usage statistics • Library of Texas/TexShare experience • Data issues • The case of metasearch • Interpretation issues • Management issues • Final comments CNI Task Force Meeting--April 16, 2004--Washington, DC

3. Why care about usage statistics • Major investment by libraries and consortia in licensed resources • Usage information as basis for: • Securing funding • Allocating funding • Choosing or deselecting resources • Understanding patterns of use • A non-intrusive method for viewing users behaviors • Log analysis for user behaviors • Metasearch applications change nature of the usage and utilization of databases CNI Task Force Meeting--April 16, 2004--Washington, DC

4. What statistics to report • At an abstract level: • To what extent do users engage with a resource? • Sessions • Searches • To what extent do users utilize a resource • Viewing result lists • Viewing full records of particular results • Downloading or viewing full text (if applicable) • Downloading data for further processing (if applicable) CNI Task Force Meeting--April 16, 2004--Washington, DC

5. The issues • Do database vendors provide usable usage data for libraries • Do database vendors provide comparable data • How do libraries deal with heterogeneous data • How to automate analysis and reporting of heterogeneous usage data • How can data be best integrated into library decision making CNI Task Force Meeting--April 16, 2004--Washington, DC

6. TexShare • A cooperative program to improve service to Texans. It maximizes the effectiveness of library expenditures by enabling libraries to: • Share staff expertise • Share library resources in print and electronic formats • Pursue joint purchasing agreements for information services • Encourage cooperative development and deployment of information resources and technologies The TexShare database service • Statewide licensing of databases for users of academic, public libraries, and libraries of clinical medicine • Core databases • TexSelect databases CNI Task Force Meeting--April 16, 2004--Washington, DC

7. Library of Texas • Envisioned as a service-based virtual library that will enable Texans to search an extensive array of resources • The LOT initiative covers four basic components: • Indexing and preserving electronic government documents • Providing a statewide resource discovery service • Training librarians on electronic resources • Continuing to offer a wide selection of TexShare databases CNI Task Force Meeting--April 16, 2004--Washington, DC

8. Longitudinal usage analysis • Part of an evaluation project for LOT • Detailed analysis and reanalysis of vendor supplied data (FY01-FY03) • Purposes: • Compile database usage and trends • Determine the extent to which analysis could be automated CNI Task Force Meeting--April 16, 2004--Washington, DC

9. The TexShare statistics challenge • Different vendors • Multiple database per vendors • Changing number of databases per reporting period • Creating comparable usage data • Sessions • Searches • Documents: Number of full-text downloads • Verification of historical data CNI Task Force Meeting--April 16, 2004--Washington, DC

10. TexShare databases available CNI Task Force Meeting--April 16, 2004--Washington, DC

11. EBSCO Logins Searches Abstract Full Text Articles Transformation: Logins  Sessions Searches  Searches Full Text Articles  Documents Gale Retrievals Searches Turnaways Views Sessions Total connect time Transformation: Sessions  Sessions Searches  Searches Views  Documents Heterogeneous vendor data CNI Task Force Meeting--April 16, 2004--Washington, DC

12. Proquest Total visitors Total pageviews Total hits Total bytes transferred Average visitors per day Average pageviews per day Average hits per day Average bytes transferred per day Average pageviews per visitor Average hits per visitor Average bytes per visitor Average length of sessions Transformation Total Visitors  Sessions “tx” directory tree Page Views  * Searches “index.html” directory Page Views  **Documents * Searches = number of “tx” Page Views / 4 ** Documents = number of “index.html” Page Views / 5 Heterogeneous vendor data CNI Task Force Meeting--April 16, 2004--Washington, DC

13. Statistics available per vendor * Statistics provided for public libraries only CNI Task Force Meeting--April 16, 2004--Washington, DC

14. Summary statistics FY2002 FY2003 * One vendor does not provide search statistics CNI Task Force Meeting--April 16, 2004--Washington, DC

15. Summary stats per vendor FY2003 * Grolier doesn’t provide data for searches ** NetLibrary doesn’t provide data for sessions and searches CNI Task Force Meeting--April 16, 2004--Washington, DC

16. Data issues • Difficulties in obtaining complete data from vendors • Differences among vendors in their definitions (or lack thereof) of key terms such as sessions, downloads, etc. • Problems encountered by TSLAC in maintaining and collating accurate data from the various vendors • Problems encountered when retrieving vendor data from earlier months and having that data differ from the data originally retrieved from the vendor site • Difficulties in developing and integrating standardized reporting tools across different vendor data to summarize usage across the TexShare databases. • Difficulties in utilizing NISO Library Statistics Standard CNI Task Force Meeting--April 16, 2004--Washington, DC

17. Data issues BOTTOM LINE: Better to have limited data and use them wisely than to have no data at all! As research and development continues we will have higher quality data and better methods to insure normalization across vendors. CNI Task Force Meeting--April 16, 2004--Washington, DC

18. Emerging community agreements • Working towards vendor data reporting standards • Standards efforts • NISO Z39.7 Library Statistics standard • ISO 2789 Library Statistics standard • Both focus on e-metric (data element) definitional standards • Library/consortia efforts • ICOLC (International Coalition of Library Consortia) • Guidelines for data elements and reporting • Publisher/vendor efforts • Project COUNTER (Counting Online Usage of Networked Electronic Services) • Audited data reporting of vendor usage statistics in standard format • Code of Practice to which vendors adhere and adherence is independently verified CNI Task Force Meeting--April 16, 2004--Washington, DC

19. Vendor usage E-metrics • From NISO • Rejected Sessions (turnaways) • Commercial Services Sessions • Commercial Services Searches (queries) • Commercial Services Full-Content Units Examined • Commercial services descriptive records examined CNI Task Force Meeting--April 16, 2004--Washington, DC

20. Usage data in metasearch • Metasearch • Single search interface to multiple resources • Library of Texas Resource Discovery Service • Opportunity to define and control usage statistics in the metasearch application, e.g.: • Total Sessions • Total Searches • Total Searches per Target Resource • Total Full Text Downloads • Total Link Outs to Target Native Interfaces per Target Resource • Total Get It Requests • Total ILL Submitted • Interaction with metasearch and vendor statistics • What are user behaviors in the metasearch environment? CNI Task Force Meeting--April 16, 2004--Washington, DC

21. LOT Search Interface CNI Task Force Meeting--April 16, 2004--Washington, DC

22. RDS Analyzer CNI Task Force Meeting--April 16, 2004--Washington, DC

23. Interpretation issues • What constitutes significant use? • What measures point to impact? • Approaches: • Annual usage data could compare the statistics per year by library (1300 academic and public libraries) • Annual use data could compare the statistics per year per person (approximately 21.3 million residents) • Cost of the TexShare databases for each fiscal year could be used to determine the average cost per session (or download) for (1) all libraries or by academic or public library, and (2) on a per capita basis for residents of Texas • Assuming the ability to provide a “legal service population” for each academic and public library participating in TexShare, the use data can also be normalized on a per capita basis for each library – which could indicate possible variations in the use (and therefore the cost) for each library • Cost avoidance without regard to usage CNI Task Force Meeting--April 16, 2004--Washington, DC

24. Management issues • Organizing for data management: Who has overall responsibility for data management and who has specific responsibility for • Collecting the data • Designing and maintaining a management information system • Entering the data in a management information system • Verifying data with vendors • Analysis of data • Reporting data CNI Task Force Meeting--April 16, 2004--Washington, DC

25. Management issues (Con’t) • What is the budget available to support data management of online usage statistics? • Who are the audiences that the usage statistics will be reported to? • Do staff have the necessary skills to collect, analyze, and report this data? • Are there analyses of usage data that should remain internal to the organization? CNI Task Force Meeting--April 16, 2004--Washington, DC

26. Parting Shots • Online database usage statistics are essential for ongoing evaluation of online database services • Basic usage statistics can be used to develop a range of performance measures and indicators for outcomes assessment • Vendors must pay attention to standards compliance in usage stats • The usage statistics can be used to justify or refine database purchases CNI Task Force Meeting--April 16, 2004--Washington, DC

27. YOU CAN DO THIS! As libraries, consortia, and statewide digital libraries continue to grow and expand they will continue to rely on the provision of online databases. Ongoing collection, analysis, and reporting of usage statistics is an essential component for collection development as well as overall library services planning. CNI Task Force Meeting--April 16, 2004--Washington, DC

28. E-Metrics Instructional System CNI Task Force Meeting--April 16, 2004--Washington, DC

29. References • The Library of Texas • http://www.tsl.state.tx.us/lot/index.html • Library of Texas Resource Discovery Service • http://www.libraryoftexas.org • ZLOT Project • http://www.unt.edu/zlot • NISO Z39.7 Library Statistics standard • http://www.niso.org/emetrics • Project Counter • http://www.projectcounter.org • ICOLC (International Coalition of Library Consortia) • http://www.library.yale.edu/consortia/2001webstats.htm • E-Metrics Instructional System • http://www.ii.fsu.edu/emis CNI Task Force Meeting--April 16, 2004--Washington, DC