Old and New Measures … Why Bother?

1. Old and New Measures … Why Bother? Georgetown University Library February 24, 2003 • Washington, DC Martha Kyrillidou Senior Program Officer for Statistics and Measurement Consuella Askew LibQUAL+™ Program Officer

2. Association of Research Libraries • Mission: Shaping and influencing factors affecting the future of Research Libraries in the process of scholarly communication • Members: 124 major research libraries in North America • Users: 3 million students and faculty served • Service measures: 73 million circulations 18.5 million reference transactions 108 thousand group training lessons 1.6 million participants

3. ARL Strategic Objectives • 8 including federal relations, scholarly communication, preservation, collections … and • PERFORMANCE MEASURES • To describe and measure the performance of research libraries and their contribution to teaching, research, learning and community service

4. ARL Statistics and Trends • ARL Statistics: • Serial and Monograph Costs • Interlibrary Loan/Document Delivery • Expenditures in ARL Libraries • Expenditures for Electronic Resources • ARL Annual Salary Survey • Salaries and Human Resources Issues • Preservation Statistics • ARL Law and Medical Library Statistics • LibQUAL+ • E-Metrics

11. Georgetown Libraries, 2000-01

12. Building the FutureMain Campus Strategic Plan: 2001-2006 • Continue library enhancement so that collection development can further support the scholarship and teaching activities of faculty and students. • Integrate technology support for research and teaching of the Library and UIS. • Expand funding for the digital library • Source: http://provost.georgetown.edu/documents/implement_plan.html

13. Building the future • What do you do to build the future? • Why is what you are doing relevant to the future? • How do you use the evidence you collect from your everyday transactions to define the future?

14. What data do you collect?

15. No Gluttony Measures Measure what is important, not just what is measurable because What you measure is what you will pay attention to and work toward

16. Measures that Matter • Input Output Outcome Impact • Consistent with organizational mission, goals and objectives • Integration with program review process • Balance customer, stakeholder, and employee interests and needs • Establish accountability • Collection and use of reliable and valid data

17. Opportunities and Pressures • Increasing demand for libraries to demonstrate outcomes/impacts in areas of importance to institution • Increasing pressure to maximize use of resources through benchmarking resulting in: • Cost savings • Reallocation

18. Reflections … Educational institutions today face new and significant challenges stemming from disruptions of financial markets, introduction of new technologies, demands for greater efficiency, and unprecedented requirements for investment in faculty, research, and infrastructure Some Early Reflections on TIAA-CREF by Herbert M. Allison (February 2003)

19. Why New Measures for ARL • Increased customer and stakeholder expectations for services and responsiveness • Greater demands for accountability • Exploding growth in use and applications of technology • Increasing competition for resources • Need for use of reliable and valid data • Benchmarking and best practice • Over time

20. New Measure How does a library answer the question, Do We Make a Difference?

21. Evaluation team representing the Middle States Commission on Higher Education A number of areas emerged from an overall observation of the many recommendations in the student section: a concern for diversity issues, desire for increased faculty/student interaction, need for additional space for student use to promote greater campus community, increased funding and better services for graduate students, and improved use of data from outcomes assessment for planning. Report to the Faculty, Administration, Trustees, Students of GEORGETOWN UNIVERSITYWashington, DC 20057 byAn evaluation team representing the Middle States Commission on Higher Education <http://provost.georgetown.edu/documents/evaluation.html>

22. The Challenge “The difficulty lies in trying to find a single model or set of simple indicators that can be used by different institutions, and that will compare something across large groups that is by definition only locally applicable—i.e., how well a library meets the needs of its institution. Librarians have either made do with oversimplified national data or have undertaken customized local evaluations of effectiveness, but there has not been devised an effective way to link the two” Sarah Pritchard

23. ARL New Measures Initiative • Collaboration among member leaders with strong interest in this area • Retreat setting to define topics • Specific projects developed with different models for exploration • Individual library may take the lead • Intent to make resulting tools and methodologies available to full membership and wider community

24. Development of Issue Papers (1)Tuscon, January 1999 • a) include an outcomes based definition • b) identify the need for the topic • c) provide some initial thoughts on data • d) identify what data is already collected • e) consider whether should be a part of ARL • f) recommendation for additional research • g) note whether topic is long- or short-term; • h) recommend possible experts • i) prepare a call for pilot projects

25. Development of Issue Papers (2)Tuscon, January 1999 • Ease and Breadth of Access • User Satisfaction • Library Impact on Teaching and Learning • Library Impact on Research • Cost Effectiveness of Library Operations and Services • Space and Facilities  • Market Penetration • Organizational Capacity Source: <http://www.arl.org/stats/newmeas/nmbackground.html>

26. ARL New Measures Projects • Investigation of role libraries can play in higher education outcomes: (a) learning (b) research • Assessing Information Literacy Outcomes (SAILS) • ILL/DD study to confirm changes in operations that lead to best practices • Project to define usage measures for electronic information resources (E-Metrics) • Demonstration project for service effectiveness measures (LibQUAL+) • Service Quality perceptions and expectations for the digital library (e-QUAL)

27. Learning Outcomes Brief History • Engage project consultant with experience in campus assessment activities • Supported with funds from interested members • Small group of leaders engaged in discussion • Preparation of strategy paper for discussion by larger ARL community in May 2000 • Kenneth R. Smith ‘New Roles and Responsibilities for the University Library: Advancing Student Learning Through Outcomes Assessment’ • Summer 2001: Working Group

28. Learning Outcomes Working Group (1) Summer 2001: Establishment of Learning Outcomes Working Group • Survey accreditation agencies to identify best practice institutions • Compile a list of learning outcomes being used by ARL institutions academic departments and general education requirements (if they exist) • Contact professional associations to identify any that measure learning • Identify offerings that libraries can develop to meet the outcomes identified

29. Learning Outcomes Working Group(2) • Establish a plan for how to take the development of offerings to the next stage with faculty in a select group of institutions • Explore collaboration with ACRL in offering training on information literacy skills within the ARL community • Define or create generalized tools for assessment • Define learning outcomes for the purpose of this ARL project and determine at what level the project should focus its attention (the student, course, program, department, etc.)

30. SAILS Brief History • Experience with SERVQUAL in many libraries over the last 10 years • Texas A&M SERVQUAL assessment • Meeting of interested ARL libraries (ALA Midwinter 2000) • Pilot with 12 ARL libraries (spring 2000) • External funding through FIPSE, U.S. Department of Education (September 2000) • Participation and endorsement by Big 12 Plus Consortium • 43 libraries participating spring 2001 • 164 institutions participating in spring 2002 including OhioLINK and AAHSL groups • 316 institutions participating in spring 2003 including NY3Rs, 21 British libraries, 1 Dutch, and three institutions experimenting with a French version of the instrument

31. SAILS Brief History • Based on ACRL Standards • 92 items developed • 70% of ACRL learning outcomes covered • Field testing at KSU two semesters; Oregon State • Ohio Board of Regents collaborative grant with Bowling Green State University • IMLS Grant • Association of Research Libraries endorsement • Advisory Council • 10 libraries currently involved in a pilot

32. ILL/DD Brief History • ARL/RLG interlibrary loan cost study: a joint effort of the Association of Research Libraries and the Research Libraries Group by Marilyn M. Roche (Washington, DC: Association of Research Libraries, 1993) • 1996 ARL ILL/DD Performance Measures Study • 97 research libraries and 22 Oberlin Group • cost, fill rate, turnaround time, and user satisfaction • Measuring the Performance of Interlibrary Loan Operations in North American Research & College Librariesby Mary Jackson (Washington, DC: Association of Research Libraries, 1998) • 2003 Assessing ILL/DD Services • 75 libraries • Replicate performance measures study • Add mediated and user-initiated (unmediated) component

33. E-Metrics Brief History • ARL Supplementary Statistics tracking expenditures for electronic resources since 1993 • Facilitated retreat at Scottsdale in February 2000 • Contract with the Information Use and Management Policy Institute at Florida State University • Phase One: Environmental Scan • Phase Two: Proposed Measures and Testing • Phase Three: Training Modules • Measures for Electronic Resources (E-Metrics) by Wonsik ‘Jeff’ Shim, Charles McClure, and John Bertot (Washington, DC: Association of Research Libraries, 2002) • 2002-2003 extended pilot with 39 libraries

34. Project COUNTER • Both libraries and publishers need usage statistics • Counting Online Usage of Networked Electronic Resources (COUNTER) • Gain industry support and increase awareness • Release 1 of Code of Practice • Focus on journals and databases • Identify and define data elements to collect • Usage report content, format and delivery • Guidelines for data processing and auditing

35. Project COUNTER • Both libraries and publishers need usage statistics • Counting Online Usage of Networked Electronic Resources (COUNTER) • Gain industry support and increase awareness • Release 1 of Code of Practice • Focus on journals and databases • Identify and define data elements to collect • Usage report content, format and delivery • Guidelines for data processing and auditing

36. LibQUAL+ Brief History • Experience with SERVQUAL in many libraries past 15 years • Texas A&M SERVQUAL assessment • Meeting of interested ARL libraries (ALA Midwinter 2000) • Pilot with 12 ARL libraries (spring 2000) • External funding through FIPSE, U.S. Department of Education (September 2000) • 43 libraries participating spring 2001 (Big12Plus) • 164 institutions participating spring 2002 (OhioLINK, AAHSL) g • 316 institutions participating spring 2003 (NY3Rs, British, Dutch, French language version)

37. LibQUAL+ Brief History • Experience with SERVQUAL in many libraries past 15 years • Texas A&M SERVQUAL assessment • Meeting of interested ARL libraries (ALA Midwinter 2000) • Pilot with 12 ARL libraries (spring 2000) • External funding through FIPSE, U.S. Department of Education (September 2000) • 43 libraries participating spring 2001 (Big12Plus) • 164 institutions participating spring 2002 (OhioLINK, AAHSL) g • 316 institutions participating spring 2003 (NY3Rs, British, Dutch, French language version)

38. e-QUAL Brief History • National Science Digital Library Program at NSF • Building on the LibQUAL+ experience • NSF funding to modify LibQUAL+ for the digital library through NSDL (August 2001) • Participation in the NSDL evaluation workgroup • Understanding NSDL Core Integration Project and distributed collections environment • Qualitative Grounding ongoing • Development of a web-based tool for measuring user perceptions and expectations of the web, digital resource, digital collection, ?

39. Future of New Measures Initiative • Incorporation of some new data elements in traditional surveys • Development of workshops and consulting services for performance measures • Establishment of data gathering and statistical analysis tools

40. “Grand Unified Theory” Collaborative action + Commitment to Learning = Progress

41. What data would support the library’s mission? The Georgetown University Library strives for excellence in acquiring, organizing, interpreting, preserving, and providing access to information resources and services that support the University's curricular and research activities. The Library achieves its mission by: • • Developing and preserving collections that support the curriculum and research interests of the main campus faculty and students• Organizing convenient and seamless access to print, electronic, and other resources to facilitate research and expand scholarship• Teaching patrons to use information resources and technologies effectively and promoting lifelong learning skills• Providing physical and networked environments that encourage study, scholarship, and collaborative learning• Retaining a staff committed to service excellence, continuous learning, and leadership within the library profession• Fostering collaborations throughout the University to create appropriate partnerships and maximize resources• Cultivating effective relationships with administrators, faculty, and students• Cooperating regionally, nationally and internationally with other libraries and consortia to acquire and share collections and resources with the scholarly community.

42. What data should you collect?

43. Self-assessment and value judgements

44. Focus Groups Interviews Structured Unstructured Site Visits Observation Obtrusive Unobtrusive Case Studies Surveys Transaction User Critical Incident Technologies User/Transaction Logs Networked-Based Data Collection Group Process Surveys Scenario Development Group Individual Measurement Tools

45. Standardized tests Pre Post Assignments Papers and essays Oral presentations Demonstrations Exhibitions Portfolios Capstone experiences Surrogates Grades/GPA Self-reports Interviews Methods of Assessing Students

46. Assessment Challenges • Resources (i.e., time and money) • Buy-in • Access to individuals to evaluate • Expertise to conduct evaluation • Project management experience • Appropriate benchmarks • Conceptual clarity • Measurement & design requirements • Instrument validity and reliability

47. The Answer is in Building the Future Library effectiveness is becoming less elusive and increasingly recognizable and actionable … define the actions, the measures, and the future before others define them for you.

48. Martha Kyrillidou, Senior Program Officer for Statistics and Measurement, martha@arl.org Consuella Askew, LibQUAL+™ Program Officer, consuella@arl.org Association of Research Libraries 21 Dupont Circle, Ste 800 Washington, DC 20036 202-296-2296 (phone) 202-872-0884 (fax)