Quantitative and Qualitative Methods for Assessing and Improving Outcomes and Service Quality

1. Quantitative and Qualitative Methods for Assessingand Improving Outcomes and Service Quality Ottawa, Canada June 14, 2006 Presented by: Dr. Colleen Cook, Dean Texas A&M University

2. Why Assess? “In an age of accountability, there is a pressing need for an effective…process to evaluate and compare research libraries.” • 700 participants in LibQUAL+™ • 123 Association of Research Libraries (ARL) alone, over $3.4 billion dollars were expended in 2003/2004 Note. M. Kyrillidou and M. Young. (2005). ARL Statistics 2003-04. Washington, D.C.: ARL, p.5.

3. Libraries Remain a CredibleResource in 21st Century 98% agree with statement, “My … library contains information from credible and known sources.” Note. Digital Library Federation and Council on Library and Information Resources. (2002). Dimensions and Use of the Scholarly Information Environment.

4. Changing Behaviors Recent Survey: Only 15.7% agreed with the statement “The Internet has not changed the way I use the library.” Note. Digital Library Federation and Council on Library and Information Resources. (2002). Dimensions and Use of the Scholarly Information Environment.

5. Faculty: Dependence onElectronic Resources Will Increase “I will become increasingly dependent on electronic research resources in the future.” http://www.arl.org/arl/proceedings/144/guthrie_files/guthrie.ppt

6. Research Behavior:Personal Control When searching for print journals for research: • Only 13.9% ask a librarian for assistance • Only 3.2% consider consulting a librarian a preferred way of identifying information Note. Digital Library Federation and Council on Library and Information Resources. (2002). Dimensions and Use of the Scholarly Information Environment.

7. Total Circulation Note. M. Kyrillidou and M. Young. (2005). ARL Statistics 2003-04. Washington, D.C.: ARL, p.6.

8. Reference Transactions Note. M. Kyrillidou and M. Young. (2005). ARL Statistics 2003-04. Washington, D.C.: ARL, p.6.

9. Web UsageTotal File Requests - UT Austin Libraries 2000-2003

10. EnterLibQUAL+™ • The necessity of assessment • Rapid shifts in information-seeking behavior • The reallocation of resources from traditional services and functions

11. The Challenge ofAssessment in Libraries • Traditional statistics emphasize inputs, expenditures, acquisitions, holdings, etc. • Helping funding agencies understand success of investment • No demonstrable relationship between expenditures and service quality • Lack of metrics describing outcomes: success from the user’s point of view • Need to redesign library services to better meet changing patterns of use • Building the climate, tools, and skill set for library assessment

12. ARL New MeasuresInitiative • Collaboration among member leaders with strong interest in this area • Specific projects developed with different models for exploration • Intent to make resulting tools and methodologies available to full membership and wider community

13. LibQUAL+™ Goals • Improve mechanisms and protocols for evaluating libraries • Develop web-based tools for assessing library service quality • Identify best practices in providing library service • Support libraries seeking to understand changes in user behavior • Assist libraries seeking to re-position library services in the new environment

14. LibQUAL+™ Outcomes • Securing information that contributes meaningfully to planning and improvement efforts at a local level • Providing analytical frameworks that institutional staff can apply without extensive training or assistance • Helping decision-makers understand success of investments • Finding useful inter-institutional comparisons

15. The LibQUAL+™ Premise “….only customers judge quality; all other judgments are essentially irrelevant” PERCEPTIONS SERVICE Note. Zeithaml, Parasuraman, Berry. (1999). Delivering quality service. NY: The Free Press.

16. 13 Libraries English LibQUAL+™ Version 4000 Respondents LibQUAL+™ Project PURPOSEDATAANALYSISPRODUCT/RESULT Emergent Describe library environment; build theory of library service quality from user perspective Test LibQUAL+™ instrument Refine theory of service quality Refine LibQUAL+™ instrument Test LibQUAL+™ instrument Refine theory 2000 Unstructured interviews at 8 ARL institutions Web-delivered survey Unstructured interviews at Health Sciences and the Smithsonian libraries E-mail to survey administrators Web-delivered survey Focus groups Content analysis: (cards & Atlas TI) Reliability/validity analyses: Cronbachs Alpha, factor analysis, SEM, descriptive statistics Content analysis Content analysis Reliability/validity analyses including Cronbachs Alpha, factor analysis, SEM, descriptive statistics Content analysis QUAL QUAN QUAL QUAL QUAN QUAL Case studies1 Valid LibQUAL+™ protocol Scalable process Enhanced understanding of user-centered views of service quality in the library environment2 Cultural perspective3 Refined survey delivery process and theory of service quality4 Refined LibQUAL+™ instrument5 Local contextual understanding of LibQUAL+™ survey responses6 Iterative Vignette Re-tooling 2005 700 Libraries English, Dutch, Swedish, German LibQUAL+™ Versions 160,000 anticipated respondents

17. 76 Interviews Conducted • University of Minnesota • University of Pennsylvania • University of Washington • Smithsonian • Northwestern Medical • York University • University of Arizona • Arizona State • University of Connecticut • University of Houston • University of Kansas

18. LoadedPT:P1:01xxxxxxxxxxxxxx.txt,S:\Admin\Colleen\ServQual Interviews\TEXT Only\01xxxxxxxxx.txt (redirected: c:\zz\atlasti\fred

20. Library Service Quality Information Affect of Service Control Empathy Scope of Content Responsiveness Convenience Assurance Ease of Navigation Library as Place Reliability Timeliness Utilitarian space Equipment Symbol - Self Reliance Refuge Model 3 Dimensions ofLibrary Service Quality

21. Affect of Service “I want to be treated with respect. I want you to be courteous, to look like you know what you are doing and enjoy what you are doing. … Don’t get into personal conversations when I am at the desk.” Faculty member

22. Library as Place “One of the cherished rituals is going up the steps and through the gorgeous doors of the library and heading up to the fifth floor to my study. … I have my books and I have six million volumes downstairs that are readily available to me in an open stack library.” Faculty member

23. Library as Place “I guess you’d call them satisfiers. As long as they are not negatives, they won’t be much of a factor. If they are negatives, they are a big factor.” Faculty member

24. Information Control “…first of all, I would turn to the best search engines that are out there. That’s not a person so much as an entity. In this sense, librarians are search engines [ just ] with a different interface.” Faculty member

25. Information Control “By habit, I usually try to be self-sufficient. And I’ve found that I am actually fairly proficient. I usually find what I’m looking for eventually. So I personally tend to ask a librarian only as a last resort.” Graduate student

26. Multiple Methodsof Listening to Customers • Transactional surveys* • Mystery shopping • New, declining, and lost-customer surveys • Focus group interviews • Customer advisory panels • Service reviews • Customer complaint, comment, and inquiry capture • Total market surveys* • Employee field reporting • Employee surveys • Service operating data capture *A SERVQUAL-type instrument is most suitable for these methods Note. A. Parasuraman. The SERVQUAL Model: Its Evolution And Current Status. (2000). Paper presented at ARL Symposium on Measuring Service Quality, Washington, D.C.

27. LibQUAL+™ Resources • An ARL/Texas A&M University joint developmental effort based on SERVQUAL. • LibQUAL+™ initially supported by a 3-year grant from the U.S. Department of Education’s Fund for the Improvement of Post-Secondary Education (FIPSE) • Initial project established a expert team, re-grounded SERVQUAL concepts, and designed survey methodology • Survey conducted at over 500 libraries resulting in a data base of over half a million user responses • NSF funded project to refocus LibQUAL+™ on the National Science Digital Library (NSDL)

28. Participating Libraries World LibQUAL+™ Survey 2005

29. Languages American English British English French Dutch Swedish In development Chinese Greek Spanish German Consortia Each may create 5 local questions to add to their survey Types of Institutions Academic Health Sciences Academic Law Academic Military College or University Community College European Business Hospital Public State Countries U.S., U.K., Canada, the Netherlands, South Africa, Sweden, France, Australia, New Zealand, Malaysia Rapid Growth in Other Areas

30. “22 items”

31. Survey Instrument

32. “And a Box” Why the Box is so Important • About 40% of participants provide open-ended comments, and these are linked to demographics and quantitative data. • Users elaborate the details of their concerns. • Users feel the need to be constructive in their criticisms, and offer specific suggestions for action.

33. Reliabilityalpha By Language By Language Service Info. Lib as Group n Affect Control Place TOTAL American (all) 59,318 .95 .91 .88 .96 British (all) 6,773 .93 .87 .81 .94 French (all) 172 .95 .90 .89 .95

34. Reliability alpha by University Type By University Type Service Info. Lib as Group n Affect Control Place TOTAL Comm Colleges 4,189 .96 .92 .89 .97 4 yr Not ARL 36,430 .95 .91 .88 .96 4 yr, ARL 14,080 .95 .90 .87 .96 Acad Health 3,263 .95 .92 .90 .96

35. Validity Correlations Validity Correlations Serv_Aff Info_Con LibPlace TOTALper Serv_Aff 1.0000 .7113 .5913 .9061 Info_Con .7113 1.0000 .6495 .9029 LibPlace .5913 .6495 1.0000 .8053 TOTALper .9061 .9029 .8053 1.0000 ESAT_TOT .7286 .6761 .5521 .7587 EOUT_TOT .5315 .6155 .4917 .6250

36. Understanding LibQUAL+™ Results • Measures the distance between minimally acceptable and desired service quality ratings • Perception ratings ideally fall within the Zone of Tolerance

37. Key to Bar Charts

38. LibQUAL+™ 2004 Summary Collegesor Universities American English (n = 69,449)

39. Score Norms • Norm Conversion Tables facilitate the interpretation of observed scores using norms created for a large and representative sample. • LibQUAL+™ norms have been created at both the individual and institutional level

40. Institutional Norms for PerceivedMeans on 25 Core Questions Note: Thompson, B. LibQUAL+ Spring 2002 Selected Norms, (2002).

41. LibQUAL+™ InteractiveInstitution Statistics

42. Adequacy GapThe difference between the minimum and perceived score LibQUAL+™

43. In Closing LibQUAL+™ • Focuses on success from the users point of view (outcomes) • Demonstrates that a web-based survey can handle large numbers; users are willing to fill it out; and survey can be executed quickly with minimal expense • Requires limited local survey expertise and resources • Analysis available at local and inter-institutional levels • Offers many opportunities for using demographics to discern user behaviors

44. LibQUAL+™ Resources • LibQUAL+™ Website:http://www.libqual.org • Publications:http://www.libqual.org/publications • Events and Training: http://www.libqual.org/events • LibQUAL+™ Bibliography: http://www.coe.tamu.edu/~bthompson/servqbib • LibQUAL+™ Procedures Manual:http://www.libqual.org/Information/Manual/index.cfm

45. LibQUAL+™Contact Information • MaShana Davis • Technical Communications Liaison • mashana@arl.org • Richard Groves • Customer Relations Coordinator • richard@arl.org • Mary Jackson • LibQUAL+™ Services Manager • richard@arl.org • Martha Kyrillidou • Director, ARL Statistics and Service Quality Programs • martha@arl.org woof