Intro - Jan AskAway – Brandon askON – Jan Conclusion – Jan Q&A – you!

3. Intro - Jan • AskAway – Brandon • askON – Jan • Conclusion – Jan • Q&A – you!

4. [http://www.slideshare.net/webbmedia/key-performance-indicator-for-libraries-presentation][http://www.slideshare.net/webbmedia/key-performance-indicator-for-libraries-presentation]

5. Performance Metrics and Value Indicators • Everything is based on user needs. Start with metrics that demonstrate satisfaction of user needs. Photo By Amanda Etches

6. Pearl-grow your metrics and keep developing them. • Link them to tell the bigger story and the value • More than proving value: • Optimize workflow • Engages VR community: partners, staff, &visitors

7. Brandon Weigel, AskAway Coordinator brandonw@eln.bc.ca

8. AskAway Background • Launched in 2006 as public/post-secondary collaborative service • Became post-secondary only in 2010 • 30 post-secondary libraries across BC • Collaboratively staffed by all partners • 7 days a week • 67 service hours per week • 39 weeks per year • 235 staff hours per week • 3-5 staff per shift

9. Funding – Today • Formerly funded by BCcampus – until it dried up • Now transitioning to being fully participant-funded • Result: ever-greater need to prove value!

11. Why do we measure? To prove our value… but what constitutes “value”?

12. Value – Libraries’ perspective: Getting a return on investment: • Students are using the service • Service provided is high quality • Professional development for staff • Worth the time and resources put into it

13. Value – Consortial perspective • Provide consistent, quality service • Limit wait times and costs with appropriate staffing levels • Improvements to cost, staff resources, and overall service by working collaboratively

14. Value – Patrons’ perspective • Contributes to learning and information literacy • Connects researchers with information they need • Leaves patrons satisfied and willing to come back

15. Tools for measuring • Usage statistics (traffic) • Question form fields • Transcripts • Patron exit surveys • Library commitments and cost data • Institution data

16. Usage Stats/Traffic • Counts of all chat sessions: • Initiated by patrons, per patron library • Total handled by each library • Number of own patrons handled, per library • Number of other libraries’ patrons handled, per library • The most commonly-used measure… And the most problematic

18. Usage Stats/Traffic: Uses What it’s good for: • Indicates patron awareness of the service • Measuring effect of promotional activities or website changes • Red flag for technical problems What it’s not good for: Measuring value!

19. Usage Stats/Traffic: Mis-uses • Assessment on service value • “Return on investment” • “If they’re not using it, they don’t need it” • Comparisons with other institutions • “We’re using it less, so we should pay less” • Says nothing about quality of interaction, type of question, student outcomes

20. Usage Stats/Traffic: Value Why is usage a poor indicator of value? • No correlation between usage and patron value indicators • Usage primarily correlates with: • Visibility on library website • Promotional activities

21. Usage Stats: Visibility Examples Qwidget added to EBSCO search results

22. Usage Stats: Visibility Examples Website redesign cuts Qwidgets and links

23. Usage Stats: Visibility Examples Qwidget added to Discovery Layer search results Discovery Layer launched

24. Usage Stats/Traffic Takeaway: Usage is under your control! (So it can’t tell you much about value.)

26. Question Form Fields • Detailed data about individual questions • Key fields: • Timestamp • Patron’s institution • Referring URL • Session length • Wait time • Resolution code • Descriptive codes

28. Form Fields: Uses What it’s good for: • Identifying and tracking problems • Estimate the depth of the questions • Find referral hotspots • Top referrers: database search results, research guides, catalogue, database lists, discovery layer results… • Essentially, pages where people are actively researching • Determine staffing needs for each shift

29. Form Fields: Uses

30. Form Fields: Uses Descriptive Codes: added by librarian post-chat • Used to identify the type(s) of question • Core codes: research, ready reference, directional, citation, technical, e-resource access, prank • Plus 16 additional codes Useful for: Debunking myths! (And demonstrating value)

32. Form Fields: Cautions • Good data, but incomplete: • Descriptive codes not always applied by busy librarians • Currently, 70% rate of applying codes (not bad) • Not much user-submitted information captured • Difficult to read – which discourages use

33. Form Fields: What I wish we could do • Compare data with other reference venues • Collect more data! • Get more people using it

34. Transcripts • Useful for: • Examining problematic interactions • Examples for training • Potentially excellent for measuring service quality (But we don’t do that)

35. Transcripts: Cautions • Strong potential, but untapped • 26/27 polled libraries do not use transcripts for staff evaluation • General sense: “It’s creepy” • Sense of being watched could hurt staff support • Comparisons harm collaboration • Result: Opinions on quality are based on feelings, not on data

36. Transcripts: In an ideal world… The impossible dream: • Consistent reference standards, with library buy-in • Non-invasive transcript analysis (impossible?) • Work with coordinators to apply those standards More realistic: • Annual anonymized random sample analysis

37. Exit Survey • Began in 2008 to measure patron satisfaction, and to collect qualitative data and useful quotes • Rewritten in 2013 to measure learning outcomes

38. Exit Survey: What it tells us • How they discovered AskAway • Why they’re using AskAway • Satisfaction level • Likelihood of returning • What could be improved • Learning outcomes • Demographic info • General comments

39. Exit Survey: Why it’s useful • Satisfaction level: rough measure of service quality • 2014: 90% high satisfaction; 93% likely to return • Demographics: Tells us who our patrons really are, and how different groups use AskAway

40. “Super fantastic help. Friendly, positive and informative. Exactly what I needed and didn't take much longer than a phone call. Thank you!” - TRU grad student “I was astounded at the ease with which the librarian found a specific paper for me, based on an incomplete name, an erroneous date, and context. This is a great service!” - UBC Faculty

42. "Thank you so much for your help once again. I really appreciate how you 'teach' us students strategies to use rather than just giving us the answer. Proves to be really effective in my learning!” - UFV Student

45. Exit Survey: What I wish we could do • Link survey responses to transcripts • Improve response rates: • Send dialog box to user when closing browser window • Other ways?? • Post monthly survey analyses, not just tables

46. Inputs and Outputs: Return on Investment • NOT ABOUT HOW MANY CHATS YOU GET • Inputs vs. Outputs • Staffing: • Commitment hours per week: 3 to 34 (based on size) • Majority contribute 3 or 5 hours weekly • Return on investment: 235 staff hours per week • 510% to 6830% return on staff time • PLUS extended reference hours

47. Inputs and Outputs: Return on Investment – Financial

49. What I wish we could measure • Larger student impact • Assignment scores • Long-term learning • Reasons people don’t use AskAway • Overall awareness of the service • Quality of chats • Librarian performance

50. Takeaways • The easiest tool to use is rarely the best for the job • Choose metrics that tell you want you want to know • Modify your tools to fit the problem • Usage is under your control • Think broadly about the investment