1 / 30

NCknows Evaluation Overview

NCknows Evaluation Overview. Jeffrey Pomerantz, Lili Luo School of Information & Library Science UNC Chapel Hill <pomerantz, luo>@unc.edu. Charles McClure School of Information Studies Florida State University cmcclure@lis.fsu.edu.

adena-chang
Download Presentation

NCknows Evaluation Overview

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NCknows Evaluation Overview Jeffrey Pomerantz, Lili Luo School of Information & Library Science UNC Chapel Hill <pomerantz, luo>@unc.edu Charles McClure School of Information Studies Florida State University cmcclure@lis.fsu.edu Tapping the vast reservoir of human knowledge --Louis Round Wilson, founder, 1931

  2. Evaluation question Is collaborative virtual reference an effective way to meet the information needs of North Carolinians?

  3. Secondary evaluation questions What is required of a library that wishes to offer virtual reference? What is the value added if different types of institutions work together? What is the impact on libraries that provide virtual reference service? Is virtual reference expandable to the whole state? How will this project increase our knowledge of effective organizational models? How can the quality of the reference service provided be measured? Can staff from different types of libraries provide quality reference service to users from other types of libraries? How will the project further greater use of existing resources such as NC LIVE? What partnering or leveraging opportunities exist?

  4. Stakeholders NCknows users The individual participating libraries The entire collaborative effort The State Library of North Carolina Evaluation from all of these perspectives

  5. Data collection methods • Service: statistical analysis • Chat sessions: peer review of transcripts • Patrons: exit surveys & follow-up phone interviews • Librarians: phone interviews

  6. Statistical analysis What patterns are emerging in the volume of questions received by NCknows? Usage over time NCknows librarians vs. 24/7 staff Net asker / Net answerer

  7. Sessions per month

  8. Sessions per day

  9. Sessions per hour: weekdays

  10. Sessions per hour: weekends

  11. Net asker / Net answerer

  12. Transcript peer review What is the quality of reference service being provided to users by NCknows? Overall quality of sessions NCknows librarians vs. 24/7 staff Academic vs. public librarians

  13. NCknows vs. 24/7:Evidence of user satisfaction

  14. Academic vs. Public librarians:Complete and correct answer

  15. Patron exit surveys &Follow-up interviews What are the demographics of the user population? How do users find out about NCknows? What is the users’ level of satisfaction with the reference service being provided by NCknows? What motivates users to use NCknows? How do users use the information provided to them?

  16. Role

  17. Answer completeness

  18. Motivation for the question For a work-related task (49%): business-related and school-related (50/50 split) For Personal Reasons (28%) Known-item search (23%)

  19. Use of the information provided Use: patron had used the information provided and found it useful (61%) Partial use: patron had used the information provided and had found it partially useful, or had partially used the information provided (24%) No use: patron had not used the information provided at all (15%).

  20. Discovery of NCknows Teacher/professor (10%) Search engine (20%) Library materials (70%)

  21. Phone interviews with librarians How has involvement in the NCknows service impacted the participating libraries and librarians?

  22. Training Most useful: Patron-librarian “role playing” Most in need of review: Co-browsing

  23. Policies & Procedures Scheduling Handling email follow-ups Quality control

  24. Tech support • Support: • NCknows: good • 24/7: ok • Infrastructure: • Adequate or better in university & large public libraries • Sometimes less than adequate in community college & small public libraries

  25. Thoughts on chat reference • Inferior to desk reference: • Lacks “non-verbal” cues • Conveys less information in more time • More difficult to conduct an interview • Good for: • Quick answers to well-defined questions Will continue to develop & evolve as an aspect of traditional reference.

  26. Additional & future work Cost/benefit analysis Sustainability Scalability: to the entire state? Situational and contextual factors unique to specific libraries that affect quality of chat reference MISs & databases to relate reference statistics to other library statistics Longitudinal data

  27. Key evaluation issues Understanding the importance of evaluation Ongoing funding/support for evaluation: a “culture of assessment” How will evaluation data be used? Quality of data: both the data reported here, and in other data collection activities

  28. The big picture The need to have ongoing evaluation data Importance of a statewide initiative in digital reference Understanding impacts and applications The context of digital reference efforts elsewhere Ultimately the question is: Do the benefits and outcomes outweigh the costs? Congratulations!

  29. Reports online ils.unc.edu/~jpom/ncknows/

  30. THANKS! Questions or Comments? Jeffrey Pomerantz, Lili Luo School of Information and Library Science UNC Chapel Hill <pomerantz, luo>@unc.edu Charles McClure School of Information Studies Florida State University cmcclure@lis.fsu.edu

More Related