300 likes | 391 Views
NCknows Evaluation Overview. Jeffrey Pomerantz, Lili Luo School of Information & Library Science UNC Chapel Hill <pomerantz, luo>@unc.edu. Charles McClure School of Information Studies Florida State University cmcclure@lis.fsu.edu.
E N D
NCknows Evaluation Overview Jeffrey Pomerantz, Lili Luo School of Information & Library Science UNC Chapel Hill <pomerantz, luo>@unc.edu Charles McClure School of Information Studies Florida State University cmcclure@lis.fsu.edu Tapping the vast reservoir of human knowledge --Louis Round Wilson, founder, 1931
Evaluation question Is collaborative virtual reference an effective way to meet the information needs of North Carolinians?
Secondary evaluation questions What is required of a library that wishes to offer virtual reference? What is the value added if different types of institutions work together? What is the impact on libraries that provide virtual reference service? Is virtual reference expandable to the whole state? How will this project increase our knowledge of effective organizational models? How can the quality of the reference service provided be measured? Can staff from different types of libraries provide quality reference service to users from other types of libraries? How will the project further greater use of existing resources such as NC LIVE? What partnering or leveraging opportunities exist?
Stakeholders NCknows users The individual participating libraries The entire collaborative effort The State Library of North Carolina Evaluation from all of these perspectives
Data collection methods • Service: statistical analysis • Chat sessions: peer review of transcripts • Patrons: exit surveys & follow-up phone interviews • Librarians: phone interviews
Statistical analysis What patterns are emerging in the volume of questions received by NCknows? Usage over time NCknows librarians vs. 24/7 staff Net asker / Net answerer
Transcript peer review What is the quality of reference service being provided to users by NCknows? Overall quality of sessions NCknows librarians vs. 24/7 staff Academic vs. public librarians
Patron exit surveys &Follow-up interviews What are the demographics of the user population? How do users find out about NCknows? What is the users’ level of satisfaction with the reference service being provided by NCknows? What motivates users to use NCknows? How do users use the information provided to them?
Motivation for the question For a work-related task (49%): business-related and school-related (50/50 split) For Personal Reasons (28%) Known-item search (23%)
Use of the information provided Use: patron had used the information provided and found it useful (61%) Partial use: patron had used the information provided and had found it partially useful, or had partially used the information provided (24%) No use: patron had not used the information provided at all (15%).
Discovery of NCknows Teacher/professor (10%) Search engine (20%) Library materials (70%)
Phone interviews with librarians How has involvement in the NCknows service impacted the participating libraries and librarians?
Training Most useful: Patron-librarian “role playing” Most in need of review: Co-browsing
Policies & Procedures Scheduling Handling email follow-ups Quality control
Tech support • Support: • NCknows: good • 24/7: ok • Infrastructure: • Adequate or better in university & large public libraries • Sometimes less than adequate in community college & small public libraries
Thoughts on chat reference • Inferior to desk reference: • Lacks “non-verbal” cues • Conveys less information in more time • More difficult to conduct an interview • Good for: • Quick answers to well-defined questions Will continue to develop & evolve as an aspect of traditional reference.
Additional & future work Cost/benefit analysis Sustainability Scalability: to the entire state? Situational and contextual factors unique to specific libraries that affect quality of chat reference MISs & databases to relate reference statistics to other library statistics Longitudinal data
Key evaluation issues Understanding the importance of evaluation Ongoing funding/support for evaluation: a “culture of assessment” How will evaluation data be used? Quality of data: both the data reported here, and in other data collection activities
The big picture The need to have ongoing evaluation data Importance of a statewide initiative in digital reference Understanding impacts and applications The context of digital reference efforts elsewhere Ultimately the question is: Do the benefits and outcomes outweigh the costs? Congratulations!
Reports online ils.unc.edu/~jpom/ncknows/
THANKS! Questions or Comments? Jeffrey Pomerantz, Lili Luo School of Information and Library Science UNC Chapel Hill <pomerantz, luo>@unc.edu Charles McClure School of Information Studies Florida State University cmcclure@lis.fsu.edu