Marie L. Radford, Ph.D. Associate Professor, Rutgers, The State University of New Jersey

1. E-Valuating Virtual Viewpoints: User, Non-User, and Librarians Perspectives on Live Chat-Based Reference Marie L. Radford, Ph.D. Associate Professor, Rutgers, The State University of New Jersey Lynn Silipigni Connaway, Ph.D. Senior Research Scientist, OCLC Panel: E-Valuating E-Reference: Transforming Digital Reference through Research and Evaluating ASIST Annual Meeting October 24-29 2008 Columbus, OH

2. Seeking Synchronicity: Evaluating Virtual Reference Services from User, Non-User, and Librarian Perspectives $1,103,572 project funded by Institute of Museum and Library Services Rutgers, The State University of New Jersey OCLC, Online Computer Library Center, Inc. Four phases: Focus group interviews Analysis of 850 QuestionPoint live chat transcripts Online surveys 176 VRS librarians 184 VRS non-users 137 VRS users Telephone interviews 283 Total

3. Critical Incident Technique • Flanagan, 1954 • Qualitative technique • Focuses on most memorable event/experience • Allows categories or themes to emerge rather than be imposed

4. Online Survey CI Questions Librarians & Users Think about one experience in which you felt a chat reference encounter achieved (or did not achieve) a positive result Non-users Think about one experience in which you felt you achieved (did not achieve) a positive result after seeking library reference services in any format

5. Interpersonal Communication Analysis: Results Relational Facilitators Interpersonal aspects of chat conversation that have apositiveimpact on librarian-client interaction & thatenhancecommunication. Relational Barriers Interpersonal aspects of chat conversation that have anegativeimpact on librarian-client interaction & thatimpedecommunication.

6. Relational Theory & Approach to Interpersonal Communication Every message has dual dimensions – both content & relational (Watzlawick, Beavin, & Jackson, 1967)

7. Librarians: Positive Result(CI N=142) Number % • Both Relational & 85 60% Content • Primarily Content 54 38% • Primarily Relational3 2%

8. Librarians: Positive Result(CI N=142) Relational Themes* Number % • Attitude 69 49% • Relationship quality 33 23% • Familiarity 3 2% *The percentages do not total to 100% because each CI can be coded into more than one theme

9. Librarians: Positive Result(CI N=142) Content Themes* Number % • Providing information 120 85% • Providing instruction 49 35% • Demonstrating knowledge 14 10% • Convenience/multi- 10 7% tasking/ time or money saving *The percentages do not total to 100% because each CI can be coded into more than one theme

10. Librarians: Negative Result(CI N=124) Number % • Primarily Relational53 43% • Primarily Content 40 32% • Both Relational & 31 25% Content

11. Librarians: Negative Result(CI N=124) Relational Themes* Number % • Attitude 67 54% • Relationship quality 28 23% • Impact of technology 7 6% • Approachability 1 1% *The percentages do not total to 100% because each CI can be coded into more than one theme

12. Librarians: Negative Result(CI N=124) Content Themes* Number % • Lack of information 64 52% • Lack of knowledge 15 12% • Task unreasonable 1 1% *The percentages do not total to 100% because each CI can be coded into more than one theme

13. Users: Positive Result(CI N=129) Number % • Primarily Content 79 61% • Both Relational & 33 26% Content • Primarily Relational 17 13%

14. Users: Positive Result(CI N=129) Content Themes* Number % • Providing information91 71% • Convenience/multi- 36 28% tasking/time saving/ money saving • Providing instruction14 11% • Demonstrating knowledge7 5% *The percentages do not total to 100% because each CI can be coded into more than one theme

15. Users: Positive Result(CI N=129) Relational Themes* Number % • Attitude 36 28% • Relationship quality 21 16% • Impact of technology 1 1% *The percentages do not total to 100% because each CI can be coded into more than one theme

16. Users: Negative Result(CI N=68) Number % • Primarily Content 46 68% • Primarily Relational15 22% • Both Relational & 7 10% Content

17. Users: Negative Result(CI N=68) Content Themes* Number % • Lack of information 48 71% • Lack of knowledge 8 12% *The percentages do not total to 100% because each CI can be coded into more than one theme

18. Users: Negative Result(CI N=68) Relational Themes* Number % • Relationship quality 17 25% • Attitude 13 19% • Approachability 1 1% • Impact of Technology 1 1% *The percentages do not total to 100% because each CI can be coded into more than one theme

19. Non-users: Positive Result(CI N=154) Number % • Primarily Content79 51% • Both Relational & 48 31% Content • Primarily Relational2718%

20. Non-users: Positive Result(CI N=154) Content Themes* Number % • Providing information 75 49% • Providing instruction 35 23% • Demonstrating knowledge 21 14% • Convenience/multi- 18 12% tasking/time saving/ money saving *The percentages do not total to 100% because each CI can be coded into more than one theme

21. Non-users: Positive Result(CI N=154) Relational Themes* Number % • Attitude 51 33% • Impact of FtF assisting 32 21% relationship development • Relationship quality 25 16% • Impact of phone/Email 5 3% assisting information seeking process • Approachability 4 3% • Familiarity 1 1% *The percentages do not total to 100% because each CI can be coded into more than one theme

22. Non-users: Negative Result(CI N=100) Number % • Primarily Content 52 52% • Primarily Relational33 33% • Both Relational & 15 15% Content

23. Non-users: Negative Result(CI N=100) Content Themes* Number % • Information 60 60% • Lack of knowledge 24 24% • Instruction 9 9% • Task unreasonable 4 4% *The percentages do not total to 100% because each CI can be coded into more than one theme

24. Non-users: Negative Result(CI N=100) Relational Themes* Number % • Attitude 47 47% • Relationship quality 24 24% • Approachability 3 3% • Impact of technology 2 2% *The percentages do not total to 100% because each CI can be coded into more than one theme

25. Implications: Librarians • Value • Delivery of accurate answers/ information • Polite, interested users • Find rude or impatient users disruptive to chat success

26. Implications: Users & Non-Users • Value • Accuracy of answers/information • Delivery of specific content • Knowledge of sources & systems • Positive attitude • Good communication skills • Younger VRS users • Impatient & want info delivered quickly - no fuss • Not as concerned as librarians w/ instruction

27. Recommendations • Provide • Specific info • Variety of formats • Friendly & courteous service • Marketing to non-users • User education needed for more realistic expectations • Do not force instruction unless wanted

28. Future Directions Online survey results informed 283 telephone interviews • Collected more critical incidents • Analysis in progress

29. Future Directions • Write, write, write!

30. Special Thanks Rutgers University & OCLC Grant Project Team Project Managers: Jocelyn DeAngelis Williams Timothy J. Dickey Research Assistants: Patrick A. Confer David Dragos Jannica Heinstrom Vickie Kozo Mary Anne Reilly Lisa Rose-Wiles Susanna Sabolsci-Boros Andrea Simzak Julie Strange Janet Torsney

31. End Notes This is an updated version of a presentation given at ALISE 2008 This is one of the outcomes from the project Seeking Synchronicity: Evaluating Virtual Reference Services from User, Non-User, and Librarian Perspectives Funded by IMLS, Rutgers University, & OCLC Online Computer Library Center, Inc. Slides available at project web site:http://www.oclc.org/research/projects/synchronicity/