Assessment of Student Learning From Reference Service

1. Assessment of Student Learning From Reference Service Jill Gremmels and Karen Lehmann Wartburg College Waverly, Iowa RUSA Reference Research Forum 2005 Chicago, June 25, 2005

2. Survey of the Literature on Reference Evaluation • Overview by Green & Peach (2003) • Reference as a teaching & learning activity • Review essays: • Bunge (1994), Smith (1991), Campbell (2000) • Traditional techniques: • Tally questions, such as length, time, date, etc. • User interviews (Mendelsohn, 1997) • Focus groups (Massey-Burzio, 1998) • Survey/Observation (Norlin, 2000)

3. Category 1: Reference Evaluation Literature • “55 percent school” • Hernon & McClure (1986) • Reference service evaluated on accuracy of responses to predetermined questions. • Questions come from “unobtrusive” people posing as patrons. • Critics: Reference is more than right and wrong answers. • Critics: If reference is communication, delivery can be as important as answers.

4. Category 2: Reference Evaluation Literature • Interpersonal communication • Durrance (1989; 1995); Jardine (1995) • More complex model. • Focuses on willingness of patrons to return to the same staff member for future needs. • Critics: Wrong answers are still wrong, even from a librarian you liked.

5. Category 3: Reference Evaluation Literature • User satisfaction + conditions of the reference transaction • Bunge; Murfin (1995); Stalker & Murfin (1995) • Based on Wisconsin-Ohio Reference Evaluation Program. • 2 part evaluation form: patron and librarian. • Allows variables (# of resources used, is library busy, subject area of inquiry, training of librarian). • Externally validated assessment instrument (can match to other libraries).

6. Shortcomings of all 3 • Correctness: • Don’t reflect reality of academic reference work. • Performance: • Approaches don’t offer proof of teaching activity of librarians. • Assessment: • Don’t connect to learning outcomes.

7. Assessment Applied to Learning • Outcomes Assessment is: • Knowing what you are doing. • Knowing why you are doing it. • Knowing what students are learning as a result. • Changing because of the information. Debra Gilchrist, Pierce Colleges, Lakewood and Puyallup, WA

8. History of Assessment in Higher Education (Ewell) • AAHE Assessment Conferences ran 1985-2004. • Assessment is still not a “culture of use” (Ewell, 2002), embedded into lives and work. • Dueling agendas: institutional change vs. “accountability for results.” • Summative or formative? • Basic (and still unanswered) question: Is there any real evidence that assessment actually promotes better learning?

9. Tying Academic Libraries into Campus Assessment • Quantitative vs. qualitative • By Services: • Adams (1996); Wallace (2001) • By Learning: • Iannuzzi (1999); Breivik (1998); Boyer Commission Report • Gremmels & Ruediger (2003); Lopez (2002) • If information literacy library culture is strong, it will influence campus culture.

10. Building Foundations for Assessment Reference Course-Related Credit Courses Information Literacy Outcomes Information Literacy Definition Mission Debra Gilchrist Pierce College Philosophy

11. Reference Mission Statement • Vogel Library’s mission is to educate information-literate lifelong learners. We strive to make each reference encounter an educational experience that reinforces information literacy concepts by building upon prior instruction and giving further opportunities for guided practice.

12. The Teaching Role of Reference: Can it be Assessed? • Elmborg (2002) • Librarians need pedagogy for the reference desk. • This implies reference is a form of teaching. • Academic librarians should approach reference transactions as academic conferences where teaching and learning occur. • Green & Peach (2003) • Assumed this connection and wanted to validate it for purposes of librarian evaluation.

13. The Wartburg Study • Wanted to know WHAT the student learned. • Does reference reinforce classroom instruction? • Based on outcomes (Info Lit Competency Standards) • They set our agenda • Create our vision • Focus our teaching • Provide our common purpose • Guide our students Debra Gilchrist, Pierce Colleges, Lakewood and Puyallup, Washington

14. Methodology • Paired and numbered surveys. • Librarian and student both answered after a reference encounter. • Questions developed by reference librarians. • Based on information literacy concepts taught in the classroom.

15. Methodology • Teaching “intentions” vocabulary: • choosing good search terms • database selection • search strategy • evaluating information • how to use a specific tool • other (specify) • No limits on taking survey multiple times.

16. Methodology • Student questionnaire Spring 2003: • Tried to tie reference encounters to prior instruction. • 143 returned (85% response rate). • Librarian questionnaire Spring 2003: • Six choices for instructional outcomes. Only one answer allowed. • 169 returned.

17. Methodology • 2003-04 Academic Year: • Student questionnaire: • Added demographic questions • Asked student to describe instruction and assign a category • 121 returned (78% response rate). • Librarian questionnaire: • Only change was allowing multiple answers if ranked. • 156 returned.

18. January-May 2003 141 useable responses (143 returned). Four librarians administered the survey. 3 of the 4 participate in ILAC classroom instruction. 2003-2004 Academic Year 121 useable responses Same four librarians administered the survey. Methodology

19. Data Analysis – First Survey • Surveys re-paired. • Answers entered into spreadsheet and narrative list and tallied. • Two librarians independently judged congruence as “related,” “not related,” or “inconclusive.” High inter-rater reliability. • Discussed and resolved differences.

20. Data Analysis – Second Survey • Surveys re-paired. • Answers entered into spreadsheet and Qualrus • Category matches identified. • Researchers independently analyzed forms for “strong,” “acceptable” or “no” description matches. Inter-rater reliability high. • Discussed and resolved differences.

21. Description Match Examples • Student: how to use the catalog • Librarian: iPac to find music CDs • Student: how to cite CQ Researcher • Librarian: citation with CQ • Student: how to find literary criticism • Librarian: Literature Resource Center and Contemporary Literary Criticism

22. Student: Showed me different places I could look for the information I was seeking. Librarian: Oxford Reference Online; defining "liberal arts" Student: He told me some important information about companies on websites recommended by the college. Librarian: LexisNexis Business and Business Source Elite Non-Match Examples

23. Demographics

24. Q. 1: Did the librarian who just helped you teach you anything while answering your question?

25. Q. 3: Did a librarian meet with your class and teach your class how to find information for this assignment?

26. Q. 4: Did what the librarian taught you relate to or build on anything a librarian taught your class?

27. Q. 5: Did what the librarian taught relate to or build on a previous lesson?

28. Students made the link – sometimes.

29. Link stronger with tools

30. What We Learned • Students understand reference to be an instructional activity. • Information Literacy instruction does seem to yield results. • Reference helps students practice and reinforce information literacy knowledge.

31. Reference Applications • Began a three-tier reference model. • 1--Frontline: Tool instruction on demand • 2--Backup: Concepts on call • 3--In-Depth: Consultations as scheduled • All staff participate.

32. Reference Applications • Focus on reference as guided practice. • Embrace role as facilitator. • Be more explicit about links to classroom instruction. • Brief questions about process and concepts. • Graphic organizers.

33. Reference Applications • Have declined to participate in consortial virtual reference project. • Beginning a campaign to promote consultations: • PSA • Faculty awareness • New librarian will create online tutorials.

34. Classroom Applications • Continue to seek information literacy opportunities in classes. • especially as survey data reveal which courses students are working on. • Have all staff sit in on information literacy lessons. • so they know what is being taught and can reinforce better.

35. Further Research • Other academic libraries adapt our form and do their own assessments? • Assess consultations at Wartburg

36. Contact Information Jill Gremmels (319) 352-8462 jill.gremmels@wartburg.edu Karen Lehmann (319) 352-8460 karen.lehmann@wartburg.edu