1 / 36

Gabrielle Wong Reference Librarian Hong Kong University of Science and Technology

Enhancing Quality of Library Instruction Programs through Delayed Assessment 6th Northumbria International Conference on Performance Measures in Libraries and Information Services, Durham, UK. August 22-25, 2005. Gabrielle Wong Reference Librarian Hong Kong University of Science and Technology.

cora-schroeder
Download Presentation

Gabrielle Wong Reference Librarian Hong Kong University of Science and Technology

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Enhancing Quality of Library Instruction Programs through Delayed Assessment6th Northumbria International Conference on Performance Measures in Libraries and Information Services, Durham, UK. August 22-25, 2005 Gabrielle Wong Reference Librarian Hong Kong University of Science and Technology

  2. Presentation Outline • Introduction • Background of the Survey • Survey Design and the Questionnaire • Sampling • Key Findings • Future Assessments PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  3. Introduction • A survey was conducted by HKUST Library to assess the enduring impact of its library instruction program • Questionnaires were sent 4-8 weeks after library classes in Fall 2004 • The survey provides data that demonstrates the value of the program, and helps librarians to better understand the characteristics and needs of users • It was a valuable learning process for HKUST librarians; this experience serves as the basis for future assessment endeavour PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  4. HKUST • Founded in 1991 • One of the three research universities in Hong Kong • Undergraduates: 5,500; Postgraduates: 3,000; Faculty: 440 • 4 Schools: Science, Engineering, Business, Humanities & Social Sciences PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  5. Library Instruction at HKUST Library • Serves the whole university community: students, staff and faculty • Over 200 library sessions every year, reaching 4,000 attendees • One-shot sessions, 50-80 minutes • Wide variety: database workshops, course-specific library sessions, topic/subject specific instruction (e.g. job hunting, microforms, web searching, research workshops for postgraduates) PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  6. Background of the Survey Evaluation Practice since 1996 • Evaluation forms were distributed at the end of sessions • Measured users' satisfaction • Instructors' summary for future improvement • Generally, over 80% attendees agreed that the class was worth recommending to their peers • Voluntary evaluation • Results were not analyzed systematically PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  7. Background of the Survey The Needs for a Formal Assessment • Do the library sessions help the attendees in their research and studies? • What do they think after a period of time, during which they are expected to apply the skills for practical purposes? • How can we do better? PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  8. Background of the Survey • The University Librarian, Dr. Samson Soong, recommended the Reference Department to launch a study to answer these questions • The survey aimed to measure the enduring impact of the library program, and produce figures that he may use to demonstrate such value to the university administration • It should also expose the strengths and weaknesses of the program, thus facilitating improvements PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  9. Survey Design: Major Considerations • first attempt of such kind • simple, straightforward and effective • cover as many classes as possible (programmatic) • librarians' workload • demand on users' side • assess the enduring/long term value Methodology adopted: a delayed perception survey PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  10. Survey Design: Delayed Assessment • The survey was administered 4-8 weeks after the sessions • The purpose of the delay period was to allow attendees to apply the skills on their assignments and projects • Skills may be reinforced, proved insufficient or simply forgotten • A mean to assess the enduring value of the classes PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  11. Survey Design: The Questionnaire • 2 pages, 14 questions, should take less than 5 minutes to complete • composed mostly of structured questions rather than open questions to encourage feedback • provided options for open comments whenever possible PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  12. Survey Design: The Questionnaire Perception Questions: • Q.2, Assessing impact: how the class helped/changed the attendees • Q.3, Specific skills learned • Q.4, Skill retention • Q.5, Overall class rating • Q.6. Overall instructor rating • Q.7 Suggestions for improvement PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  13. Survey Design: The Questionnaire Other Questions: • Library use habit • information search confidence • no. of library classes attended previously • research frequency • Demographic • status • place of permanent residence • gender PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  14. Sampling • Selection criteria: • class size > 15 • if it was a course-specific session, there should have been assignment or project associated with it • No intention to mimic the university population • 25 sessions selected: 15 Course-Specific, 10 Open Workshops PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  15. About Class Types PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  16. Responses • 688 questionnaires sent out • Received (Print): 388 (90.7%) • Received (E-mail): 78 (30.0%) • Total: 466 (67.7%) • 89% of respondents attended the library sessions • 11% had not attended the sessions, but answered the demographic and library use habit questions PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  17. Findings: Impact Question 2, rate on a 4-point scale ("1" Strongly Disagree to "4" Strongly Agree) : As a result of the class, I a. learned about sources to find needed information. b. learned about search methods to find needed information. c. learned how to find needed information more quickly. d. was more confident in researching information. e. have increased my interest in using the Library. PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  18. Findings: Impact PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  19. Findings: Impact The differences of these mean scores are statistically significant (p<.05). PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  20. Findings: Skills Learnt Question 3: Please indicate the skills in which this class has helped you (you may check more than one item): a. Choose the best databases for my information need b. Form better search strategies c. Select more appropriate keywords to search d. Export or save my results more effectively e. Find more relevant internet resources f. Use printed resources more effectively g. Others PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  21. Findings: Skills Learnt • Average no. of choices per respondent: 2.06 • Top 3 choices: • Form better search strategies • Find more relevant internet resources • Select more appropriate keywords to search PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  22. Findings: Skills Retention Question 4: Have you continued to use the skills that you learned in this class? PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  23. Findings: Skills Retention Written Comments – "Yes" PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  24. Findings: Skills Retention Written Comments – "No" PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  25. Findings: Overall Ratings Question 5 and 6: 5.What is your OVERALL rating of the class: 6.What is your OVERALL rating of the instructor: Very good Good Above average Average Below average Bad Very bad PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  26. Findings: Overall Ratings PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  27. Findings: Overall Ratings PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  28. Findings: Overall Ratings To benchmark with the grading scale in the University's credit course evaluation: • Very bad=0, Bad=16.7 • Below average=33.3, Average=50, Above average=66.7 • Good=83.3, Very good=100 PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  29. Findings: Suggestions for Improvement Question 7: How could this class be improved? (you may check more than one item) a. I already find the class satisfactory b. Shorter time c. Longer time d. Cover fewer topics, please specify e. Cover more topics, please specify f. Cover some topics in more detail, please specify g. Cover some topics in less detail, please specify h. Instructor should speak more slowly i. Instructor should speak more clearly j. Offer students more hands-on practice k. Have a smaller class l. Others, please specify PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  30. Findings: Suggestions for Improvement • Average no. of choices per respondent: 1.48 • Top Three Choices: PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  31. Findings: Suggestions for Improvement Written Comments: • Fewer topics: 2 commented on the flood of information in the class • More topics: 7 comments, 4 called for more coverage on databases • More details: 14 comments on various issues: 4 about search skills, 2 called for more examples, 2 for databases • Others: 15 various comments, 6 asked for more and better handouts PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  32. Other Observations • Class size: • Bigger classes gave lower class ratings, instructor ratings and lower skill retention • Information search confidence: • Respondents with higher confidence in information search rated higher in these three key indicators PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  33. What the Findings show Us • It demonstrates positive value of the Library's bibliographic program by yielding these figures: • After a period of time, over 85% of users remain affirmative that the class was useful • Almost 70% of users retain the skills learnt • The overall ratings are comparable to credit courses in the University • It tells us what users want: • Smaller class • More hands-on • Better class handouts PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  34. What we Learnt from the Survey • There are significant differences between the Open Workshop Group and the Course-Specific Group • Why users did not use the skills: • did not see the need/did not have the chance • forgot what they learnt • Users with higher confidence in information search are more receptive and appreciative of library instruction • Class size does matter PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  35. Where do We Go from Here? • Nurture a culture of assessment at HKUST Library • Next assessment in 2006 • Administer a comparable immediate evaluation attempting to quantify the "delay" effect • Two separate schemes for CS and OW? • Focus on the outcomes of one particular class instead of a programmatic assessment? • Consider testing attendees' library skills instead of measuring perception? PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library

  36. Thank you!

More Related