Measuring the impact of information skills sessions: pre- and post-testing student competence

1. Measuring the impact of information skills sessions: pre- and post-testing student competence Anne Worden Subject Librarian: Languages, Geography, Social and Political Studies University of Portsmouth

2. Who is being tested? • Students in 2nd year of languages and area studies degree courses • All are about to begin dissertation research • Research unit contains 3 x 50 minute sessions taught by subject librarian • 509 students have completed the pre-test from Feb 2001-2005 • 360 have completed the post-test from March 2001-2005

3. What is the test? • 14 multiple choice questions (MCQs) • 6 are “pure” MCQs • 8 are multiple response MCQs • Originally some MCQs had 5 options, some had 4 and 1 MCQ had 3 options • In 2005 standardised on 4 options for all except last MCQ which had 3 options • Questions piloted on academic staff • Pre-test taken as part of introduction to Research Management unit 2 weeks prior to library input • Post-test uses identical questions and is taken at the end of the last librarian-led session

4. Reasons for testing • To find out what the students know already • Most have had library input in their 1st year • Some are direct entrants from Europe • To make students aware of knowledge gaps which need filling in order to complete a dissertation successfully • Academic staff believe it’s a useful motivational tool • Post-test enables me to check the effectiveness of my teaching • Evidence-based practice rather than satisfaction ratings

5. What topics are covered in the test? • Relevant databases and their content • Efficient search strategies • Evaluating websites from an academic perspective • Referencing – journals and Internet • Dissertation-focused locating and accessing info – COPAC, ILLs, past dissertations

6. Sample question: Efficient searching • When using databases, certain shortcuts help you get good results more quickly. Which of the following statements are true? • Using the Boolean operator AND helps you get more focused results • Using the Boolean operator OR helps you narrow down your search • Using a truncation character (e.g.*) helps you find alternative word endings • Using a wildcard character (e.g.?) helps you find alternative spellings.

7. Student responses: Efficient searching • Pre-test: • 1% of students got all 3 correct responses • 19% chose the wrong answer • 34% gave no answer • Post-test: • 62% of students got all 3 correct responses • 3% chose the wrong answer • 1% gave no answer N=99 OR ?

8. Sample question: Web page created by a subject expert? • Which of the following factors help you judge whether those who have created a Web page are authorities on the subject being discussed? (Only 1 is correct) • They have many books and articles in print • They have published a lot on the Web • The information is well organised • The page has a Web award which is displayed on the screen Question reproduced with permission from RDN Virtual Training Suite Internet for Historians http://www.vts.rdn.ac.uk

9. Student responses: Is a Web page created by a subject expert?

10. Sample question – URL knowledge • Here are some statements about Internet addresses (URLS). Tick the only one that is correct! • If a URL no longer works, the item has been removed from the Internet • If a resource has the URL http://www.detr.gov.uk/annual00/index.htm it is located on a UK government site • If a resource has the address http://www.harvard.edu.us/ it is part of the Harvard University Web site • URLS are removed from the Internet when their content becomes outdated Question reproduced with permission from RDN Virtual Training Suite Internet Politician http://www.vts.rdn.ac.uk

11. Student responses: URL knowledge

12. Sample question - ILL • Which of the following statements about interlibrary loans are correct? • You need to know where something is held before you apply for an interlibrary loan • You need to get your tutor’s signature for each interlibrary loan request • You have to pay a fee for each interlibrary loan you request • Your department pays for your interlibrary loan requests

13. Student responses: Interlibrary loans

14. Sample question - COPAC • Which is the only true statement about the COPAC library catalogue? • COPAC covers all the public libraries in the UK • COPAC covers all the university libraries in the UK and Ireland • COPAC covers over 20 major research libraries in the UK and Ireland • If you find a useful book on COPAC, you can go to that library and borrow the book • Prior to 2005 a 5th option was also included: • COPAC does not include references to journal articles

15. Student responses: COPAC MCQ was changed to single correct response in 2005

16. Questions most likely to get right Identifying Oxford URL www.exeter.ox.ac.uk Internet referencing Athens accounts Questions most likely to get wrong Search strategy COPAC URL statements Interlibrary loans Is a web page written by a subject expert? Trends detected in pre-test 2001-2005

17. Issues for consideration • Paper-based as easiest way of getting quick snap-shot of cohort • Problems scaling up for greater numbers? • Test has been automated for those in USA • Wouldn’t be easy to achieve same high response rates if whole cohort had to take test online • Only 1 correct answer makes questions easier to get right • Multiple correct responses provide useful feedback for librarians (and academics)

18. How the test has affected my teaching • Allows me to see which parts of the teaching were least effective • Has influenced other teaching sessions • E.g. using MCQs as a warm-up activity • Presenting the test results at UoP has led to other departments asking for library input for 2nd years

19. Do the sessions have an impact? • Changes in the lowest and highest scores obtained Lowpre=2 post=9Highpre=16 post=26 2005 pre=4 post=13pre=20 post=29 2004 pre=1 post=7pre=20 post=28 2003 • Individual student improvements • Students at least doubling their initial scores 2002=35%, 2003=55%, 2004=46%, 2005=58% • Students starting from v low score often make significant improvements e.g. pre-test 1, post-test 13 • Number of “No answer” responses drops from 251 on pre-test to 11 on post-test 2005 151 15 2004 183 10 2003

20. Do the sessions have an impact? In 2001 & 2005 the score was out of 26 In 2002-2004 the score was out of 30

21. Do the sessions have an impact? Paired sample t-test on 296 results 2002-2005: t(295 )=-39.6 p<0.001 Pre M=10.32 Post M=20.02 In 2001 & 2005 the score was out of 26 In 2002-2004 the score was out of 30

22. Useful references • Barclay, D. (1993). Evaluating library instruction: doing the best you can with what you have. RQ, 33, 195-201. • Carter, E.W. (2002). “Doing the best you can with what you have:” lessons learned from outcomes assessment. Journal of Academic Librarianship, 28(1-2), 36-41. • Clausen-May, T. (2001). An approach to test development. Slough: National Foundation for Educational Research. • Lawson, M.D. (1999). Assessment of a college freshman course in information resources. Library Review,48(2), 73-78. • RDN Virtual Training Suite (2000-2006) http://www.vts.rdn.ac.uk (Review section quizzes within each subject link are of particular interest)