Department of Information & Communications

Online Information Literacy Audits: a longitudinal study Bob Glass & Jill Griffiths Department of Information & Communications

A three year study of two undergraduate cohorts The activities undertaken address the following questions: 1. Is there value in administering online information literacy audits? 2. What do the results of these audits tell us? 3. Does intervention make a difference in the results? 4. Is there a need in for a UK designed, generic, online Information Literacy Audit? The testing is based around an online web based Information Literacy Test from Steven Wise & Lynn Cameron (2004). The test is hosted byThe Institute for Computer Based Assessment, James Madison University (2006) Department of Information & Communications

Who are the students? Information & Communications 65 students on 5 degree routes all studying the same six first year units Economics 19 students from1 degree route all studying the same six first year units Background A very mixed group socio economically, ethnically and chronologically Some students were working librarians Department of Information & Communications

What is the test? • An Online, web based, multiple choice, using radio • buttons for answer selection • Asks 65 questions (60 live plus 5 trial questions) • Uses the “Adaptex” Assessment System (Wise, S. L. & Yang, S. 2003) to manage the responses. • Measures four of the five ACRL standards • Testing takes around an hour to complete, no time limit set • Students receive a score on completion of the test • Tutors receive detailed performance data in a range of formats • Devised by library and psychology practitioners Department of Information & Communications

What is the test? ACRL Objectives – designed around ACRL standards for information literacy: ACRL1 – Determines the nature and extent of the information needed ACRL2 – Accesses needed information effectively and efficiently ACRL3 – Evaluates information and its sources critically and incorporates selected information into his or her knowledge base and value system ACRL4 – Assessed differently (essay) ACRL5 – Understands many of the economic, legal and social issues surrounding the use of information and access and uses Department of Information & Communications

What is the test? Format: Department of Information & Communications

Is there value in administering online information • literacy audits? • can provide a cohort baseline for tutors/ practitioners • can provide interaction, information & feedback for the students • can highlight areas for development with students • can highlight areas for development in the curriculum • can highlight areas for development for library practitioners Department of Information & Communications

What do the results tell us? • the majority of students scored well above the scorerequired for “Proficient” • 2. the mean score for ICO students was in the “Advanced” classification the mean score for Economics was almost identical • 3. Only one in every 18 of the students failed to reach the Proficient grade (4 ICO students and 1 Economics) • 4. just over a quarter of students were classed as Proficient two thirds were classed as Advanced Department of Information & Communications

Preliminary results Department of Information & Communications

What is the test? Standards assessed : 1: can the student define and articulate the nature and extent of the information needed. 2: can the student access needed information effectively and efficiently. 3: can the student evaluate information and its sources critically and incorporate selected information into his or her knowledge base and value system. 5: can the student understand many of the ethical, legal and socio-economic issues surrounding information and information technology. Standard 4 (written work) cannot be assessed by this type of test Department of Information & Communications

Preliminary results Department of Information & Communications

What do the results tell us? • the majority of students were able to answer questions concerningdefining and articulating the nature and extent of the information needed correctly • 2. around half were not able to answer questions concerning • accessing needed information effectively and efficiently correctly • 3. two thirds were able to answer questions concerning evaluating information and its sources critically and being able to incorporate selected information into his or her knowledge base and value system • 4. a similar number were able to answer questions concerning understanding many of the ethical, legal and socio-economic issues surrounding information and information technology correctly. Department of Information & Communications

What do the results tell us? • ACRL2:Students struggled with answering questions about: • Selecting the most appropriate methods and strategies for accessing the • needed information, for example: • Did not understand about different types of database and what a library • catalogue was • Could not determine the provenance of the information or sources • Constructing and implementing effectively designed search strategies • Did not know how to expand searches or apply them to different • databases • Refining the search strategy if necessary • Unable to use boolean operators, truncation, phrase searching • Extracting records and managing the information and its sources • Could not recognise different item formats from bibliographic search • results presented to them from databases Department of Information & Communications

The answers to our remaining questions Q. Does intervention make a difference in the results? Q. Is there a need in for a UK designed, generic, online Information Literacy Audit? In progress Stage two testing completed 14th March 2008 Stage two analysis completed by the end of May 2008 Stage three testing commences November 2008 Department of Information & Communications

References The Association of College and Research Libraries (2000). Information Literacy Competency Standards for Higher Education. ACRL Wise, S. L. & Yang, S. (2003). The Adaptex Assessment System (Version 1.5) . U.S. Department of Education Fund for the Improvement of Postsecondary Education. Wise,S,.L. Cameron, L. ,Yang, S. & Davis, S. (2005) Information Literacy Test: Test Development and Administration Manual. Harrisonburg: Institute for Computer-Based Assessment; Center for Assessment & Research Studies, James Maddison University. The Institute for Computer Based Assessment, James Madison University(2006) Information Literacy Test (ILT) [Online: http://www.jmu.edu/icba/prodserv/instruments_ilt.htm accessed 21st February 2008) Department of Information & Communications

Department of Information & Communications