Assessing Student Learning Outcomes for Information Literacy Instruction

1. Assessing Student Learning Outcomes for Information Literacy Instruction Lori Buchanan Instructional Services Librarian and Professor Austin Peay State University

2. Assessing Student Learning Outcomes Project The purpose of this Association of College & Research Libraries (ACRL) project was to give librarians the skills to create baseline data that supports the merits of information literacy programs. The grant provided training for academic librarians to work with faculty to design, implement, and evaluate tools for assessing student learning outcomes. Austin Peay State University’s Woodward Library was one of 23 United States academic libraries selected to participate in this project funded through a grant by the Institute for Museum & Library Services.

3. APSU Information Literacy Assessment Study Berg, Berwind, Buchanan, Luck, and Hogan pilot tested the assessment instruments (activities) with Humanities 1010 (Heritage) freshmen during Fall 2001. Buchanan then asked 157 students enrolled in six Spring 2002 English 1010 classes to complete the activities as part of the official study. ____________________________ Specific information about the APSU Study are found at http://library.apsu.edu/library/3_9assess.htm. Results are published in Assessing Student Learning Outcomes for Information Literacy Instruction in Academic Institutions, E.F. Avery (Ed.), Chicago: Association of College & Research Libraries, 2003.

4. APSU Spring 2002 Study Overview • Focus on the information literacy outcomes specific to evaluative criteria, which are defined as authority, accuracy, coverage, currency, and objectivity • Compare the impact of different instructional interventions (no instruction, lecture format, active learning) on student learning • Measure student knowledge, application of skills and perceptions • Administer six assessment instruments (activities), including • Preliminary Questionnaire (prior to instructional intervention) • Follow-up Questionnaire (same questions as Preliminary) • Website Evaluation Assignment with accompanying grading criteria • Self-Assessment • Peer Assessment • Instructor Assessment

5. APSU Study Student Demographics • 87 women, 50 men • 90 Caucasian, 31 African-Americans, 9 Hispanic, & seven Other • Ages ranged from 17 to 50, with a mean age of 22 • 83 First-time, 31 Continuing, 12 Re-entry and 11 Transfer • 115 Freshmen, 16 Sophomores, 4 Juniors, and one Senior • 100 Commuted & 37 Lived on-campus • 106 had no prior college library instruction, 31 had prior

6. Instructional Conditions Utilized in the Study • No instruction control group (two classes of 48 students) received no instruction prior to completing activities; this group was instructed after they completed all activities. • Lecture experimental group (two classes of 46 students) participated in a lecture-formatted session on evaluative criteria, then completed activities. • Active learning experimental group (two classes of 47 students) participated in an active learning instructional session on evaluative criteria then completed activities.

7. Critically Evaluating WebsitesLecture When Evaluating World Wide Web Sites, just…Remember Your A-B-C's! Evaluation Examples

8. Critically Evaluating WebsitesActive Learning Select a topic and find several websites to share with students. The selected websites should 1) cover opposing viewpoints on the issue, 2) contain at least one very authoritative site (Congressional Research Service web pages), and 3)contain an obviously unsuitable source of information for an academic paper or speech. Have students break into groups to discuss which websites are suitable for academic work and discuss why. The point of the exercise is to help students determine the criteria for selecting high quality, academic sources to use in their college work. Reinforce points by using the A-B-C webpage at http://library.apsu.edu/guides/1_1_13.htm.

9. Example Websites for Gun Control If You Don’t Kill It & Grill It, Who Will? Says Ted Nugent Issue Brief for Congress received through the CRS Web – Gun Control Legislation in the 108th Congress The Coalition to Stop Gun Violence, “Mr. President, You Broke Your Promise” National Rifle Association-ILA, “The Top 10 Reasons The Clinton Gun Ban Should Expire”

11. Source: Gun Control Legislation in the 108th Congress (Congressional Research Service)

12. Source: The Coalition to Stop Gun Violence Website excerpt from the “About Us” page: The Coalition to Stop Gun Violence (CSGV) emerged from the civil rights movement in the early 1970s and pushes a progressive agenda to reduce firearm death and injury. We were founded on the principle of collaboration, meaning that we work closely with other organizations to achieve our common goals.

13. Source: National Rifle Association Website Who We Are, And What We Do Established in 1975, the Institute for Legislative Action (ILA) is the “lobbying” arm of the National Rifle Association of America. ILA is committed to preserving the right of all law-abiding individuals to purchase, possess and use firearms for legitimate purposes as guaranteed by the Second Amendment to the U.S. Constitution.

14. Preliminary & Follow Up Questions About Evaluating Information      Your Name  _______________________________ 1)        I use criteria to evaluate information.  Circle one. Never --- Almost never --- Sometimes --- Almost always --- Always 2)        List five criteria for evaluating information.   1. 2. 3. 4. 5. 3)        Mark the steps listed below which apply to the following statement.    I verify information by ___            communicating with the author ___            consulting another source ___            asking a friend ___            I don’t; instead, I assume the information is correct ___            Write any other steps you take below

15. Website Evaluation Assignment select a high-quality website about your paper topic use the five established evaluative criteria (accuracy, authority, currency, coverage and objectivity) and associated questions write a thorough typed double-spaced paper (at least 500 words) covering the website you identified. 1)      What is your topic? 2)      What is the complete/full URL of your website?  Print out and attach the first pageonly of your website to your paper. 3)      How well does this website cover your specific topic or information need?  How useful will it be in your paper? (Be specific.) 4)      How accurate is this website? Are there spelling or grammatical errors? Does the website contain outrageous or misleading information?

16. Website Evaluation Assignment – cont’d 5)      Who wrote or sponsored this website?  Is it an educational, commercial, governmental or organizational website?  What audience was the website written for?  Be sure to look in the website’s header, footer and any links for clues. 6)      How objective is the website?  Is a particular viewpoint presented in the website?  If so, what is the viewpoint?  If not, give evidence of how the website covers more than one viewpoint.  Be sure to state what viewpoints are covered. Does the website present ·        Facts ·        Opinions ·        Both facts and opinions Give at least one specific example of a fact and/or an opinion covered by the website. 7)      How current is this website?  When was the information first posted?  Has it been updated within the last year? If present, how current are the sources cited or links contained in the website? 8)      What is the breadth and depth of the website’s coverage?  What is its purpose?  Does it cover a highly specialized topic area in great detail or does it merely provide a general overview of a broad topic?  9)      What are some steps you can take to verify the accuracy of the information contained in this website?  (Be specific.)

17. Website Evaluation Assignment Grading Criteria

18. Self-Assessment of Information Evaluative Skills Circle the ONE answer out of the available choices that comes closest to what you perceive your abilities to be.  Please be honest.  Completing this questionnaire is required, but your specific answers to the statements will NOT impact your grade. 1)     Overall, based on the established evaluative criteria, I think the website I selected was high-quality. Yes                     No 2)     As it relates directly to the topic about which I plan to write a paper, the website I selected is of no use      of very limited use      of some use    useful      very useful 3)     I successfully applied the following number of the established evaluative criteria in my Website Evaluation Assignment. one       two                three                    four                     five 4)     Before taking this class, I applied evaluative criteria to information and its sources. never   almost never              sometimes     almost always      always 5)     At the present moment, I can list from memory the following number of established evaluative criteria covered in the assignment.           one       two                three                    four                   five 6)     Before taking this class, I was in the habit of checking other sources to either confirm or question the point of view or bias of an information source. never       almost never       sometimes           almost always   always

19. Peer Assessment of Information Evaluative Skills Circle the ONE answer out of the available choices that comes closest to what you perceive your classmate’s abilities to be,  based on your examination of her/his completed Website Evaluation Assignment.  It will be helpful to review the assignment (attached) and the website your classmate evaluated before you read your classmate’s work and answer the questions.   Please be honest.  Taking the time to complete this questionnaire in a serious fashion is required, but your specific answers to the statements below in no way impact your grade or your classmate’s grade.  Your classmate will NOT see the results of your assessment. 1)   Overall, based on the established evaluative criteria, my peer selected (see attached completed Website Evaluation Assignment) is high-quality. Yes                        No 2)   In relation to the paper topic my peer has selected (see attached completed assignment), the selected website is   of no use of very limited use      of some use    useful      very useful 3)   In my opinion, my peer successfully applied the following number of established evaluative criteria that were either discussed in class and/or found within the assignment.         one                   two                      three              four                         five

20. Instructor Preliminary Assessment of Information Evaluative Skills Circle the ONE answer out of the available choices that comes closest to what you perceive the student’s abilities to be, based on your examination of her/his completed Website Evaluation Assignment.  It will be helpful to review the assignment (attached) and the website the student evaluated before you read the student’s work and answer the questions.  The student will NOT see the results of this particular assessment.  Please forward this sheet to the librarian as soon as you complete the preliminary assessment. 1)   Overall, based on the established evaluative criteria, the student selected (see attached completed Website Evaluation Assignment) is high-quality. Yes                        No 2)   In relation to the paper topic the student has selected (see attached completed assignment), the selected website is of no use of very limited use      of some use    useful      very useful 3)   In my opinion, the student successfully applied the following number of established evaluative criteria.             one                   two                      three              four                         five

21. Students as a whole averaged one evaluative criterion on the preliminary activity while averaging three criteria on the follow up activity. Student knowledge as a whole increased between the preliminary activity and the follow up activity with respect to individual criteria. Study Results: Student Knowledge

22. Study Results: Student Knowledge – cont’d No significant difference existed among the groups on the preliminary questionnaire; however, differences among the groups were significant on the follow up questionnaire. 19 of the 41 active learning students listed all five criteria, compared to two lecture & no control students.

23. Study Results: Student Application & Perceptions • 49% of the students were able to successfully apply all five criteria in the Web Evaluation Assignment. • Of the students completing self-assessments, 88 of the 105 believed they could list three to five of the criteria; 46 of the 112 students actually did list three to five criteria on the follow up. • Student self-assessments were significantly correlated to the instructor’s assessments of their work. Peer assessments did not correlate to either the instructor or the self.

24. Project Impact on APSU What was learned in the Assessing Student Learning Outcomes Project regarding selecting outcomes, developing curriculum and pedagogy, and creating assessment activities and grading criteria, helped inform the information literacy instruction included in APSU 1000. • Selecting outcomes: What do we want students to be able to do as a result of this instruction? • Developing curriculum and pedagogy: What does the student need to know in order to do this well? What activity will facilitate the learning? • Assessment: How will the student demonstrate the learning? • Criteria: How will I know that the student has done this well? Questions based on an instructional planning model presented by Deb Gilchrist (Lead Faculty of the Information Literacy Institute)

25. Integrating Freshman Level Information Literacy Librarians were involved in the planning and implementation of first-year experience programs from the beginning. Collaboration between librarians and instructors was crucial in achieving the integration of information literacy outcomes into the course’s curriculum. Formative assessment of information literacy learning outcomes occurred in APSU 1000. Surveys & focus groups were used to gather instructor and librarian opinion about the information literacy instruction conducted.

27. How do we use the LILT Quiz Results? • Determine library instruction content specific to each APSU 1000 section • Future modification of LILT content • Further collaborative efforts between librarians and instructors to insure that students become information literate

29. The following resources have been designed for librarians, instructors, faculty, and other educators interested in learning more about LILT. If you have contributions or suggestions for one of these resources, please e-mail us. Use of LILTContains information about the intended audience for LILT, how instructors can use it with their courses, and more information on using the quizzes. Objectives of LILTLists objectives for all of the modules of LILT. Project Development and CreditsBrief information about the tutorial's resources, development, and the LILT team. yourLILTA version of the LILT tutorial distributed at no charge under an Open Publication License. After signing the license and registering, individuals and institutions may download a copy of the LILT text, graphics, and scripts to use and customize for their own campus. Directory DescriptionA description of the directories and individual pages in yourLILT.

30. Quiz ResponsesLibrarian Reports Select a section below to see the quiz report for that section Quiz ResponsesProfessor Reports Select a section below to see the quiz report for that section LILT Dynamic Quiz Database Report System built by Aaron Dobbs, APSU

31. What do the LILT Quiz Results Tell Us? – one example Question 4, Module 6 You can avoid plagiarizing by: Percent Correct: 57%; Number of correct responses: 461 IL Standard 5 (Ethical use of information), Outcome: Demonstrates an understanding of what constitutes plagiarism and does not represent work attributable to others as his/her own IL Standard 4 (Effective use of information to accomplish purpose), Outcome: Integrates the new and prior information, including quotations and paraphrasings, in a manner that supports the purposes of the product or performance

32. Course Assessment: Instructor & Librarian Surveys(Instructor Completion Rate = 37 of 44; Librarian Completion Rate = 12 of 12) How many library sessions should be included in APSU 1000: • 29 instructors suggest two or more; 11 librarians suggest two plus _________________________________ Instruction deemed most helpful to students by the instructors (#s): • How to use search strategy techniques (e.g., Boolean operators) 24 • How to use the Library as a gateway to quality academic sources 22 • How to cite, paraphrase, and avoid plagiarism 20 • How to evaluate and select appropriate sources for college work 16 • Felix 9 Lives Self-Guided Tour/Librarian-led tour 13

33. Created by Nancy Snyder

34. Created by Nancy Snyder

35. Created by Nancy Snyder

36. APSU 1000 Instructor/Librarian Focus Group Results • LILT is a helpful introduction to the library which should continue to be required of all students. All instructors should become familiar with LILT content so that they may hold students accountable for knowing covered concepts and applying skills. • Make sure that all instructors understand the goals of the Career Profile Assignment, which is important to the course and provides students with an opportunity to practice library skills. The assignment might benefit from being modified into a journal format or called a “report” so that students understand that research is required. • All library instruction sessions need to occur in the Library Instruction and Computer Room (LICR).

38. Final Thoughts • Librarians and instructors should collaborate to make sure that students gain information literacy knowledge, skills, and values that will prepare them to succeed both academically and in life. • Assessment should be included in the planning process and the results used to improve instruction. • Now that an information literacy foundation has been laid in APSU 1000, librarians and instructors should work together to see that more advanced information literacy outcomes are addressed and become articulated across the curriculum.

39. QUESTIONS? Thank you for attending today!