Work in Progress Critical Thinking and Information Literacy: Assessing Student Performance

1. Work in ProgressCritical Thinking and Information Literacy: Assessing Student Performance The InfoSkills Research Team Dr. SenayPurzer, Mr. Michael Fosmire, Ms. Amy Van Epps, and Ms. Ruth Wertz Project Collaborators Ms. Megan Sapp Nelson, Dr. Brian Dillman, and Mr. Austin Saragih Presented at the 120thASEE Annual Conference & Expo – Atlanta, GA June 24th, 2013

2. Overview Project Background – Information Literacy and Critical Thinking Research Methods – RQs and Research Design Project Results – Correlation of CELT and CAT Project Conclusions – Implications & Further Study

3. Definitions Project Background Information Literacy A set of skills that enables the ability to recognize the need for information, and the ability to search for, access, evaluate, and use information to fulfill a specific purpose.1 Critical Thinking The cognitive process of “conceptualizing, applying, analyzing, synthesizing, and/or evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or communication, as a guide to belief and action.”2 SET OF SKILLS3 COGNITIVE PROCESSES3 synthesize recognize need evaluate search for/gather conceptualize use/apply access analyze American Library Association. (2000). Information literacy competency standards for higher education. Retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/standards/standards.pdf. The Foundation for Critical Thinking. (2013). Defining critical thinking. Retrieved from http://www.criticalthinking.org/pages/defining-critical-thinking/410. Albitz, R. S. (2007). The what and who of information literacy and critical thinking. portal: Libraries and the Academy, 7(1), 97-109.

4. Impetus for Our Research Project Background • Impetus for our research: • Important 21st century skills • ABET criterion 3i (lifelong learning) • Engineering decision-making • Primary focus of our research: • How do we assess information literacy and critical thinking?

5. About the CELT(v2.1) Project Background • InfoSEAD framework for item generation1 • Seeking – Gathering and access • Evaluation – Assess quality, credibility, etc. • Application – Using information to satisfy a need • Documentation – Citation and in-text referencing • Two Scenarios • 18 total items • 16 multiple choice; 2 select all that apply Wertz, R. E. H., Purzer, S., Fosmire, M. J., & Cardella, M. E. (in press). Assessing information literacy skills demonstrated in an engineering design task. Journal of Engineering Education.

6. Correlational Study Research Methods • Research question: • How well does the CELT (v2.1) measure critical thinking? • Research method: • Pearson’s correlational analysis between CELT instrument and the Critical Thinking Assessment Test (CAT) developed by Tennessee Technology University.1 Center for Assessment and Improvement of Learning at Tennessee Technological University. (2010). CAT technical information, from http://www.tntech.edu/files/cat/reports/CAT_Technical_Information_V7.pdf

7. Population and Setting Research Methods • N = 44 first-year engineering students • CELT reliability measures • KR-20 = 0.671 • N = 188 first-year students (Fall 2012) • Based on 16 MC questions • CAT reliability measures • 15 constructed response items; scoring reliability 0.822 • Cronbach’s alpha 0.702 Wertz, R. E. H., Saragih, A., Fosmire, M. J., & Purzer, S. (2013). An Evaluation of the Critical Engineering Literacy Test (CELT) Instrument through Item Analysis and Comparison to the Critical Assessment Test (CAT). Paper presented at the 2013 Illinois/Indiana - ASEE Section Conference, Angola, IN. Center for Assessment and Improvement of Learning at Tennessee Technological University. (2010). CAT technical information, from http://www.tntech.edu/files/cat/reports/CAT_Technical_Information_V7.pdf

8. CELT Results Project Results • Overall Correlational Analysis: • CELT total score was positively related the CAT total score (r = 0.47, p <0.01) • Itemized Correlational Analysis: • Items 5, 11, 16, 17 had positive associations with the CAT total score

9. CELT Results Project Results Item 5 (r = 0.51, p <0.01) [Evaluate] which of the citations is incorrect or incomplete? Item 11 (r = 0.32, p <0.05) Which of the following comparisons of omega3/omega6 ratios is most relevant in determining whether GE fish is equivalent to its non-GE counterpart? Item 16 (r = 0.37, p <0.05) What would help the review panel validatethe data presented? Item 17 (r = 0.35, p <0.05) Where would you likely find authoritativeinformation on a typical omega-3 levels of salmon?

10. General Conclusions Project Conclusions • 14 of 18 CELT items positively correlated to some portion of CAT • Correlation between the CELT and CAT total scores is moderately strong (r = 0.47, p <0.01) • CELT items 5, 11, 16, and 17 provide insight into a possible subset of skills where information literacy skills and critical thinking overlap.

11. Future Study Project Conclusions • What are the common elements in CELT items 5, 11, 16, and 17 that made them correlate to CAT? • Expand sample • Item-to-item analysis • Verbal protocol

12. Contact Information THANK YOU! Dr. SenayPurzer – senay@purdue.edu Mr. Michael Fosmire – fosmire@purdue.edu Ms. Amy Van Epps – vanepa@purdue.edu Ms. Ruth Wertz – rwertz@purdue.edu