Pilot Administration of the Critical Thinking Assessment Test (CAT)

Pilot Administration of the Critical ThinkingAssessment Test (CAT) Phil Dunwoody, Kathy Westcott , Jerry Kruse Carlee Ranalli, Jim Roney, Kim Roth, Peter Rothstein, Jim Tuten Juniata College Huntingdon, PA

Overview of CAT • The focus of the CAT is to… • “assess and promote the improvement of critical thinking and real-world problem solving skills.” • Engage faculty in the assessment of critical thinking • Be reliable and valid • Be administered in 1 hour

Overview of CAT • Administration of CAT • Short essay answer test with a variety of “real-world” problems • Questions focus on interpreting information and reaching valid conclusions • 1) There is an increase in campers getting sick. Students evaluate causal claims based on the evidence provided. • 2) A family hiking scenario is described along with the pros and cons of various water filters. Students are prompted to select the water filter that meets the family’s needs. • Prompts are ordered from vague to specific

Overview of CAT • Reliability and Validity of CAT • Correlates with the SAT (r=0.52) and the California Critical Thinking Skills Test (r=0.65) • NSSE correlations • (2a) Memorizing facts, ideas, or methods from your courses and readings so you can repeat them in pretty much the same form. (negative relationship) -.341* • (3b) Number of books read on your own (not assigned) for personal enjoyment or academic enrichment. .277 ** • (11e) Thinking critically and analytically &(11m) Solving complex real-world problems .244 ** • Faculty scored • Rubric ensures reliable coding of responses • Closing the loop

General Education Assessment Goals: • Develop communication skills • ________higher order thinking skills IC, CA, upper-level FISHN • Develop knowledge about and ways of understanding the world

CAT Administration • One section each: • CWS • Psychology Research • Quantitative Methods • Pre/post (using local codes), early and late in semester • $200/year and $6/test • We are the graders • Aim for one grader per 10 tests • Transparent: No proprietary institutional scoring (CLA)

CAT Report (below) and Excel file w/raw data

CAT Results

CAT Question Profile

CAT Score Breakdown

Grader Comments and Discussion

Critical Thinking Assessment Test (CAT) • Face validity“looks like a reasonable measure…” • Inexpensive$6 / test vs. Collegiate Learning Assessment (CLA) • SensitivityFr – to – Sr, effects within a course, no ceiling or floor effects • Culturally fairno effects from gender, race, or ethnic origin • “Reasonable to Moderate” correlation to other measures of critical thinking (CAAP and CCTST) and academic performance (ACT/SAT) • Negative correlation with question 2a on the NSSE,Do the student's courses primarily focus on rote retention of information?

We almost all agree that Critical Thinking is the answer “With all the controversy over the college curriculum, it is impressive to find faculty members agreeing almost unanimously that teaching students to think critically is the principal aim of undergraduate education.” Derek Bok, Our Underachieving Colleges: A Candid Look At How Much Students Learn and Why They Should Be Learning More, Princeton, New Jersey: Princeton University Press, 2006.

A definition to “critically analyze” http://www.tntech.edu/cat/home/

Pilot Administration of the Critical Thinking Assessment Test (CAT)