Assessing Critical Thinking: A Search for a Holy Grail

1. Assessing Critical Thinking: A Search for a Holy Grail?

2. Critical Thinking Assessment Framework* Critical Thinking Skill Analysis Evaluation Inference Divergent Thinking Instructor Identified Assessment Measure California Critical Thinking Skills Test Writing Rubric CCTST ? Instructor Responsibility

3. CCTST Basics The test booklet is a comforting peach color. A 34-item forced-choice measure of critical thinking skills Normative data allow comparison to larger population Measures multiple aspects of critical thinking including inductive and deductive reasoning

4. CCTST Basics Our three domains: analysis, evaluation, and inference Peter Facione: CCTST measures them all without question Douma and Jones: CCTST measures analysis and inference domains well Douma and Jones: Pete Facione is a nice guy, but we are probably right. Douma and Greg: Now in syndication on UPN

5. CCTST and Heuristics1 Influenced by Kahneman and Tversky�s decision making research and theory Heuristic: a problem solving method that provides a reasonable, probably correct decision selecting among actions multi-attribute alternatives A very practical outcome of critical thinking

6. Heuristics and Decision Making Heuristics and thinking errors Availability heuristic Representative heuristic Base-rate fallacy

7. Heuristics and Decision Making Availability heuristic If a word of three letters or more is sampled from an English text, is it more likely that the word starts with �r� or has �r� as the third letter?

8. Heuristics and Decision Making Representative Heuristic Linda is 31-years old, single, outspoken and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations. Considering this description, which of the following statements is most likely? A: Linda is a bank teller. B: Linda is a bank teller involved in the feminist movement.

9. Heuristics and Decision Making Base rates A taxi-cab was involved in a hit-and-run accident. Two cab companies, the Green and the Blue, operate in the city. You are given the following data: Eighty-five percent of the cabs in the city are Green while 15 percent are Blue. In court, a witness identified the cab in question as Blue.

10. Heuristics and Decision Making The court tested the ability of the witness to identify cabs under a variety of conditions. When presented with a series of pictures of cabs in which half were blue and half were green, the witness correctly identified the cabs in 80% of cases. What was the probability that the cab involved in the accident was Blue rather than Green?

11. Heuristics and Student Thinking These errors are how typically people think. Bad news: This is how people typically think. Good news: Adults seem to naturally use heuristics when making decisions. Challenge 1: Get students to use more valid heuristics. Challenge 2: Measure students use of valid heuristics.

12. CCTST and Heuristics �Even as a special interest reporter and �weatherman� in Indiana, I realized the importance of assessing critical thinking. You there, the third person from the left in the second row, what do you wonder about the CCTST?�

13. Critical Thinking Dispositions Alas, a topic only an academician could love�

14. Critical Thinking Dispositions Suppose that you, taking a page from Super Dave Osborne�s playbook, decided to teach your students every critical thinking skill known to humankind; every law of logic, every handy heuristic, every rule of rhetoric, every competency a student needs in order to ace the CCTST (which the Faciones insist has never been done)�

15. Critical Thinking Dispositions �would your CT course float? I�m not convinced. (Paul, you agree?) First, like Super Dave, you would hurt yourself. Second, even if your students can think critically, they may not order their intellectual behavior so that they do think critically.

16. Critical Thinking Dispositions Is our ultimate goal to develop students who can think critically, or students who do think critically? These characteristics are not identical; in recognition of this, the good folks at Insight Assessment have developed the CCTDI, as a companion to the CCTST, to measure student development toward the latter (greater?) end.

17. Critical Thinking Dispositions The CCTDI measures intellectual behavior along seven scales: Truthseeking: the intellectual integrity to follow reason and evidence wherever it may lead Open-Mindedness: tolerant of divergent views, self-monitoring for possible bias Analyticity: demanding application of reason and evidence, inclined to anticipate consequences Systematicity: valuing organization, persistence even in complex problems CT Self-Confidence: trusting one�s own reasoning Inquisitiveness: eager to acquire knowledge, even when applications of the knowledge are not immediately apparent Maturity of Judgement: prudence in making, suspending, or revising judgement; awareness of circumstances that require closure even in absence of complete knowledge http://www.insightassessment.com/test-cctdi2.html

18. Critical Thinking Dispositions The CCTDI measures intellectual behavior along seven scales: These categories were selected to correspond to the attributes of critical thinkers identified in a 1990 report from the American Philosophical Association: �The ideal critical thinker is habitually inquisitive, well-informed, trustful of reason, open-minded, flexible, fair-minded in evaluation, honest in facing personal biases, prudent in making judgements, willing to reconsider, clear about issues, orderly in complex matters, diligent in seeking relevant information, reasonable in selection of criteria, focused in inquiry, and persistent in seeking results which are as precise as the subject and the circumstances of inquiry permit.� (P. Facione, Delphi Report, 1990, p. 3)

19. Critical Thinking Dispositions Is this anything? Consider the following: The CCTDI is normed [4], with descriptive and quantitative classifications

20. Critical Thinking Dispositions Is this anything? Consider the following: The CCTDI has been tested for reliability [2], with Cronbach�s Alpha .90 for overall disposition and a range of .72 to .80 for each of the seven subscales.

21. Critical Thinking Dispositions Is this anything? Consider the following: In prior studies, r 2 values for the relationship between CT skills and CT dispositions were consistently below 0.16, with r 2 = 0.04 for the overall relationship between CCTST and CCTDI scores [2].

22. Critical Thinking Dispositions Is this anything? Consider the following: While the CCTST focuses largely on a particular set of critical thinking skills (it�s no surprise that r = .708 for CCTST vs GRE Analytic), the CCTDI may allow us insight into other critical thinking characteristics. (Nancy, here�s a partial answer to your question.)

23. Critical Thinking Dispositions Fast-Breaking News A �new and improved� dispositional inventory, the CM3, is now available. Preliminary studies on groups of adolescents suggest that the CM3 may provide greater reliability and factor stability than the CCTDI [3].

24. Critical Thinking Dispositions Fast-Breaking News The CM3 classifies CT disposition according to four factors [3]: Learning Orientation: disposed toward increasing one�s knowledge, values learning process, seeks challenges Creative Problem Solving: solves problems with original ideas, engages in puzzles, desires to understand underlying function of objects Mental Focus: diligent, systematic, organized, clear-headed Cognitive Integrity: interacts with different viewpoints for the sake of seeking truth or making best decision, intellectually curious, fair-minded

25. Critical Thinking Dispositions Two Guys and aWorkshop Recommend: If you are teaching a CT course, you might be well advised to consider using skills and disposition tests in tandem as a means of course assessment. (CCTST+CCTDI or TER+CM3)

26. Course-Level Assessment of CT What are the domains of critical thinking that we assessing? What measures must you use as CT course to assess these domains?* Are you limited to using these measures to assess CT?

27. Excellent answer, my friend. You are a lot better than those suckers at State U. Now, please accept this toast on a stick as a reward for your superior insight and generally pleasant demeanor.

28. Critical Thinking Assessment Framework* Critical Thinking Skill Analysis Evaluation Inference Divergent Thinking Instructor Identified Assessment Measure California Critical Thinking Skills Test Writing Rubric CCTST ? Instructor Responsibility

29. Assessing CT in Your Course Ideas and examples Start Build on what you already do to teach and assess critical thinking Start

30. University and General Education Level Assessment A Modest Proposal: Administer the CCTST+CCTDI (or TER+CM3) to a random sample of freshmen on Assessment Day, and readminister to the same sample (those persisting, matched pairs design) on Assessment Day in the junior year.

31. University and General Education Level Assessment A Modest Proposal: This design would ostensibly allow us to measure and assess change in CT skills and dispositions, and would allow us to control for potential confounding (general change in maturity and intellectual skills/habits through first to years of college), as some fall semester juniors�hopefully�have not yet completed a CT course.

32. University and General Education Level Assessment A Modest Proposal: By classifying data according to number and type of CT courses taken by junior year, we could�in principle�discern the impact of the designated CT course model on student CT skills and disposition.

33. University and General Education Level Assessment A Modest Proposal: Warning: This is not official policy or process�just an idea at this point. Lots of people far more important than Jones and Douma need to weigh in on this first.

34. An Exercise in Understanding and Evaluating an Argument in a Biased Context http://www.crispinsartwell.com/meds/math.htm In light of Dr. Sartwell�s article, should USF require all of its students to study mathematics, regardless of their major or career plans? (assignment used in various forms in MAT151)

35. Sample CT Proposal and Course Syllabus

36. �If you need h�ep, here I am�

37. Critical Thinking Dispositions List of References [1] American Philosophical Association, �The Delphi Report: Research Findings and recommendations prepared for the committee on pre-college philosophy,� Critical Thinking: A Statement of Expert Consensus for Purposes of Educational Assessment and Instruction, P. Facione (project director), 1990. [2] Facione, P., N. Facione, and C. A. Giancarlo, �The Disposition Toward Critical Thinking: Its Character, Measurement, and Relationship to Critical Thinking Skill,� Informal Logic (2000), 20(1), 61-84. [3] Giancarlo, C.A., S. Blohm and T. Urdan, �Assessing Secondary Students� Dispositions Toward Critical Thinking: Development of the California Measure of Mental Motivation,� Educational and Psychological Measurement (2004), 64(2), 347-364. [4] Giancarlo, C.A. and P. Facione, �A Look Across Four Years at the Disposition Toward Critical Thinking Among Undergraduate Students,� The Journal of General Education (2001), 50(1), 29-55.

38. End Note 1. The material on heuristics was adapted from Gilhooly, K. J. (1996). Thinking: Directed, undirected, and creative. San Diego: Academic Press.

39. CT Assessment Resources http://www.criticalthinking.org/ This is the Foundation for Critical Thinking (Richard Paul�s group).� Like a corporate conglomerate, they have merged with several other groups and changed some of their names (�International Congress on Critical Thinking� is now �International Conference of Critical Thinking�).� Still, I believe this group is one of the most listened-to voices in the critical thinking community. http://innumeracy.com/criticalthink.htm This is a nifty collection of information that is consonant with both the CCTST perspective as well as Richard Paul�s group (indeed it includes links to both).� The emphasis here, as the �innumeracy� title suggests, is toward the quantitative, deductive, and analytic; however, I believe some of this material is universal.� This is actually the site that received my praises in this afternoon�s conversation.� The alleged �New Mexico� site (see mea culpa below) is one of the dozens of links from this site which I had followed a few weeks ago. �

40. CT Assessment Resources http://admin.santafe.cc.fl.us/~acres/ct/tutorial/ See how another school is using the CCTST.� � http://www.usiouxfalls.edu/faculty/Development/Critical%20Thinking%20Proposal%20Guidelines.doc Okay, you probably could have accessed this link yourself, but just for convenience here�s the document that outlines current APC policy for approving designated CT courses.