Critical Thinking at Texas Tech March 6, 2009

1. Critical Thinking at Texas Tech March 6, 2009

2. Teaching and Assessing Critical Thinking: How to Make Critical Thinking a REAL Learning Outcome Diane F. Halpern, PhD Claremont McKenna College, Claremont, CA With special thanks to a lifetime of coauthors, especially Milt Hakel, Art Graesser, Keith Millis, my NCUEP Colleagues, and the real thinkers—our wonderful students

3. They are the children of late Baby Boomers (1945-64) or Gen Xers (1965-81) They spend 6.5 hours a day on various media They are 36% of US population They are more diverse than any prior group Our Students—the Millennials

4. Let’s Start by Thinking About the Lives of Our Students Present and Future • What do they need to know in a world where knowledge is accumulating an unprecedented rate? • Where our students will work at jobs that do not exist today. • Where a college degree is a requirement for 90% of the fastest growing jobs. • Where pollution is a major problem, along with racism, poverty, and terrorist attacks

5. Information, information, information… • "A weekday edition of The New York Times contains more information than the average person was likely to come across in a lifetime in 17th. century England." Information Anxiety, R.S.Wurman

6. Multitaskers--Attention is limited, and when it is divided among too many tasks or the tasks are difficult, performance suffers. Of course, people have always been able to multitask (that is, attend to more than one thing): Parents monitor their children while cooking dinner, we carry on conversations while walking, students do homework while listening to music, and so on. But those who overload their systems by, for example, studying while checking e-mail, instant messaging, and watching television will do worse at all these tasks than they would if they focused on one at a time. • The more they use the web, the more difficult it becomes to stay focused on long pieces of writing.

7. Rethinking the Purpose of Education The best education for life in the 21st century must be built on the twin pillars of "learning how to learn" and how to think critically about the vast array of information that confronts us.

8. Using What We Know About Teaching and Learning • Systematically apply what we know about how people learn to curricular design and class planning. • Better use of technology—for example learning games • Recognize what students really need to know—how to think critically and learn in a world where we are drowning in information

9. So, if you are thinking critically, you are wondering if it is possible to help students improve how they think? Why not? We teach writing, oral communication, math with the belief that these skills will transfer to appropriate situations.

10. Evidence That Better Thinking Can Be Learned with Appropriate Instruction • "Blind" evaluations of programs designed to enhance thinking skills (e.g., the Venezuela project) • Student self reports (weak evidence, but students believe that they have improved) • Gains in cognitive growth and development (e.g., Piagetian tasks that measure cognitive stages) • More expert-like mental representations following instruction (relative to control groups) • Decision makers trained to “reorganize” existing knowledge in naturalistic settings showed more “expert-like” performance • Tests of cognitive skills (e.g., standardized tests for critical thinking) • Spontaneous and uncued transfer (e.g., call students at home months after the class is completed under the guise of a survey) • Inductive reasoning tasks were taught to college students using realistic scenarios form many different domains. The authors conclude that critical thinking is “a skill” and that “it is transferable”

11. Most important robust finding: “The majority of studies report positive impact on pupil’s attainment across a range of noncurriculum measures (such as reasoning or problem-solving). No studies reports reported negative impact on such measures (Thinking Skills Review Group, October 2003).

12. Critical thinking is the use of those cognitive skills and abilities that increase the probability of a desirable outcome. It is purposeful, reasoned, and goal directed. It is the kind of thinking involved in solving problems, formulating inferences, calculating likelihoods, and making decisions. Critical thinkers use these skills appropriately, without prompting, and usually with conscious intent, in a variety of settings. That is, they are predisposed to think critically. When we think critically, we are evaluating the outcomes of our thought processes--how good a decision is or how well a problem is solved. Critical thinking also involves evaluating the thinking process--the reasoning that went into the conclusion we've arrivedat or the kinds of factors considered in making a decision.

13. Important Robust Findings from Research: • Critical thinking will transfer when it is taught for transfer. • The effect may not be even across all groups. • Some of the benefit derives from making the thinking skills explicit. • The disposition to think critically, like the learning-performance distinction is important

14. Thinking About Critical Thinking Instruction

15. Effective critical thinking instruction is predicated on two assumptions: • There are clearly identifiable and definable thinking skills that students can be taught to recognize and apply, and (2) If recognized and applied, the students will be more effective thinkers.

16. Teaching & Learning to Think Critically: A Four-Part Model 1.Explicitly teach/learn the skills of critical thinking 2.Encourage/develop the disposition of effortful thinking and learning 3.Direct learning activities in ways that increase the probability of transfer 4.Make metacognitive monitoring explicit and overt (Halpern, 1998, 2004)

17. Here is a generic list of thinking skills that would be applicable in many situations: • recognizing that a problem exists • developing an orderly, planful approach so that tasks are prioritized and problems are recognized as differing with regard to how serious and urgent they are • understanding how cause is determined, • recognizing and criticizing assumptions, • analyzing means-goals relationships, • giving reasons to support a conclusion, • assessing degrees of likelihood and uncertainty, • incorporating isolated data into a wider framework • using analogies to solve problems.

18. Generic Thinking Skills, continued • Relating new knowledge to information that was previously learned • Using numerical information, including the ability to think probabilistically and express thoughts numerically • Using matrices and other diagrams to communicate • Synthesizing information from a variety of sources • Determining credibility and using this information in formulating and communicating decisions • Selecting among alternatives with the use of a reasoned method

19. Build on critical thinking skills throughout the curriculum • Identify the skills you want students to learn in your General Education critical thinking courses, then be sure they are used in other classes • Deliberately add new skills throughout the curriculum—may be some differences by discipline, but ensure overlap among GE courses • Some critical thinking skills that are firmly rooted in psychology • Understanding that small samples yield more extreme results • Recognizing and avoiding hindsight bias • Knowing why we need control groups • Avoiding either-or thinking (e.g., is it nature or nurture) • Being aware of the fallibility of memory • Self-serving attributes for success and failure • Halo effects

20. Making Arguments WorksheetExample 1 Does violence on television really have a negative influence on children's behavior?1. State your conclusion. (although you may begin your formal writing here, but sure that the conclusion follows from your reasons). As you work, this is the last part this is filled in, not the first.2. Give three reasons (or some other number) that support your conclusion. 3. Rate each reason as weak, moderate, strong, or very strong.

21. 4. Give three counterarguments (or some other number) that weaken your conclusion. Rate how much each counterargument weakens the conclusion: little, moderate, much, or very much.5. List any qualifiers (limitations on the reasons for or against—for example some evidence may be restricted to early childhood)6. List any assumptions.7. Are your reasons and counterarguments directly related to your conclusion?8. What is the overall strength of your argument: weak, moderate, strong, or very strong? Now that you have completed this worksheet, rate the overall strength of your argument.

22. Dispositions for effortful thinking and learning Model and provide explicit instruction in the • willingness to engage in and persist at a complex task • conscious use of plans and suppression of impulsivity • flexibility and open-mindedness • willingness to abandon nonproductive strategies and self-correct

23. We need to direct learning activities that make transfer more likely • Draw a diagram/graphic that organizes information. • What additional information would you want before answering the question? • Explain why you selected (a particular) multiple choice question. • State the problem in at least two ways. • Which information is most important? Why? • Which information is least important? Why? • Categorize the information in a meaningful way.

24. What Makes Great Teachers Great? “Create a natural critical learning environment. "Natural" because what matters most is for students to tackle questions and tasks that they naturally find of interest, make decisions, defend their choices, sometimes come up short, receive feedback on their efforts, and try again. "Critical" because by thinking critically, students learn to reason from evidence and to examine the quality of their reasoning, to make improvements while thinking, and to ask probing and insightful questions. This is, by far, the most important principle -- the one on which all others are based and which commands the greatest explanation.” Ken Bain Chronicle of Higher Education, April9, 2004; Volume 50, Issue 31, Page B7

25. Measuring Critical Thinking

26. Halpern’s Taxonomy of Critical Thinking SkillsA Guide for Instruction & Assessment

27. Halpern’s Critical Thinking Assessment • Verbal reasoning: ability to comprehend & defend against persuasive techniques embedded in everyday language; • Argument Analysis: ability to judge how well evidence supports a conclusion; • Hypothesis testing: accumulating observations, forming beliefs/hypotheses/seeking convergent evidence;

28. HCTA, continued • Likelihood & Uncertainty: recognizing regression to the mean, understanding the limits of extrapolation; probability & chance. • Decision making/problem solving: Seeking information to reduce uncertainty; listing alternatives; recognizing bias.

29. Assessing Critical Thinking – the Halpern Critical Thinking Assessment (HCTA) • Available on the Internet/Paper (2 versions) • Scenario-based • Each competency is evaluated with forced- choice & constructed responses • Forced-choice: recognition memory • Constructed response: free recall

30. Halpern Critical Thinking Assessment: Sample Question • A recent report in a magazine for parents showedthat adolescents who smoke cigarettes also tend to get low grades in school. As the number of cigarettes smoked each day increase, GPA decreased. One suggestion made in this report was that we could improve school achievement by preventing adolescents from smoking. • Based on this information, would you support this idea as a way of improving the school achievement of adolescents who smoke? • Type “yes” or “no” and explain why or why not.

31. Based on this information, which is the best answer? 1. School grades probably will improve if we prevent adolescents from smoking. 2. School grades might increase, but we cannot be certain because we only know that grades go down when smoking increases. 3. There is no way to know because we only know that smoking and grades are related, not whether smoking causes grades to change. 4. There will probably be no effect on grades if we prevent adolescents from smoking because the magazine is written for parents, so it is probably biased against teen smoking.

32. Based on this information, which is the best answer? 1. School grades probably will improve if we prevent adolescents from smoking. 2. School grades might increase, but we cannot be certain because we only know that grades go down when smoking increases. 3. There is no way to know because we only know that smoking and grades are related, not whether smoking causes grades to change. 4. There will probably be no effect on grades if we prevent adolescents from smoking because the magazine is written for parents, so it is probably biased against teen smoking.

33. IES Grant that is incorporating what we know about the science of Learning • A program to teach critical thinking/ scientific reasoning skills using what we know about current students (play games on line, use teaching agents, and more). • Grant is with Keith Millis at Northern Illinois University and Art Graesser at University of Memphis—(they are the brains behind this project) • ARIES Acquiring Research Investigative and Evaluative Skills (a brief look early in the project) • Special thanks to other ARIES folks—Patricia Wallace, Zhiqiang Cai, Cristian Moldovan, Carol Forsyth, Anne Britt, Joseph Magliano, Katja Wiemer

34. ARIESAcquiring Research Investigative and Evaluative Skills • Animated Agents—a teacher and a student guide the student through the tutor lessons and will be an expert on scientific inquiry—the heart of critical thinking • For example: “My roommate and I got into an argument yesterday on who was more influential on hip hop: James Brown or Stevie Wonder” to which the Teacher-Agent might respond, “You know, you could have resolved the argument by using what scientists call an operational definition.”

35. Some Key Concepts in Scientific Inquiry • Developing Research Ideas • Theories, hypotheses, pseudoscience, falsifiability • The Independent and Dependent Variables • Operational definitions, reliability, accuracy, precision, validity, objectivity of scoring • Experimental Control • Comparison groups, random assignment, subject bias, attrition/mortality • The Sample & Experimenter • Representative, sample size, experimenter bias, conflict of interest • Drawing conclusion • Alternative interpretations, limits of correlation research, quasi-experimental designs, replication of results

36. ARIESAcquiring Research Investigative and Evaluative Skills • We use “science of learning” in the program • Self-explanation • Generate reasons why a study is faulty or not faulty • Reciprocal teaching • Students teach the fellow student • Spacing, testing effects • Students must recognize concepts across many examples • Variable encoding • Psychology, biology and chemistry problems • Authentic learning • Case studies are magazine, news articles, advertisements • Motivation, engagement • Consequences for their performance • Auto-tutor platform that allows students to hold a dynamic conversation with the learner (dialog interactivity). • Principles of serious gaming.

37. Operation ARIES! Your mission: to become an agent of the Federal Bureau of Science (FBS) to hunt down Fuath spies who are on Earth stealing our resources. Be careful because they look and act human! The Fuaths have been publishing faulty research in a variety of fields in order to confuse future human generations about the scientific method. They are also milking our economy dry by selling products on the Internet, based on suspect research.

38. Aliens taking over the earth with bad science—now that is scary!!

39. How can you become an FBS agent? Step 1: Take a Science Training Course You will learn key concepts in the scientific method like control groups, validity, independent and dependent variables (20 in all). You will read (and be tested on) a science book used by the Fuath spies. You will be guided by an FBS handler and be joined by a fellow (animated) student

40. How can you become an FBS agent? Step 2: Analyze Case Studies You will analyze examples of research written by the Fuaths. They are from magazines, the Internet, and newspapers, covering topics in Psychology, Biology and Chemistry. With help from your FBS handler and a Fuath defector, you will learn to identify flaws in the research. Pay attention because the Fuaths use the flaws to communicate with one another. You will likely uncover important clues to their plans.

41. How can you become an FBS agent? Step 3: Interrogate Suspected Alien ‘Scientist’ Spies You are close to finding and stopping the mastermind behind the invasion. FBS has captured a number of scientists, some of whom are Fuaths and some whom are Human. It is your job to tell the difference between them by asking questions about their research. If a study is faulty, then you have found a Fuath and you are one step closer to saving the Earth. Be alert!

42. A Sample Screen

43. Example of Research to EvaluateStudents Have to Ask Questions

44. Scientific Thinking/ Critical Thinking • For each chapter • Multiple choice questions • Types: Definitional, Function/importance, Example • 2 of each, 6 total • “Challenge Test” • Options: take before or after reading chapter • If before, and any error on first three, then given chapter • Order of questions • D F E D F E • “Trialog” after the last 3 questions

45. Types of Trialogs • Dr. Quinn (teacher) to Glass (student agent) • Vicarious learning for students with low scores • Low knowledge good for vicarious learning (Craig et al., 2004) • Dr. Quinn (teacher) to human • Standard AutoTutor for students with intermediate scores • Partial knowledge, zone of proximal development • Human to Glass Tealman (student agent) • Teachable agent for students getting all or most questions correct • Reinforce and use existing knowledge

46. Here is an example of a session in which the student evaluates information to decide if the person doing the research is a human (good research) or an alien (flawed research)

47. Summary • ARIES is an initial attempt to incorporate AutoTutor in a game-like setting • Dialogs in AT • Make learning active, increasing motivation • Can serve different tasks (testing, evaluating studies, promoting questioning skills, & advancing story line) • Making engaging games that lead to deep learning is still a new frontier in education

48. Many Experiments are Underway • 1. Does Having Different Content Domains in ARIES Increase Transfer? • 2. Does Role Playing and the Type of Activity Affect Learning? • 3. Do We Need Full Dialog Exchanges for all Active Applications Problems for Deep Learning to Occur? • 4. Does the Reciprocal Teaching in ARIES Affect Learning? • 5. Does the Effectiveness of “Teachable Agents” Depend on the Perceived Knowledge of the Agent?

49. Contact Information Dr. Diane F. Halpern Claremont McKenna College 850 Columbia Ave. Claremont, CA 91711 (909) 607-9647 To try out the critical thinking assessment, send an e-mail to diane.halpern@cmc.edu