Iglika Pavlova, University of Chicago, Chicago, IL

1. Improving Understanding of Science by Teaching Explicit Principles of Good Reasoning in an Evolution and Intelligent Design Course Iglika Pavlova BSLC Rm. 208 924 E. 57th St. Chicago, IL, 60637 773-702-0683 iglikap@uchicago.edu INTRODUCTION METHODS Pre/Post Surveys Short Essay Responses Major concepts • Course philosophy • To improve scientific understanding, a successful group of current approaches involves increased access to authentic science experiences. This study reports on a fundamentally different approach that emphasizes explicitly learning principles of good reasoning, in a net cast wider than specific disciplinary examples, and that is informed by philosophy of science, probability, and logic. • The comparison between evolution and Intelligent Design (ID) serves as a springboard for inquiry into issues such as justification and reasoning with uncertainty. • The hypothesis tested in Fall 2009 and Winter 2010 trials is that by the end of the course students will be able to: • Identify and explain features of good explanations • Correct misconceptions about science and justification • Epistemological fundamentals. Issues in forming explanations. • The relationship between knowledge, truth, justification, and belief. Ethics of belief. • Issues in justification, including: • Evidence and belief come in degrees * • Withholding belief can be a justifiable option * • Multiple explanations can account for the same observation * • Better explanations can be identified in comparison with alternatives through their higher explanatory power, well-supported mechanism, lack of ad hoc hypotheses, etc. • Worldviews: Scientific, religious, skeptical, relativistic • Scientific evidence is more reliable than personal experience * • Social aspects of science can be a strength; aspects of justification can go beyond culture, historical period, etc.* • Surveys were given on the second day of class (Pre) and after 10 weeks, during the In-class Final Exam (Post) in 3 parts: • Science attitudes and understanding (19 items) • Epistemic understanding (54 items) • Case scenario (on astrology) • Items for part I (except one) and II are on an extended 7-point Likert scale (Strongly agree +3, +2, +1, 0 No opinion/Don’t know, -1, -2, -3 Strongly disagree) • Each concept tested with 2 or more items, phrased in both affirmative and negative terms, in a randomized order • Survey created over the course of several years, with extensive colleague and student feedback, to fill need for deeper testing for epistemic concepts that seems to be currently unavailable • Short essays were included in several types of assignments: • Worksheets - in class (typically small group work) and homework (typically individual work) • Reflections at the start, middle, and end of the quarter (part of the Final Exam) • Final Exam - In-class and Take-home • Same concept tested 2 or more times throughout the quarter • Same concept tested in 2 or more different ways: • Reflection: What do you think/feel? Compared to before? • Argument:Argue for a specific (given) point • Response to actual or real life-like argument: How would you respond? What would you say? • The guiding principles are that students should: • Be exposed to multiple viewpoints • Obtain tools for critical assessment by explicitly learning concepts in epistemology and philosophy of science. • Apply concepts learned to real-life issues, both for the topic of the class and outside, and on multiple occasions • Critically assess the issues and make up their own minds through continuous reflection • Class combines original readings, lectures, discussions, and short essay responses to prompts (see Methods) • Misconceptions are identified and explicitly addressed in the course. Major concepts that are commonly associated with misconceptions are marked with a * in the box to the left. RESULTS Scientific Understanding – Essay Responses Science Attitudes – Survey & Essay Responses Iglika Pavlova, University of Chicago, Chicago, IL Table 1 Table 3. What are features of good explanations? Table 4. What are common problems with bad explanations? A. A. “ A scientific theory is just a possible explanation of how things are” A. B. B. • "Most of the time, when science contradicts your personal experience about medical issues it is best to trust your personal experience" B. • Table 4. Analysis of responses to the In-class Final exam question “What are common “red flags” for a bad explanation?”. Results presented from Winter 2010 quarter. • Question based on analysis of several case studies of poor explanations from class readings and discussions Table 3. Analysis of responses to the In-class Final Exam question “How can you tell if an explanation is correct?”, asking to identify and explain the features of good explanations that tend to increase their accuracy. Results presented from Winter 2010 quarter.

2. Table 5. “Science is just a lot of models – there is no truth in science” • Figure 1. Detailed results for the 7-point Likert scale responses for 2 of the survey items from Table 1 showing a significant change after the course. Results presented from Fall 2009 and Winter 2010 quarters. • Table 1. Summary for Likert scale responses for 9 of the survey items focusing on science as a truth-finding endevor. Results presented from Fall 2009 and Winter 2010 quarters. • Data for all “Agree” (+3, +2, +1) and “Disagree” (-3, -2, -1) responses are presented as a sum, along with the 0 response for “No opinion/Don’t know” • Survey items are labeled as + for those associated with a positive attitude toward science and/or science as truth-finding, and -- for those associated with a negative attitude and/or limiting the role of science as truth-finding • The Post-Survey responses are color-coded red if there is an increase compared to the Pre-Survey, blue if there is a decrease, and black if there is no change • A. % of students who describe that a good explanation must be compared to alternatives (first column) and who describe 1 to 5 of the 5 major criteria studied in class for choosing a good explanation that is also best among its alternatives (next columns): • More supporting evidence and less disconfirming evidence • More well supported supplemental hypotheses and less ad hoc hypotheses • Well supported mechanism • Genuine higher predictive strength • Genuine higher explanatory power • B. % of students who provide excellent, good, or not acceptable explanations of the criteria they list. • A clear and specific explanation of the criteria is linked to understanding how a criterion is linked with the accuracy of explanations Table 2. Why do scientists disagree? Future Direction Table 5. Analysis of responses to the In-Class Final Exam Response prompt “Someone says to you: “Science is just a lot of models – there is no truth in science”. How do you respond?”. Results reflect % of students who provide excellent, good, acceptable, or not acceptable answers. Results presented from Fall 2009 and Winter 2010 quarters. A different approach Reflective Essay Responses • Selected quotes from reflective essay responses from the Take-home Final Exam (from Fall 2009 and Winter 2010 quarters) • I am now aware of how science is epistemologically distinguished from other pursuits. I can appreciate and value science as a strong mode of inquiry that emphasizes warranted evidence. Because I see that science is an epistemologically distinguished pursuit, I understand why scientific reasoning is an essential component in the search for truth. The correlation here is that epistemologically distinguished endeavors are more likely to yield the truth. Before this course I considered most of my interests to lie outside of science. For example, most of my academic interests are centralized around the humanities. I did not consider science or scientific reasoning, for that matter, to be particularly relevant to me or my pursuits. Prior to this course, if asked to describe scientific reasoning, I likely would have referenced the standardized method science so often employs. I could not have told you, however, what is particularly distinguished about science’s methods. I now feel confident that I can assess my own methods when reaching conclusions and forming beliefs. Following this course I am excited to apply epistemologically distinguished methods to my pursuits in other academic fields, such as the humanities and religious studies. • At the onset of this course, I would have readily accepted Intelligent Design models because they corresponded with my belief in God. Now, however, I can now distinguish a good model from a bad model (through applying Giere’s expanded framework.) I recognize that the models employed by the ID movement are weak and deficient (an occurrence caused largely by the ID movement’s failure to specify the properties of a designer.) Additionally, they provide no exact mechanisms to explain their hypothesis. My faith in God has never hinged on how the world was created, but I assumed that since I believe in God I should just accept models that advocate a designer (no matter how flawed they are.) Following this course, I do not believe that science and religion are mutually exclusive, or that you can only have one or the other. I feel perfectly confident in maintaining a belief in God along with an interest and acceptance of evolutionary biological principles. • I did spend a lot of time on scientific topics in school – physics, biology, etc. However, the idea of scientific inquiry was simply something I learned early on without focusing on its meaning. I knew the process of hypothesis, data collection. In my mind, I always viewed science as a group of specific areas of study. When someone asked me if I liked science, I equated the question with whether or not I liked biology, chemistry, etc. After this class, however, and especially after reading Giere’s chapter, I have come to appreciate more science as a theory… It [science] is a method of investigation that can be applied everywhere. One does not need science to be scientific. The scale of evidence/belief is a way to scientifically approach problems in life, and it really is useful. For arguments, I will now attempt to use this tool, so as not to believe anything without justification, or cite as evidence things that aren’t. • When I first started this class, I thought that you had to believe something 100% or not believe it at all. If there were equally strong reasons for and against an idea, then I thought you still had to choose a side. I believe in objective truth, and thought that I needed to have a strong opinion on every issue. Now I still believe in objective truth, but I understand… that doesn’t mean that I have to be 100% sure of what it is… If there is very little evidence for a claim, then I should have very little belief for the claim. • Learning explicitly principles of reasoning while applying to several real life cases, in a discussion with peers and with an opportunity for feedback, is an approach that has high potential to lead to more stable and deeper knowledge, and one that can be applied in real life and transferred to other fields • Addressing explicitly underlying assumptions, misconceptions and other considerations (such as worldviews) that are not typically addressed in formal coursework may also be very important for a true change in beliefs and lifelong learning • From the data presented here and additional data not presented, it seems that most students in this group come in with an overall positive attitude toward science (even though most plan to declare or have declared a non-science major and informally talk about themselves as “not science” people) • However, many students display doubts about science in several ways or in certain situations that tend to diminish or discount science as truth-finding, such as: • Thinking of scientific results as primarily influenced by financial and political interests as well as cultural and individual biases • Considering personal evidence as more reliable than scientific evidence, even for complex cases such as efficacy of medicines • In addition, while many students have a general idea about the “scientific method” and understand some of the specific considerations, they are missing important components and hold some misconceptions CONCLUSIONS Incoming attitudes and knowledge After the course • After the course, students increase their positive attitude toward sciencefor all relevant Survey items • The Survey (which is anonymous) also indicates that students are changing their minds about issues they had misconceptions about. In particular, studentsfeel more confident in science as a truth-seeking and truth-finding process(Tables 1 and 2 and Figure 1). • Essay responses identify that this confidence comes from amore complete and deeperunderstanding of the process of science, specifically components of good explanations that increase the likelihood of truth-finding (Tables 3, 4, and 5 and reflective essay responses). • Table 2. Results reflect % of students choosing their top 2 choices in answer to the question “What do you think are the most common reasons when scientists disagree on a scientific issue (.e.g., whether global warming is a significant threat or whether artificial sweeteners are safe to eat)?”. Results presented from Fall 2009 and Winter 2010 quarters. • Acknowledgments • Kayla Lewis, co-developer and co-teacher of these courses • Biological Sciences Collegiate Division, and especially Jose Quintans • ASM Research Residency Scholars Program, especially our leaders! • Continue to improve learning in this course! • Complete analysis for Fall 2009 and Winter 2010 quarters • Continue with the studies (longitudinal study; research other questions, such as role of religious belief) • Improve accuracy of the tools for this study by improving existing tools, in collaboration with specialists in the field • Share teaching method with other faculty • Expand teaching method to other populations (beyond 2 pilots, with high school students and grad students/postdocs)