Safeguards Against Error in Research Methods

1. 2 Research Methods Safeguards against error Slides prepared by Matthew Isaak

2. Lecture Preview • The beauty and necessity of good research design • The scientific method • Ethical issues in research design • Statistics • Evaluating psychological research

3. Why We Need Research Designs LO 2.1 Identify two modes of thinking and their application to scientific reasoning. • In the early 1990s, an autism treatment was developed called "facilitated communication." • The developers thought that autism was a motor disorder. • The facilitator sat next to child with autism and guided the child's hand over a keyboard, allowing the children to type out words.

4. Why We Need Research DesignsLO 2.1 Identify two modes of thinking and their application to scientific reasoning. • Students seemed to make stunning progress in communication, telling parents "I love you" and writing poetry. • However, some students began making allegations of sexual abuse against parents. • There was no physical evidence, just the communicators via the facilitators.

5. Why We Need Research DesignsLO 2.1 Identify two modes of thinking and their application to scientific reasoning. • Dozens of controlled studies examined the phenomenon and found that the words came solely from the minds of the facilitators. • Still, some people continue to practice facilitated communication.

6. Figure 2.1 Putting Facilitated Communication to the Test.

7. Why We Need Research DesignsLO 2.1 Identify two modes of thinking and their application to scientific reasoning. • Even well-educated, intelligent people can be fooled. • Well-planned designs can help to eliminate biases when examining phenomena.

8. Why We Need Research DesignsLO 2.1 Identify two modes of thinking and their application to scientific reasoning. • Prefrontal lobotomy is example of what happens when we rely on subjective impressions. • Developer won the Nobel Prize • In it, the neural fibers connecting frontal lobes to the thalamus were severed. • Control studies showed it didn't work.

9. Common SenseLO 1.1 Explain why psychology is more than just common sense. Most of us trust our gut intuitions about how the world works. Birds of a feather flock together. Absence makes the heart grow fonder. Better safe than sorry. Two heads are better than one. Actions speak louder than words. Opposites attract. Out of sight, out of mind. Nothing ventured, nothing gained. Too many cooks spoil the broth. The pen is mightier than the sword. Even though each of these ring true, they are in fact opposites!

10. Naïve RealismLO 1.1 Explain why psychology is more than just common sense. • The belief that we see the world precisely as it actually is in truth – “seeing is believing”Works well in ordinary life, but consider: • The earth seems flat. • We seem to be standing still, yet the earth is moving around the sun 18.5 miles/sec.

11. Which table is longer?LO 1.1 Explain why psychology is more than just common sense. These two tabletops are identical in length.

12. When Common Sense is RightLO 1.1 Explain why psychology is more than just common sense. • Not all common sense is wrong. • Common sense should serve as a generator for hypotheses, which can then be tested. • But learning to think like a scientist means learning when—and when not—to trust our common sense.

13. Theories and HypothesesLO 1.2 Explain the importance of science as a set of safeguards against biases. • A scientific theory is an explanation for a large number of findings in the natural world. • A hypothesis is a specific prediction based on a theory, which can then be tested. • Theories are general explanations; hypotheses are specific predictions derived from them.

14. Theory MisconceptionsLO 1.2 Explain the importance of science as a set of safeguards against biases. “A theory explains one specific event.” “A theory is just an educated guess.” Why are these both wrong?

15. Science as a Safeguard against BiasLO 1.2 Explain the importance of science as a set of safeguards against biases. • Confirmation bias • Tendency to seek out evidence that supports our hypothesis and neglect or distort contradicting evidence • Scientists need to design studies that may disprove their theories.

16. Figure 1.3 Diagram of Wason Selection Task. In the Wason selection task, you must pick two cards to test the hypothesis that all cards that have a vowel on one side have an odd number on the other. Which two will you select?

17. Science as a Safeguard against BiasLO 1.2 Explain the importance of science as a set of safeguards against biases. • Belief perseverance • Tendency to stick to our initial beliefs even when evidence contradicts them • The “don’t confuse me with the facts” bias

18. Metaphysical ClaimsLO 1.2 Explain the importance of science as a set of safeguards against biases. • Non-testable assertions that fall outside the realm of science • The existence of God, the soul, or the afterlife

19. FIGURE 1.4 Nonoverlapping Realms. Scientist Stephen Jay Gould (1997) argued that science and religion are entirely different and nonoverlapping realms of understanding the world. Science deals with testable claims about the natural world that can be answered with data, whereas religion deals with untestable claims about moral values that can’t be answered with data. Although not all scientists and theologists accept Gould’s model, we adopt it for the purposes of this textbook. (Source: Gould, 1997)

20. We Might Be WrongLO 1.2 Explain the importance of science as a set of safeguards against biases. • Good scientists are aware they might be wrong. • Scientific knowledge is always tentative and open to revision. • Science forces us to question our findings and conclusions.

21. Thinking ClearlyLO 1.4 Identify reasons we are drawn to pseudoscience. • Learning to think scientifically can help us avoid falling prey to pseudoscience. • Emotional reasoning fallacy—using emotions rather than evidence as the guide • Bandwagon fallacy—Lots of people believe it, so it must be true. • “Not me” fallacy—Other people may have those biases, but not me.

22. Why Should We Care?LO 1.4 Identify reasons we are drawn to pseudoscience. • Because pseudoscience can be very dangerous. • Three major reasons to be concerned: • Opportunity cost • Direct harm • Inability to think scientifically • Although not foolproof, scientific thinking is our best safeguard against human error.

23. Scientific SkepticismLO 1.5 Identify the key features of scientific skepticism. • Being scientifically skeptical does not mean being closed-minded. • Evaluate claims with an open mind, but insist on persuasive evidence before accepting them. • Skeptics are willing to change their minds, but must have good evidence before doing so.

24. Critical ThinkingLO 1.6 Identify and explain the text’s six principles of scientific thinking. • A set of skills for evaluating all claims in a open-minded and careful fashion • This allows us to overcome our own biases (especially the confirmation bias). • Six critical thinking principles will be emphasized in this course.

25. Critical Thinking PrinciplesLO 1.6 Identify and explain the text’s six principles of scientific thinking. • Ruling out rival hypotheses • Have important alternate explanations for the finding been considered? • Correlation isn’t causation • Can we be sure A causes B? • Falsifiability • Can the claim be disproven?

26. Critical Thinking PrinciplesLO 1.6 Identify and explain the text’s six principles of scientific thinking. • Replicability • Can the results be duplicated in other studies? • Extraordinary claims require extraordinary evidence • Is the evidence as convincing as the claims?

27. Critical Thinking PrinciplesLO 1.6 Identify and explain the text’s six principles of scientific thinking. • Occam’s razor • Does a simpler explanation fit the data just as well? • Parsimony: logical simplicity

28. Figure 2.2 The Prefrontal Lobotomy.

29. Two Modes of ThinkingLO 2.1 Identify two modes of thinking and their application to scientific reasoning. System I or intuitive thinking • Quick, reflexive, almost automatic • Relies on heuristics System 2 or analytical thinking • Slow, reflexive, effortful

30. Two Modes of ThinkingLO 2.1 Identify two modes of thinking and their application to scientific reasoning. • Heuristics are mental shortcuts or rules of thumb that we use daily. • They reduce the cognitive energy required to solve problems but we oversimplify reality. • Imagine yourself driving from Reno, Nevada to San Diego, California—which compass direction would you take?

31. San Diego is actually EAST of RenoLO 2.1 Identify two modes of thinking and their application to scientific reasoning. FIGURE 2.3 In Which Compass Direction Would You Travel to Get from Reno, NV to San Diego, CA?

32. So, how do we prevent ourselves from being fooled by our own (and other people's) biases? LO 2.1 Identify two modes of thinking and their application to scientific reasoning.

33. The Scientific Method Toolbox LO 2.2 Describe the advantages and disadvantages of using naturalistic observation, case studies, self-report measures, and surveys. • Allows us to test specific hypotheses derived from broader theories of how things work. • Theories are never "proven," but hypotheses can be confirmed or disconfirmed. • We can use a number of different types of SM tools to gain information and test hypotheses.

34. Naturalistic ObservationLO 2.2 Describe the advantages and disadvantages of using naturalistic observation, case studies, self-report measures, and surveys. • Watching behavior in real-world settings • High degree of external validity - extent to which we can generalize our findings to the real world • Low degree of internal validity - extent to which we can draw cause-and-effect inferences

35. Case Study DesignsLO 2.2 Describe the advantages and disadvantages of using naturalistic observation, case studies, self-report measures, and surveys. • Studying one person or a small number of people for an extended period of time • Common with rare types of brain damage or mental illness • Helpful in providing existence proofs, but can be misleading and anecdotal

36. Self Report Measures and SurveysLO 2.2 Describe the advantages and disadvantages of using naturalistic observation, case studies, self-report measures, and surveys. • Psychologists often need to ask people about themselves or others. • Self-report measures or questionnaires asses characteristics such as personality or mental illness. • Surveys ask about a person's opinions or abilities. • Not all measures and surveys are equal.

37. Random SelectionLO 2.2 Describe the advantages and disadvantages of using naturalistic observation, case studies, self-report measures, and surveys. • The key to generalizability in surveys and questionnaire studies • Ensures every person in a population has an equal chance of being chosen to participate • Non-random selection can skew results and make them inaccurate when applied to the population as a whole.

38. Evaluating MeasuresLO 2.2 Describe the advantages and disadvantages of using naturalistic observation, case studies, self-report measures, and surveys. • To trust results, the measures must have: • Reliability—consistency of measurement • Validity—extent to which a measure assesses what it claims to measure • A test must be reliable to be valid, but a reliable test can still be completely invalid.

39. Self-Report MeasuresLO 2.2 Describe the advantages and disadvantages of using naturalistic observation, case studies, self-report measures, and surveys. • Pros • Easy to administer • Direct (self) assessment of person's state

40. Self-Report MeasuresLO 2.2 Describe the advantages and disadvantages of using naturalistic observation, case studies, self-report measures, and surveys. • Cons • Accuracy is skewed for certain groups (narcissists) • Potential for dishonesty • Response sets - tendencies of research subjects to distort their responses • Positive impression management • Malingering

41. Rating DataLO 2.2 Describe the advantages and disadvantages of using naturalistic observation, case studies, self-report measures, and surveys. • People can also be asked to rate others on different characteristics. • This can do away with some biases in self-report, but still has problems. • Halo effect—tendency of ratings of one positive characteristic to spill over to influence the ratings of other characteristics

42. Correlational Designs LO 2.3 Describe the role of correlational designs and distinguish correlation from causation. • Examine how two variables are related • Correlations vary from -1 to +1 and can be: • Positive (as one increases, so does the other) • Negative (as one increases, the other decreases) • Zero (no relationship between variables) • Depicted in a scatterplot

43. ScatterplotsLO 2.3 Describe the role of correlational designs and distinguish correlation from causation. Figure 2.4 Diagram of Three Scatterplots.

44. Correlational DesignsLO 2.3 Describe the role of correlational designs and distinguish correlation from causation. • Illusory Correlation—perception of a statistical association where none exists • Crime rates and the full moon • Arthritis and weather • Examining a probability table helps to explain why we are all prone to seeing relationships where none exists.

45. The Great Fourfold Table of LifeLO 2.3 Describe the role of correlational designs and distinguish correlation from causation. Did a crime occur? Yes No Yes Did a full moon occur? No Humans tend to overemphasize cell A and ignore the non-events.

46. Correlation vs. CausationLO 2.3 Describe the role of correlational designs and distinguish correlation from causation. • Just because two things are related, does not mean that one causes another. • There are three possible explanations: • A causes B • B causes A • C causes both A and B

47. Determining Causation LO 2.4 Identify the components of an experiment, the potential pitfalls that can lead to faulty conclusions, and how psychologists control for these pitfalls. • The only way to determine if one thing is casually related to another is via an experimental design. • This is because in an experiment, you purposefully manipulate variables, rather than just measure already existing differences.

48. What Makes a Study an Experiment?LO 2.4 Identify the components of an experiment, the potential pitfalls that can lead to faulty conclusions, and how psychologists control for these pitfalls. • Random assignment of participants groups • Experimental Group - receives the manipulation • Control Group - does not receive the manipulation

49. What Makes a Study an Experiment?LO 2.4 Identify the components of an experiment, the potential pitfalls that can lead to faulty conclusions, and how psychologists control for these pitfalls. • Manipulation of an independent variable • The dependent variable is what the experimenter measures to see whether manipulation had an effect.

50. What Makes a Study an Experiment?LO 2.4 Identify the components of an experiment, the potential pitfalls that can lead to faulty conclusions, and how psychologists control for these pitfalls. • Confounds - any difference between the experimental and control groups aside from IV • Makes IV effects uninterpretable • Cause and effect - possible to infer, with random assignment and manipulation of independent variable