Chapter 13: Experiments and Observational Studies

1. Chapter 13: Experiments and Observational Studies AP Statistics

2. Observational Studies Observational Studies: Researchers observe. They don’t assign choices or manipulate anything (unlike experiments). We just use an existing situation (or data), neither choosing who or what treatments. Example: A recent study showed that men who have had a heart attack, have a greater chance of having a second heart attack if a certain protein is present in their blood.

3. Observational Studies • They are notbased on random samples, nor do they randomly impose treatments. The results cannot be generalized, nor can they show cause-and-effect. • They, however, are not worthless. • They can show us trends and possible relationships—even if we can’t show cause-and-effect. • They can show us variables related to certain outcomes

4. Types of Observational Studies Retrospective:Subjects are selected and then their previous conditions and behaviors are determined • restricted to small part of population • Prone to errors—looking at historical data • Usually focus on estimating differences between groups or associations between variables

5. Types of Observational Studies Prospective: Study where subjects are followed to observe future outcomes. • Focus is on estimating differences among groups that might appear as groups are followed • Because no treatment is applied, it is NOT and experiment.

6. Randomized, Comparative Experiments • Only method by which we can prove cause-and-effect. • We want to see if learning math on a computer is better than learning it in a traditional classroom—randomly assign half of a group of students to classroom where the content was only taught on computer and the other half to a classroom where the content was never taught on the computer, then we would compare the results.

7. Randomized, Comparative Experiments Comparative just means we are comparing the results at the end of the experiment.

8. Randomized, Comparative Experiments Each factor has levels—values that the experimenter chooses for the factors Also called a “factor”

9. Randomized, Comparative Experiments An experiment is designed to test the claim that those people who sleep less than 8 hours a night have a decreased ability to remember information. The experimenter has obtained 50 subjects and has randomly placed them in two groups. All subjects will be given a memory test as a baseline. One group will be required to sleep at least 8 hours for one night and the other groups will be prevented from sleeping 8 hours a night. The next day, each group will be given a test of memory and differences in the test will be recorded.

10. Randomized, Comparative Experiments Important Concepts • The experimenter actively and deliberately manipulates the factors to control the details of the possible treatments. • The subjects are assigned to the treatments randomly.

11. Four Principles of Experimental Design • Control • Randomization • Replicate • Block

12. Control • We want to control sources of variation other than the factors we are testing by making conditions as similar as possible for all treatment groups. • We control a factor by assigning subjects to different factor levels because we want to see how the response will change at those different levels. • We control other sources of variation to prevent them from changing and affecting the response variable.

13. Control • Controlling extraneous sources of variation reduces the variability of the responses, making it easier to detect differences among the treatment groups. • Making generalizations from the experiment to other levels of the controlled factor can be risky.

14. Randomize • Allows us to equalize the effects of unknown or uncontrollable sources of variation • Doesn’t eliminate those effect of these sources, but it spreads them out across all treatment levels, so that they “even out” and can be looked past. • If not randomized, you will not be able to draw conclusions from the experiment • “control what you can, randomize the rest”

15. Replicate • 1st type: We need to repeat the experiment, applying the treatment to a number of subjects. If we don’t assess the variation, it is not complete. The outcome of an experiment on a single subject is an anecdote—not an experiment

16. Replicate • 2nd type: Occurs when our experimental units (subjects) are not representative of the population of interest. We will need to repeat the experiment with a different experimental units. Replication of an entire experiment with the controlled sources of variation at different levels is an essential step in science. If your subjects are from an Intro to Psychology class, you can’t generalize the results—so you will need to replicate the experiment

17. Block • Sometimes random assignment to treatments from our subjects is not the way to go. • Sometimes we need to block. This is when we group experimental subjects that are known before the experiment to be similar in some way that is expected to affect the response to the treatments. • The randomization comes within the blocks—where we assign treatments in each block

18. Logic to Experimental Design • Randomization produces groups of subjects that should be similar in all respects before we apply treatments • Comparative design ensures that influences other than the experimental treatment operate equally on all groups. • Therefore, differences in the response variable must be due to the effects of the treatments

19. Experimental Diagram Diagram of a randomized comparative experiment. An experiment that was designed to test the effectiveness of the drug hydroxyurea for treating sickle cell anemia. There were 299 adult patients who had at least three episodes of pain from sickle cell anemia in the past year.

20. Experimental DesignRandomized Comparative Experiment Experiment to test the effectiveness of see what treatment may reduce the number of repeat offenders.

21. Randomized Blocked Design • Blocking is used instead of randomizing subjects to treatments. • Blocking is used if it is believed that there will be differences in how set groups of subjects will respond to the explanatory variable(s) • Blocking may be used if we suspect that some issue we cannot control may introduce variability in the response. (maybe gender will produce variability in the response, therefore, we will block by gender)

22. Randomized Block Design • Randomization is introduced when we randomly assign treatments within each block. • By blocking, we isolate the variability attributable to the differences between the blocks, so that we can see the differences caused by the treatments more clearly. • We block to reduce variability so we can see the effects of the factors • When we block, we are not usually interested in studying the effects of the blocks themselves (no need to compare the results between/among the blocks)

23. Randomized Block Design Example: Suppose that the drug we are testing works effectively. That should show up as a difference in response between the experimental group and control group. However, if both groups are mixed gender and men and women respond differently to the drug, then the variability between the genders can drown out the true effect of the drug in each gender. We won’t see that the drug is effective.

24. Randomized Block Design Example (cont.): We cannot cope with this variability problem with randomization (can’t randomize by gender). Instead, we block by gender to reduce this variability.

25. Randomized Block Design

26. Matched Pairs Design • Subjects are sometimes paired because they are similar in ways not under the study • When we match subjects in this way we can reduce variability in much the same way as does blocking • If we have study that is trying to determine if playing sports increases mathematical achievement, we might want to pair a subject that has a high IQ and plays a sport, with a subject that has a high IQ and does not play a sport (this would be an observational study). • The matching would reduce the variation due to IQ differences.

27. Matched Pairs Design • When we have a matched pairs design that is an experiment, we need to introduce randomization • Suppose we use a matched pairs design in an experiment that looks at whether or not children can determine the difference between the facial expressions of fear and anger. In this situation, we could match subjects and then randomly assign the order in which the pictures of facial expression are shown. One part of pair will get anger and then fear, the other part of the pair will get fear and then anger.

28. Matched Pairs Design Diagram

29. Confounding and Lurking Variables

30. Confounding and Lurking Variables

31. Confounding and Lurking Variables

32. Confounding and Lurking Variables • Lurking variables are most common in observational studies • Confounding are most common in experiments

33. Statistically Significant How do we know if the results of an experiment really show that there is a difference? Suppose we tested a medication to see if it reduced blood pressure better than on older medication. In our results, we calculated that, on average, the medication reduced blood pressure by about 10% more than the older medication. Is this evidence that the new medication worked better than the new? What if it only reduced it by 3%? 20%

34. Statistically Significant In order to determine if the difference is enough we need to say that the differences are statistically significant. This means that the differences we observed are too big to be explained by chance differences. If they are too big to be explained by chance, then we can attribute the differences to the experiment.

35. Statistically Significant Suppose we flip a coin 100 times and it comes up heads 47 times. Is the coin fair, or is it weighted? We know it should come up heads 50 times (theoretically), but this difference between what we got and what we should get is not large and can be explained by chance differences (it is not unlikely to get those results if the coin was fair). However, if we got 30 heads, we may determine that a difference that big is too large for us to say that it is due to chance differences (it is very unlikely to get those results if the coin was fair). This last result shows that the differences are statistically significant and that the coin is probably not fair.

36. Statistically Significant We will learn how to determine that in later chapters. Always be skeptical of studies and experiments that discuss how much better (or worse) one thing is than another without stating that the results are statistically significant. That is the only way to determine if there really is a difference between two (or more) things.

37. Experiments and Samples(differences, similarities and other info) Similarities • Both use randomization to get unbiased data

38. Experiments and Samples(differences, similarities and other info) Differences • Samples try to estimate the population parameters, so randomizing is an attempt at having the sample be representative of the population • Experiments try to assess the effectiveness of treatments, so randomization is used to assign treatments so as to reduce (eliminate) unwanted problems. Experiments rarely draw their subjects from random samples of the population.

39. Experiments and Samples(differences, similarities and other info) Differences • If our objective is to learn something about a population Sample Survey • If our objective is to see if there is a difference in the effects of two treatments Experiment • If our objective is just to use an existing situation to look for trends and/or contributing factors Observational study

40. Experiments and Samples(differences, similarities and other info) Other • If the subjects in an experiment are not random samples from the population, be cautious about generalizing the results of an experiment until the experiment has been replicated using different subjects, environments, etc. • Experiments typically draw stronger conclusions than surveys (even if experimental subjects are not random samples). • Because, by looking only at the differences across treatment groups, experiments cancel out many of the sources of bias.

41. Control Treatments If we are attempting to see if a medication effectively reduces anxiety, we don’t just want to give all our subjects the medication and record if their anxiety decreased or not. Instead, we want to compare how much their anxiety decreased compared to people who did not take the medication. This comparison group is the control group and that group’s measurement is called the control treatment.

42. Blinding Blinding is when we create an environment where subjects and/or researchers (physicians, technicians, psychologists, etc) do not know who gets treatment and who gets placebo. Who can affect the outcome of an experiment? • Those who could influence the results (the subjects, treatment administrators, or technicians, etc) • Those who evaluate the results (judges, treating physicians, etc)

43. Blinding When individuals in one of those two groups is blinded we say that the experiment is single-blind. When individuals in both of those groups are blinded we say that the experiment is double-blind.

44. Blinding It is important to blind because it is easy for a person’s knowledge about which treatment is given to which people to influence there actions and beliefs. Therefore, blinding will eliminate this form of bias.

45. Placebo A placebo is a “fake” treatment that looks just like the treatments being tested. It many times is used for the control group’s treatment It is the best way to blind subjects

46. Placebo Sometimes groups who are treated with the placebo (control group) will show an improvement. This is not uncommon. This effect (when the control group show an improvement when treated with placebo) is called the placebo effect. It is not uncommon that 20% or more of subjects who are given a placebo report things such as reduction of pain, decreased depression and improved health issues

47. Characteristics of Good Experiments • Randomized • Comparative • Double-Blind • Placebo-controlled