CHAPTER 4 Designing Studies

1. CHAPTER 4Designing Studies 4.2 Experiments

2. Experiments • DISTINGUISH between an observational study and an experiment. • EXPLAIN the concept of confounding. • IDENTIFY the experimental units, explanatory and response variables, and treatments in an experiment.

3. Observational Study vs. Experiment In an observational study, treatments are not imposed. They look at data for a sample of individuals (retrospective) or follow a sample of individuals into the future collecting data (prospective). In an experiment, different treatments are assigned to participants. The distinction between observational study and experiment is one of the most important in statistics.

4. Observational Study vs. Experiment • The purpose of an observational study can be to describe some group or situation, to compare groups, or to examine relationships between variables. • The purpose of an experiment is to determine whether the treatment causes a change in the response. • When our goal is to understand cause and effect, experiments are the only source of fully convincing data.

5. Observational Study or Experiment? • To find out if reducing screen brightness increases battery life in laptop computers, researches obtained 30 new laptops of the same brand. They chose 15 of the computers at random and adjusted their screens to the brightest setting. The other 15 laptop screens were left at the default setting-moderate brightness. Researchers then measured how long each machine's battery lasted. • To find out if there is an association between video game usage and academic performance in teenage boys, a sample of 1,000 teenage boys across the United States was surveyed. They were asked how many hours per week they spend playing video games and their current GPA.

6. Does Taking Hormones Reduce Heart Attack Risk after Menopause? Should women take hormones such as estrogen after menopause, when natural production of these hormones ends? In 1992, several major medical organizations said Yes. Women who took hormones seemed to reduce their risk of heart attack by 35% to 50%. The risks of taking hormones appeared small compared with the benefits. The evidence came from observational studies that compared women who were taking hormones with others who were not. But the women who chose to take hormones were richer, better educated, and saw doctors more often than women who didn’t take hormones. Because the women who took hormones did many other things to maintain their health, it isn’t surprising that they had fewer heart attacks.

7. Observational Study vs. Experiment Observational studies of the effect of an explanatory variable on a response variable often fail because of confounding between the explanatory variable and one or more other variables. Well-designed experiments take steps to prevent confounding. A confounding variable in an experiment is a variable that is related to the explanatory variable and influences the response variable, and may create a false perception of association between the two. If asked to identify a possible confounding variable in a setting, you are expected to explain how the variable you choose is associated with the explanatory variable and affects the response variable.

8. Confounding Example To find out if there is an association between video game usage and academic performance in teenage boys, a sample of 1,000 teenage boys across the United States was surveyed. They were asked how many hours per week they spend playing video games and their current GPA. What are some examples of possible confounding variables? Grade level, class schedule, amount of time per week spent on homework, etc. When do they become confounding? When you can't tell whether that variable or the length of time spent on video games is effecting the GPA.

9. Early Reading Interventions linked to lower rate of Advanced Placement Coursework In a certain Midwest suburban school district, students who were identified in first grade as having difficulty reading were later found to have a lower-than-average number of Advanced Placement courses on their high school transcript. The students identified as having reading problems in first grade were given an intensive reading intervention program and successfully exited the program after a year. At the end of the high school years, those students were found to be less likely than their peers to have taken at least one AP course in high school. • Based on this study, should we conclude that early reading difficulty causes a student to avoid AP courses or to not successfully earn credit in AP courses? Explain. • Explain the concept of confounding in the context of this study. • Is there any way to prove that early reading difficulty leads to AP coursework difficulty or avoidance?

10. The Language of Experiments An experiment is a statistical study in which we actually do something (a treatment) to people, animals, or objects (the experimental units) to observe the response. Here is the basic vocabulary of experiments.

11. Designing Experiments Researchers are investigating the effectiveness of using a fungus to control the spread of an insect that destroys trees. The researchers will create four different concentrations of fungus mixtures: 0 milliliters per liter (ml/L), 1.25 ml/L, 2.5 ml/L, and 3.75 ml/L. An equal number of the insects will be placed into 20 individual containers. The group of insects in each container will be sprayed with one of the four mixtures, and the researchers will record the number of insects that are still alive in each container one week after spraying.

14. Example Does adding fertilizer affect the productivity of tomato plants? How about the amount of water given to the plants? To answer these questions a gardener plants 24 similar tomato plants in identical pots in his greenhouse. He will add fertilizer to the soil in half of the pots. Also, he will water 8 of the plants with 0.5 gallon of water per day, 8 of the plants with 1 gallon water per day, and the remaining 8 plants with 1.5 gallon water per day. At the end of three months, he will record the total weight of tomatoes produced on each plant. Identify the experimental units, explanatory and response variables, and list all of the treatments in this experiment.

15. Experiments • DISTINGUISH between an observational study and an experiment. • EXPLAIN the concept of confounding. • IDENTIFY the experimental units, explanatory and response variables, and treatments in an experiment.

16. Homework Pg. 259-260 #45-55 odd

17. Experiments • EXPLAIN the purpose of comparison, random assignment, control, and replication in an experiment. • DESCRIBE a completely randomized design for an experiment. • DESCRIBE the placebo effect and the purpose of blinding in an experiment. • INTERPRET the meaning of statistically significant in the context of an experiment.

18. Are Online SAT Prep Courses Effective? A high school regularly offers a review course to prepare students for the SAT. This year, budget cuts will allow the school to offer only an online version of the course. Suppose the group of students who take the online course earn an average increase of 45 points in their math scores from a pre-test to the actual SAT test. Can we conclude that the online course is effective? This experiment has a very simple design. A group of subjects (the students) were exposed to a treatment (the online course), and the outcome (increase in math scores), was observed. A closer look showed that many of the students in the online review course were taking advanced math classes in school. Maybe the students in the course improved their scores because of what they were learning in their math classes, not because of the online course. This confounding prevents us from concluding that the online course is effective.

19. How to Experiment Badly Many laboratory experiments use a design like the one in the online SAT course example: Experimental Units Treatment Measure Response In the lab environment, simple designs often work well. Field experiments and experiments with animals or people deal with more variable conditions. Outside the lab, badly designed experiments often yield worthless results because of confounding.

20. How to Experiment Well The remedy for confounding is to perform a comparative experiment in which some units receive one treatment and similar units receive another. Most well designed experiments compare two or more treatments. Comparison alone isn’t enough, if the treatments are given to groups that differ greatly, bias will result. The solution to the problem of bias is random assignment. In an experiment, random assignment means that experimental units are assigned to treatments using a chance process.

21. Principles of Experimental Design Principles of Experimental Design The basic principles for designing experiments are as follows: 1. Comparison. Use a design that compares two or more treatments. 2. Random assignment. Use chance to assign experimental units to treatments. Doing so helps create roughly equivalent groups of experimental units by balancing the effects of other variables among the treatment groups. 3. Control. Keep other variables that might affect the response the same for all groups. 4. Replication. Use enough experimental units in each group so that any differences in the effects of the treatments can be distinguished from chance differences between the groups.

22. When a tractor pulls a plow through an agricultural field, the energy needed to pull that plow is called the draft. The draft is affected by environmental conditions such as soil type, terrain, and moisture. A study was conducted to determine whether a newly developed hitch would be able to reduce draft compared to the standard hitch. (A hitch is used to connect the plow to the tractor.) Two large plots of land were used in this study. It was randomly determined which plot was to be plowed using the standard hitch. As the tractor plowed that plot, a measurement device on the tractor automatically recorded the draft at 25 randomly selected points in the plot. After the plot was plowed, the hitch was changed from the standard one to the new one, a process that takes a substantial amount of time. Then the second plot was plowed using the new hitch. Twenty-five measurements of draft were also recorded at randomly selected points in this plot. Describe how the 4 principles of experimental design were or were not included in this experiment.

23. Example: The Physicians’ Health Study Read the example in your notes about determining if aspirin or beta-carotene help reduce heart attacks. Explain how each of the four principles of experimental design was used in the Physician’s Health Study.

24. Random Assignment How can we randomize (randomly assign n individuals to the treatments)? Hat Method: Write all the names on identical papers in a hat, mix well, select n of them. The names you choose indicate those assigned to the treatment. Table D: Number each item, using labels that are equal lengths (1-9, 01-99, 001 to 999, etc.) & then use a random number table to choose n unique ___ digit numbers. The individuals linked to those numbers are assigned to the treatment. Technology: Number each item from 1 to some value & then use technology to choose n unique numbers. The individuals linked to those numbers are assigned to the treatment.

25. Example Using the Physicians’ Health Study, describe how you would randomly assign the 21,996 participants to the four treatment groups using 21,996 identical slips of paper.

26. Completely Randomized Design In a completely randomized design, the treatments are assigned to all the experimental units completely by chance. Some experiments may include a control group that receives an inactive treatment (placebo) or no treatment at all in order to determine if the treatment has an effect. Treatment 1 Group 1 Experimental Units Compare Results Random Assignment Group 2 Treatment 2

27. Control Group • The primary purpose of a control group is to provide a baseline for comparing the effects of other treatments. • Usually a control group receives an inactive treatment, but not always. • Some experimental designs do not include a control group. That’s appropriate if researchers simply want to compare the effects of several treatments and not to determine whether any of them works better than an inactive treatment.

28. Example Many utility companies have introduced programs to encourage energy conservation among their customers. An electric company considers placing small digital displays in households to show current electricity use and what the cost would be if this use continued for a month. Will the displays reduce electricity use? One cheaper approach is to give customers a chart and information about monitoring their electricity use from their outside meter. Would this method work almost as well? The company decides to conduct an experiment to compare these two approaches (display, chart) with a group of customers who receive information about energy consumption but no help in monitoring electricity use. Describe a completely randomized design involving 60 single-family residences in the same city whose owners are willing to participate in such an experiment. Write a few sentences explaining how you would implement your design.

29. Experiments: What Can Go Wrong? The logic of a randomized comparative experiment depends on our ability to treat all the subjects the same in every way except for the actual treatments being compared. Good experiments, therefore, require careful attention to details to ensure that all subjects really are treated identically. Single-Blind - when either subjects or evaluators don't know which group subjects were assigned to. Double-Blind - when neither subjects nor evaluators know which group subjects were assigned to. Placebo - treatment (in medical studies) contains no medication, used in "blind" studies. Placebo Effect – occurs when experimental units have a response to a placebo Control Group - receives no treatment or a placebo

30. Placebo Effect • We want to use placebos so that the placebo effect does not become a confounding variable. • Say we are testing a new pill to cure arthritis, and half of the subjects get it and half get no pill. The arthritis was cured in 32% of the subjects who took the pill-is it because of the medication or because they were taking a pill? WE CAN’T TELL!! • If half were taking the pill and half were taking a placebo, we could say that the cure for arthritis in the medicated group was because of the medication and not just from taking a pill.

31. Example In an interesting experiment, researchers examined the effect of ultrasound on birth weight. Pregnant women participating in the study were randomly assigned to one of two groups. The first group of women received an ultrasound; the second group did not. When the subjects’ babies were born, their birth weights were recorded. The women who received the ultrasounds had heavier babies. a) Did the experimental design take the placebo effect into account? Why is this important? b) Was the experiment double-blind? Why is this important? c) Based on your answers above, describe an improved design for the experiment.

32. Inference for Experiments In an experiment, researchers usually hope to see a difference in the responses so large that it is unlikely to happen just because of chance variation. We can use the laws of probability, which describe chance behavior, to learn whether the treatment effects are larger than we would expect to see if only chance were operating. If they are, we call them statistically significant. An observed effect so large that it would rarely occur by chance is called statistically significant. A statistically significant association in data from a well-designed experiment does imply causation.

33. Experiments • EXPLAIN the purpose of comparison, random assignment, control, and replication in an experiment. • DESCRIBE a completely randomized design for an experiment. • DESCRIBE the placebo effect and the purpose of blinding in an experiment. • INTERPRET the meaning of statistically significant in the context of an experiment.

34. Homework Pg. 261-262 #57-63 odd, 69, 71

35. Experiments • EXPLAIN the purpose of blocking in an experiment. DESCRIBE a randomized block design or a matched pairs design for an experiment.

36. A Smarter Design? Suppose that a mobile phone company is considering two different types of keyboard designs (A and B) for its new smart phone. The company decides to perform an experiment to compare the two keyboards using a group of 10 volunteers. The response variable is typing speed, measured in words per minute. How should the company deal with the fact that four of the volunteers already use a smart phone, whereas the remaining six do not?

37. Blocking When a population consists of groups of individuals that are “similar within but different between,” a stratified random sample gives a better estimate than a simple random sample. This same logic applies in experiments. A block is a group of experimental units that are known before the experiment to be similar in some way that is expected to affect the response to the treatments. In a randomized block design, the random assignment of experimental units to treatments is carried out separately within each block.

38. Blocking Form blocks based on the most important unavoidable sources of variability (possible confounding variables) among the experimental units. Randomization will average out the effects of the remaining possible confounding variables and allow an unbiased comparison of the treatments. Control what you can, block on what you can’t control, and randomize to create comparable groups.

39. Example: Doing the Laundry Suppose researchers want to test whether a new detergent for clothes that require hand-washing cleans better in warm or in cold water. They decide to perform an experiment using 200 pieces of dirty laundry as the experimental units. The response variable is a cleanliness rating on a scale of 0 (very dirty) to 10 (very clean). a) How should researchers deal with the fact that light-colored clothing tends to come cleaner in warm water? b) Make an outline of a randomized block design for this experiment. Assume there are 80 pieces of light-colored clothing and 120 pieces of dark-colored clothing. Describe how you would carry out the random assignment required by your design.

40. Example The progress of a type cancer differs in women and men. Researchers want to design an experiment to compare three therapies for this cancer. They recruit 500 male and 300 female patients who are willing to serve as subjects. • Which are the blocks in this experiment: the cancer therapies or the two sexes? Why? • What are the advantages of a randomized block design over a completely randomized design using these 800 subjects? • Suppose the researchers had 800 male and no female subjects available for the study. What advantage would this offer? What disadvantage?

41. Matched Pairs Design A common type of randomized block design for comparing two treatments is a matched pairs design. The idea is to create blocks by matching pairs of similar experimental units. A matched pairs design is a randomized blocked experiment in which each block consists of a matching pair of similar experimental units. Chance is used to determine which unit in each pair gets each treatment. Sometimes, a “pair” in a matched-pairs design consists of a single unit that receives both treatments. Since the order of the treatments can influence the response, chance is used to determine with treatment is applied first for each unit.

42. Standing and Sitting Pulse Rate We want to measure whether sitting or standing results in higher pulse rates. How could we use the people in this class to test this using a matched-pairs design?

43. Example Cardiologists at Athens Medical School in Greece wanted to test whether chocolate affected blood flow in the blood vessels. The researchers recruited 17 healthy young volunteers, who were each given a 3.5-ounce bar of dark chocolate, either bittersweet or fake chocolate. On another day, the volunteers received the other treatment. The order in which subjects received the bittersweet and fake chocolate was determined at random. The subjects had no chocolate outside the study, and investigators didn’t know whether a subject had eaten the real or the fake chocolate. An ultrasound was taken of each volunteer’s upper arm to see the functioning of the cells in the walls of the main artery. The researchers found the blood vessel function was improved when the subjects ate bittersweet chocolate, and that there were no such changes when they ate the placebo.

44. Questions • What type of design did the researchers use in their study? • Explain why the researchers chose this design instead of a completely randomized design. • Why is it important to randomly assign the order of the treatments for the subjects? • Explain how and why researchers controlled for other variables in this experiment.

45. Experiments • DISTINGUISH between an observational study and an experiment. • EXPLAIN the concept of confounding. • IDENTIFY the experimental units, explanatory and response variables, and treatments in an experiment. • EXPLAIN the purpose of comparison, random assignment, control, and replication in an experiment. • DESCRIBE a completely randomized design for an experiment. • DESCRIBE the placebo effect and the purpose of blinding in an experiment. • INTERPRET the meaning of statistically significant in the context of an experiment. • EXPLAIN the purpose of blocking in an experiment. DESCRIBE a randomized block design or a matched pairs design for an experiment.

46. Homework Pg. 262-265 #73, 75, 79, 85, 87-94