PART I: Experimental Design

1. PART I: Experimental Design CHAPTER 1—Controlled Experiments

2. PART I – EXPERIMENTAL DESIGN:CHAPTER 1 — CONTROLLED EXPERIMENTS Experiments are done to see if a treatment has an effect. The treatment could be a drug, an advertising campaign, an educational method, etc. If the researcherdecideswhogets the treatment, the investigation is called an experiment. If the researcher just observes the effects of the treatment, but plays no part in choosing who gets the treatment then the investigation is called an observational study (see next chapter). If the experiment compares the effects of getting the treatment to not getting the treatment, it’s called a controlled experiment. If the researcher just gives the treatment to a group of subjects and records the results without making an explicit comparison, it’s called an uncontrolled experiment. In a controlled experiment the researcher decides who gets the treatment (treatment group) and then compares their responses to the responses of those who didn’t get the treatment (control or comparison group).

3. Page 1 cont. `` ``` MAIN IDEA of a controlled experiment: Controls should be as much like the treatment group as possible– the only difference being the treatment. That way any differences in the responses of the 2 groups can be attributed to the treatment and not to some other difference between the groups. CONTROL TREATMENT

4. Discussion Question --- Not in Notes Suppose we wanted to test whether or not a drug designed to make students learn Stat 100 better really worked. What would be the ideal experimental design?

5. Clicker/Discussion Question --- Not in Notes Suppose we wanted to test whether or not a drug designed to make students learn Stat 100 better really worked. Which of the following would be the ideal experimental design? A) Give all 1200 students the drug this semester and compare their grades to students of previous semesters? B) Give the drug to those students who want it and then compare them to those who didn’t want the drug. C) Have the instructors decide in the best interests of the students who would benefit the most from the drug and who wouldn't, and then make assignments to treatment and control that way. D) Divide the class into 2 groups that are as alike as possible in all relevant ways (ACT scores, previous math courses, GPA, etc) and assign one group to treatment and the other to control. E) Randomly assign each student to treatment or control.

6. CLASS EXPERIMENT: On the next page you’ll see a set of numbers. Randomly pick one of the numbers. The numbers will only be shown for a few seconds so pay attention.

7. 1 2 3 4

9. TYPES OF CONTROLS in Designed Experiments: Randomized and Non-randomized page 2 Randomized-The evaluator uses an objective chance procedure (like flipping a coin) to decide who’s in the treatment group and who’s in the control group. Pros: Cons: Non-Randomized: The treatment and control group are chosen by some other non-random method. 4 Basic Types of Non-randomized Controls: 1. Experimenter tries to match treatment and control to make the 2 groups as alike as possible. Pros: Cons:

10. Different Types of Non-Randomized Controls (Page 2 cont.) 2. Historical—Experimenter chooses people to be in treatment group and compares them to subjects from the past who didn’t get the treatment. Pros: Cons: 3.Experiment chooses the treatment and control groups in the best interests of the subjects. Pros: Cons: 4.Experimenter decides to let the subjects choose whether to be in treatment or control group. Pros: Cons:

11. Chapter 1 Page 3 Random assignment is ideal. As long as there are enough subjects, flipping a coin to decide who's in treatment and who's in control is ideal for 2 basic reasons: • Random assignment is most likely to make treatment and control groups as alike as possible because it eliminates human bias. With enough subjects, random differences average out, not only on the characteristics that the researcher has identified as relevant, but on all characteristics, including hidden ones that the researcher might not realize are important • Random controls are based on chance procedures. Without a chance process we can’t use statistics to analyze whether the treatment really worked. This will be explained later in the course but the basic idea is this: You see a difference in the responses of the treatment and control group. There’ll always be some difference in responses even if the treatment is a dud, just due to chance. So the question is how big a difference must you observe before you can determine that it’s real and not just due to chance. Without random controls you have no way of answering that.

12. Chapter 1 Page 3 cont. BLIND and DOUBLE-BLIND controlled experiments Subject Bias-The Placebo Effect- Whether people think they’ve received a treatment affects their response. To separate the effects of the actual treatment from the idea of treatment, the assignment to treatment and control should be blind. In other words, the subjects shouldn’t know which group they’re in. This can be achieved by giving the control group a fake treatment called a placebo.Since both groups believe they’ve been given the treatment, any differences in their responses can be attributed to the treatment itself and not the idea of treatment. Evaluator Bias- Knowing which subjects received the treatment and which didn’t can bias those evaluating the results. To eliminate such bias, the evaluators should blind to this knowledge. An experiment is called double-blind when neither the subjects nor those evaluating their responses know who’s in the treatment group and who’s in the control group. IDEAL EXPERIMENTAL DESIGN: Randomized, controlled, double-blinddesign  Basic Method is comparison. Investigator randomly assigns subjects into 2 groups, treatment and control. Treatment gets treatment, control gets placebo. The responses of the 2 groups are compared to see if the treatment made any difference. Double-blind: neither the subjects nor the experimenters who measure their responses know who has received the treatment and who has received a placebo.

13. Chapter 1 Page 4 Example 1: Salk Vaccine Field Trial In1954 Jonas Salk developed a killed-virus vaccine that produced antibodies against polio in the lab. The vaccine needed to be tested in the real world. Some polio facts: Caused by virus Prone to Epidemics Difficult to Diagnose Higher incidence among higher income people Clicker Question Who should get the vaccine (treatment group) and who should be in the control group? 2 Options that were considered but not done: • Vaccinate all school-age children and see if the incidence of polio goes down. i) What type of controls? • Random • Non-Random, Researcher tries to match treatment and control • Non-Random Historical • Non-Random, Researcher lets subjects decide • Non-Random, Researcher decides in subjects’ best interest   ii) Main Problem with this design?  A. No Placebo B. Difficult to Diagnose C. Prone to Epidemics D. Higher incidence among higher income people E) None of the Above

14. Chapter 1 Clicker Page 4 cont. Example 1 cont.  Some polio facts: • Caused by virus • Prone to Epidemics • Difficult to Diagnose • Higher incidence among higher income people Who should get the vaccine (treatment group) and who should be in the control group? Second Option that were considered but not done: 2. Let the parents decide if they want their children to be vaccinated or not and compare those who volunteer to those who don’t. i) What type of controls? • Random • Non-Random, Researcher tries to match treatment and control • Non-Random Historical • Non-Random, Researcher lets subjects decide • Non-Random, Researcher decides in subjects’ best interest   ii) Main Problem with this design? Important fact: Parents who consent tend to be wealthier than parents who don’t consent.  A. No Placebo B. Difficult to Diagnose C. Prone to Epidemics D. Volunteers are richer are more high risk for polio than non-volunteers E. None of the Above

15. 2 Options that were actually done: one Non-Randomized and one Randomized. (page 5) 1. Non-randomized Experiment (conducted by NFIP): Subjects were children in grades 1, 2, and 3 from school districts at high risk for polio. All grade 2 children whose parents consented were given the vaccine. Children in grades 1 and 3 were controls.  The Treatment and Control group were different in 3 basic ways how do you think the differences will bias the results? 1.Kids were in different grades. Treatment was just Grade 2, Control was Grades 1 and 3. How could this bias the results? 2.The Experiment wasn’t double-blind.  3.The Treatment group was consent only, while the Control group included everyone—kids whose parents would have consented if asked and kids whose parents would not have. Important fact: Parents who consent tend to be wealthier than parents who don’t consent.

16. Page 6 2. Randomized Experiment –(Public Health Study) Treatment and control were randomly chosen from the SAME population—children whose parents had consented to the vaccine. The goal is to have the treatment and control group as similar as possible on all relevant variables-income, general health, social habits, etc. Impersonal chance works better than human judgment. The control group was given a placebo. The experiment was double-blind. Neither children nor the researchers who evaluated them knew who got the vaccine and who got the placebo. This was an IDEAL experiment.

17. Page 6 cont. Results of both the randomized and non-randomized studies (Rate is per 100,000) • The RCDB design shows the vaccine to be much more effective. (Rate is cut from 71 to 28 compared to from 54 to 25.) Do you think the results are biased because the sample sizes are unequal? • Clicker Question • Neither control group got the vaccine, what accounts for the difference in rates (71 vs 54)? • a) The Control group in the Randomized Double-Blind design got a placebo so they thought they were protected and probably engaged in more risky behavior. • b) The Control group in the Randomized Double-Blind design was consent onlyso they were likely to be higher income on the average and therefore higher risk for polio. • c) The Control group in the Non-Randomized study weren’t even told about the vaccine so they didn’t stress out about it, making them less likely to contract it.

18. Page 7 The non-randomized polio study was unusual because it showed a bias against the vaccine.Medical Studies with no controls, historical controls and other non-randomized controls often show bias toward the treatment. Example 2: Portacaval Shunt—surgery to redirect the flow of blood in patients who are at risk for hemorrhaging due to cirrhosis of the liver. 51 studies were done to see if the operation helped. Why do you think the no-controls study were most enthusiastic about the surgery(24/32 = 75%)? The non-randomized studies were much more enthusiastic than the randomized studies. (10/15 compared to 0/4) What are some possible explanations for that?

20. Compare 3 year survival rates in randomized vs. non-randomized studies. (page 7 cont.) Results from surgery are the same. The difference lies in the results from no surgery. The non-randomized controls are probably sicker to begin. Doctors often exclude the sickest patients from surgery. Could you design these studies to include placebos and be double-blind? If so, how much of a difference do you think it would make?

21. Page 8 Example 3: Arthroscopic Knee Surgery A popular operation for arthritis of the knee was recently found to be no better than a placebo procedure in which patients were sedated while surgeons pretended to operate. 180 patients were randomly assigned to have the operation or to have placebo surgery in which surgeons simply made cuts in the knee so the patients wouldn't know if they had the surgery. Most patients in both treatment and control said their knee pain had improved, but neither group improved on any objective measures. See http://www.nytimes.com/2002/07/11/us/arthritis-surgery-in-ailing-knees-is-cited-as-sham.html And http://www.nytimes.com/2008/09/11/health/research/11knee.html

22. Page 8 cont. Example 4: Hormone Replacement Therapy At least 6 million women in this country had been taking HRT for years on the basis of numerous studies which showed that women on the hormone therapy had a lower incidence of heart disease, stroke, colon cancer and bone fracture. But none of the studies used randomized controls.  Then a study sponsored by the National Institutes of Health, enrolled 16,608 healthy women from ages 50 to 79 and randomly assigned them to take hormone replacement therapy or a placebo. Much to everyone's surprise, the study was prematurely stopped in the summer of 2002 because it showed that while hormone treatment did lower the risk of hip fractures, it actually increased the risk of heart attack, stroke and breast cancer. What are some possible reasons why the randomized controlled study showed an increase in heart disease and stroke while the non-randomized studies showed a decrease?

23. Page 9 Summary: A well-designed study has treatment and control groups that are as alike as possible. • Randomized Controls work best to make treatment and control alike— In studies with enough subjects random differences average out. In smaller studies, researcher should first match subjects into pairs, and then use an objective random procedure to split each pair into treatment and control. As long as the final assignment into treatment and control is random, it’s a randomized experiment. • Non-Randomized Controls are always worse: Different Types of Non-randomized Controls: • Historical • Experimenter tries to match treatment and control to make the 2 groups as alike as possible. • Experiment chooses the treatment and control groups in the best interests of the subjects. • Experimenter decides to let the subjects choose whether to be in treatment or control group. • Placebos eliminate the effect that the IDEA of the treatment may have on the response. The idea of treatment is more or less important depending on the nature of the study, i.e. placebos are more necessary in testing anti-depressives than in testing polio vaccines. • Double-Blind – Eliminates both subject and evaluator bias.