Design of Experiments

Design of Experiments FPP Chapters 1 - 2

Main topics • Designed experiments • Comparison • Randomization • Observational studies • “control” • Compare and contrast • Pitfalls to avoid

The five steps of statistical analyses • Form the question • Collect data • Model the observed data • Check the model for reasonableness • Make and present conclusions

Experiments • Many questions correspond to some cause and effect relationship • Does smoking cause cancer • Does crop rotation A produce more corn yield than regularly used crop rotation • How to collect data that will answer causal questions • Experimental design!

Causal Studies • First some vocabulary • Treatments: variables that are potentially manipulable • Explanatory variable: variable describes levels of treatment • Response variable: variable whose values are set after treatment assignment • Background variable: variables whose values are set before treatment assignment • The term concomitant is used in the Reiter exposition • Subject/unit: Individual/object to which treatments are assigned • Control: A treatment without the active ingredient • Placebo: A treatment that outwardly resembles the active treatment but with out an active ingredient • Replication: Administering each treatment level to more than one unit

More terms • Terms that will be discussed in more detail later are: • Confounding/lurking variable • Double-blind randomized trial

Philosophical approach to causality • A perfect, and impossible, casual study: • Obtain an exact copy of each unit • Expose one copy to treatment a • Expose the other copy to treatment b • At the end of the study, take the difference in the responses for each copy • Differences are the causal effect of treatment a relative to treatment b for each unit • Average causal effect: average response to treatment a minus average response to treatment b

Fundamental problem • WE CAN NEVER DIRECTLY OBSERVE CAUSAL EFFECTS. • Why? • Statistics provides a way to overcome the fundamental problem • Creat two groups of units, so that one group recieves treatment a and the other receives treatment b • Estimate the average causal effect from the observed responses in each group (average response in group a) – (average response in group b)

Potential problem • When groups not well constructed, this measures effects of variables other than treatments • This is often called confounding • A solution is to design groups so that the background variables are as similar as possible in the groups • This type of grouping can be had if units are assigned RANDOMLY to each group • These types of experiments are called randomized experiments

Example 1 • To compare 3 new test fertilizers, a farmer applies them to several corn fields. Each field has 3 plots and the 3 fertilizers are randomly assigned, one to each plot within each field. Harvested corn yield is compared for the 3 fertilizers • Subject: Each corn plot • Treatments: Three test fertilizers • Explanatory variable: Fertilizer type • Response variable: Corn yield • Control: None

Example 2 • National supported work demonstration • 1970s U.S. social experiment of effects of job training for low income workers. • Treatment: Attend job training • Units/subjects: 1602 low-income applicants • Explanatory variable: Attend job training or not • Response variable: Salary one year after program • Units assigned randomly to attend or not to attend the job training

Example 3 • Infant health development program • 1980s U.S. study of effects of intensive child care intervention for low birth weight babies. • Treatment: Attend child care program • Units/subjects: 985 low birth weight babies • Response: Score on vocabulary test • Babies assigned randomly to attend or not to attned the child care program.

Principles of good experimental design • Control or comparison • What: comparing active treatment with control group or compare two or more treatments • Why: to neutralize the effect of lurking variables and measure treatment differences • Randomization • What: using random device to assign subjects to treatments • Why: attempt to minimize bias and invoke assumptions for statistical inference • Replication • What: Applying each treatment to more than one subject in each treatment group • Why: to measure and reduce chance variation in the results by increasing the number of subjects in each group

Confounding /lurking variables • Lurking variable • A variable that affects the relationship between the response variable and the explanatory variable but is not included among the variables studied • Confounding • A condition where the effects of two different variables on the response variable cannot be distinguished from each other

Risky design (that is often used) • Measure response variable before and after administrating the treatment • Then claim causality when there are differences in before and after responses • Ex: Foreign language teachers attend a summer of training program to increase language skills. They take a language test before the program starts and a similar test after the program is completed

Risky design cont. • For sake of argument suppose average test score increases • Is this due to the program? Why? • How could this experiment be improved? • Randomly assign some teachers to not attend • This is incorporating a control

Observational studies • Is a randomized experiment available for all studies? • Sometimes randomizing a human subject to one of the treatment groups would be unethical • For example trying to establish the causality of smoking and lung cancer • When randomizing subjects to treatment groups isn’t possible, typically we use observational studies • These are usually based on existing records from databases with units in both treatment groups • Ex: collect data on smokers and non-smokers from hospital records to compare lung cancer incidence rates

Treatment groups in observational studies • We should not simply compare the two groups in the databases; they are likely to differ in background variables • Therefore, construct groups with similar background characteristics • Ex: Smoking and lung cancer • For each smoker, find a nonsmoker with the same race, age, sex, job type etc. • When there are many variables to match, statisticians use advanced statistical matching methods.

Effect of matching in observational studies • Matching mostly eliminates groups’ differences for the variables that were used in the matching • This mitigates these variables effects on comparisons of groups’ sample mean responses • However!!!! (a massively important however…) • There may be unobserved background variables that differ in the groups (Lurking variables and confounding) • Hence in observational studies we have no assurance that the estimate of the average causal effect is free from the effects of unobserved variables that differ in the two groups

Observational study vs designed experiment Media often makes observational studies look like experiments

Think about it • Does wearing a bike helmet prevent injuries? • design an experiment to answer this question • design an observational study to answer this question • Do people that wear bike helmets get injured less? • How is this question different from the previous one? • How will the designed experiment change? • How will the observational study change?

Causal study warnings • Randomized experiments • Hidden bias • Double blind • Placebo effects • Noncompliance • Order effects • Observational studies • Confounding from unobserved background variables • Different background variables in treatment groups • Both randomized experiments and observational studies • Study conditions may not be realistic • Results may no generalize

Important aspects of causal studies: Comment 1 • In many randomized experiments, the units are not selected at random from a population (e.g., volunteers) • Causal conclusions are valid for the units in the randomized experiment • An issue is whether or not such results can be generalized to other units

Important aspects of causal studies: Comment 2 • Many randomized experiments are not simple two-treatment randomizations • They may involve randomizing within groups of units (e.g., randomly assign treatments within male and female groups.) • They may involve randomizing more than one type of treatment (e.g., some cancer patients get chemotherapy, some get radiation, some get nothing, some get both.) • Methods for analyzing such studies are beyond the scope of this course

Inference primer • Purpose of Experiment: • To determine whether treatments affect the response • Observed effect: • The difference between what we see in the data and what we expectto see in the data. • Statistically significant: • An observed effect that is too large to attribute plausibility to chance variation • If the differences between the responses for two treatments is statistically significant, then the treatments affect the response

Inference primer cont. • Polio example • The polio rate of those receiving the vaccine was 0.028% compared to 0.071% for those receiving the placebo • In statistical tests that compare two treatments we generally “expect” to see no difference. • Thus the observed effect here would be (0.071 – 0.028) – 0 = 0.047 • Is the observed effect of a 0.047% increase in polio rate small enough to be chance variation or large enough to attribute to the vaccine? • We answer this question using probability later in the course

Design of Experiments