Principles of Experimentation

1. Principles of Experimentation ST711 Fall 2014

2. Learning objectives • Differentiate an observational study from a comparative experiment • Identify treatments and blocks • Differentiate experimental units (EUs) from observational units • List four principles of experimentation and their impact on statistical analysis

3. Correlation does not imply causation • Hilarious Examples • Observational studies • Observe and record environment without interrupting it • Find relationships but limited in explaining why they’re there • Lurking variables • Unobserved/unrecorded events that may be behind relationship • Firefighter example • Confounded variables • Recorded variables that could also explain a relationship

4. Can smoking cause cancer?Study 1 • 50 smokers/nonsmokers each (age 55-65) • Compare rates of cancer in two groups • Rate of cancer for smokers was higher than for nonsmokers • Potential lurking/confounding variables • Smokers may be genetically predisposed to cancer • Smokers made other poor health choices (diet and exercise) • Non-smokers may be more likely to visit doctor • When they started smoking and how much could affect rate

5. Does smoking cause cancer?Study 2 • 50 similar mice • Randomly assign 25 mice to be subjected to same daily amount of cigarette smoke for fixed time • Otherwise given nearly identical treatment (exercise, diet, etc.) • Advantage: each mouse has same chance of being a smoker (small chance for confounding/lurking variable) • Disadvantage: we care about humans smoking, not mice!

6. Sources of variation • Source of variation: anything that causes two responses to be different from each other • Identify potential sources of variation prior to data collection • Goal: Determine whether identified, controllable sources explain the majority of variation • Need to reduce impact of unknown or uninteresting sources to increase sensitivity of tests • Control of major sources of variation allows us to make causal inferences

7. Elements of comparative experiments • Comparative Experiment: a study in which an experimenter • Assigns a delineated set of conditions to a group of subjects • Compares conditions’ effects on a response variable • Treatment: a possible condition that may be assigned • Different treatments are always considered to be potential major sources of variation • Other sources lead to experimental variability • Experimental unit (EU): whatever we assign a treatment to • Why isn’t Study 1 a comparative experiment? • Identify the treatments and EU’s in Study 2.

8. Does smoking cause cancer?Study 3.1 • Conduct Study 3 this way: • Break up 50 mice into 10 chambers with 5 mice in each • Randomly assign 5 of 10 chambers to receive smoke • Mice are put in the same chamber each day • Claim: the individual mice are not the EU’s • What are the EU’s?

9. Observation units and pseudo-replication • Observational unit (OU): “part” of EU on which response is measured • Study 3: each mouse is both an EU and OU • Study 3.1: mouse is OU, chamber is EU • Different EU’s implies different treatments or different re-creations of the same treatment • Why important: two OU’s from the same EU are correlated since they received the same treatment re-creation • Pseudo-replication: Treating OU’s as EU’s will underestimate experimental variability (increase Type I error)

10. Design and analysis of experiments • Design implies a certain analysis • Goals of analysis should impact design • Design questions: • Sample size: how many EU’s and OU’s to use? • How to make inference as broad as possible? • How to handle inherent variability among EU’s? • How to allocate treatments to EU’s? • How do I collect/record data? • Potential analysis goals: • Different treatments lead to change in response(s) • Conditions lead to maximum or minimum response • Build statistical model

11. Key principles of design • Representativeness • Replication of treatments • Error control (blocking and covariates) • Randomization

12. representativeness • Applies to all studies, not just experiments • Population of EU’s should be representative of the population we want to make inferences on • Study 3: extend results to humans? • Homogeneous EU’s will reduce experimental variability at the cost of representativeness • Design techniques exist that broaden pool of EU’s

13. Replication • Replication: a re-creation or copy of the same treatment applied to a different EU • In order to be certain of a treatment’s effect, it must be observed repeatedly. • Increasing replication leads to • Better estimate of experimental variability • Increased precision of treatment comparisons • Assurance against aberrant results due to random chance • Increase in cost • Ideal sample sizes will minimize cost without sacrificing benefits of increased replication • How many times do we replicate treatments in Study 3.1?

14. Error control – Blocking • Techniques to reduce experimental variability without sacrificing representativeness • Block: group of EU’s that are more similar than other EU’s • Idea: make treatment comparisons within each block and pool results together • Confounding: If we assign the same treatment to every EU in a block we can’t separate block and treatment effects

15. Error control - Blocking • Example: Three different crop management practices are to be compared on a vineyard. • Have 3 vines, each vine is broken up into 3 sections Each section of a vine is EU Sections of one vine more similar than those of another Assign each treatment to a section of every vine Comparing treatments within each vine “cancels out” the characteristic of that vine Vine 1 Vine 2 Vine 3

16. Error control - Blocking

17. Error control - Blocking

18. Error control - Blocking • Common blocking factors • Time • Proximity (closer EU’s more similar) • Physical characteristics (age, sex, etc.) • Useful to generalize experiments (not afraid of different EU’s) • If blocks are not a major source of variation we could increase experimental variability • Cannot assign block characteristics, otherwise they would be treatments

19. Error control - covariates • Covariates: additional explanatory variables (categorical or continuous) that • Are measured just prior to or during treatment assignment • Could significantly reduce experimental variability • Requires assumptions about how the covariates and response are related • Could even be an interaction between covariates and treatment • Example: interested in multiple nutrition and workout regimes to help people lose weight • Potential weight loss likely restricted by starting weight • Measure weight prior to treatment

20. Error control - covariates

21. Error control - covariates

22. Error control - covariates

23. randomization • Given a fixed number of replications for each treatment, determine allowable assignments of treatments to EU’s (depends on experiment) • A design has been properly randomized when all allowable assignments are equally likely to be used • Example 1: 3 treatments, 9 EU’s, 3 replications per treatment. There are possible randomizations

24. randomization • Example 2: Let’s say we group the EU’s into 3 blocks each with 3 EU’s. Each treatment appears in each block once (still 3 replications each). • In order to maintain that each treatment appears in each block, we have to randomize within each block • There are 3!=3*2*1=6 allowable randomizations in each block so there are a total of allowable randomizations • Block designs will always require separate randomizations

25. randomization • Haphazard assignment does not equal proper randomization! • Avoids bias and accusations of it • Reduces chance of observing a treatment effect due to random chance alone (confounding) • Justifies statistical models and analysis in some cases (randomization tests) • Foundation for causal inference

26. Ethical considerations • Obvious ethical considerations that need to be considered (e.g. human testing) • Clever experimental designs can maximize information using minimal resources and reduce impact on environment and animals • Ethical considerations/constraints can often lead to interesting design problems • Crossover designs • Clinical trials