Experimental Design making causal inferences

1. Experimental Design making causal inferences Richard Lambert, Ph.D.

2. The Gold Standard • What are the differences between these types of studies? • True experiments • Observational studies • Surveys

3. Observational Studies • Intact groups, not randomly assigned • If there is a treatment, it may not be under the control of the researcher • The goal is often just to look at relationships between variables • Correlation not causality

4. Surveys • Population is defined • Sample is selected from the population so that is can be representative of the population – SRS, Stratified RS • The focus is on estimation of parameters, not the effects of a treatment

5. Surveys • Compare and contrast SRS and Stratified RS • SRS must give everyone in the population an equal chance of selection AND give every possible sample of size n an equal probability of selection • For Stratified RS, sampling is done within homogeneous strata

6. Surveys • For Stratified RS, we divide the population into homogeneous subgroups according to some variable we expect to be related to the survey questions • We then sample within strata • Compare and contrast Stratified SRS and blocking

7. True Experiments • The Gold Standard for Causal Inference • Random assignment to treatment and control conditions • A treatment is imposed on the subjects, the treatment is under the control of the researcher

8. Treatment is Imposed • Presence or Absence of Treatment • Dosage Level Controlled By Researcher • Multiple Types of Treatment Conditions

9. True Experiments • The Gold Standard for Causal Inference • Why is it so hard to do in education? • Is it the only “way of knowing” in educational research?

10. Causal and Effect • The IV precedes the DV in time • The IV and DV are related • There are no plausible additional variables that could reasonably explain the relationship

11. Causal and Effect • When we say the IV and DV are related, think of a dose – response relationship in a drug study. • The amount of treatment exposure is related to a corresponding amount of some effect.

12. The Essential Characteristics • Random Selection • Random Assignment • Treatment is imposed • Control Condition

13. Additional Factors to Consider • Placebo or Comparison Group(s) •   Multiple Measurements Over Time •  Control Over Confounding Variables 

14. Controlling Confounds • The experimental environment • Admissibility criteria • Blocking 

15. Blocking • Creates homogeneous subsets • Builds potential confounds into the design •  Reduces error term • Makes a more sensitive and therefore powerful experiment

16. Blocking • When using blocking, be sure to: • Divide the sample into homogeneous subsets •  Randomly assign subjects to treatments within blocks

17. Released Questions • Work on 1999 Free Response #3 with your partner

18. Released Questions – 1999 #3 3. The dentists in a dental clinic would like to determine if there is a difference between the number of new cavities in people who eat an apple a day and in people who eat less than one apple a week. They are going to conduct a study with 50 people in each group. Fifty clinic patients who report that they routinely eat an apple a day and 50 clinic patients who report that they eat less than one apple a week will be identified. The dentists will examine the patients and their records to determine the number of new cavities the patients have had over the past two years. They will then compare the number of new cavities in the two groups. Why is this an observational study and not an experiment? Explain the concept of confounding in the context of this study. Include an example of a possible confounding variable. If the mean number of new cavities for those who ate an apple a day was statistically smaller than the mean number of new cavities for those who ate less than one apple a week, could one conclude that the lower number of new cavities can be attributed to eating an apple a day? Explain.

19. The Essential Characteristics • Random Selection • Random Assignment • Treatment is imposed • Control Condition

20. Confounding Variables • Have to be related to the DV • Are mixed up with, or inseparable from Group membership • Create non-equivalence of treatment groups • Disrupt your ability to conclude that the treatment and only the treatment caused the outcomes

21. Released Questions • Work on 2003 Free Response #4 with your partner 2003 Free Response #4

22. 2003 #4

23. Rubric for Part A • Identify a plausible example of a problem • “Because a deadline has been moved back...” • Relate the identified problem to the change in stress level • “...the stress levels of those working in the department have been lowered...” • State that the problem effects can not be distinguished from the treatment effects • “...which could be mistakenly attributed to the treatment.”

24. Rubric for Part A • Give a reason for the necessity of random assignment. • State that randomization is relied upon to create comparable groups. • State that randomization helps reduce the influence of potential confounding variables.

25. Rubric for Part A • “Without random assignment of volunteers to the two programs, it is possible that the two treatment groups could differ in some way that affects the outcome of the experiment. Randomization “evens out” the possible effects of potentially confounding variables.”

26. Rubric for Part B • Indicate that a control group does provide additional information • Explain that the control group allows the company to determine if either or both treatments are effective in reducing stress • Explain that the control group provides a baseline for comparison, an indication of what might have happened anyway, even without the treatment

27. Rubric for Part B • “Without the control group, the company could compare the two treatments, but would not be able to say whether the observed reduction in stress was attributable to participation in the programs. For example, a change in the work environment during this period might have reduced the stress level of all employees. The addition of a control group would enable the company to assess the magnitude of the mean reduction attributable to each treatment, as opposed to just determining if the two programs differ.”

28. Rubric for Part C • Indicate that one cannot generalize, and give a plausible reason, such as... • The participants were volunteers and volunteers may not be representative of the population • The participants were not randomly selected from the population

29. Rubric for Part C • “No it is not, for this experiment we took volunteers but the problem with it is that the people who volunteered are very likely the ones who needed the stress reduction the most...Therefore, it is not reasonable to generalize because most likely the people who volunteered are not representative of the population.”

30. Common Student Errors • Did not understand the difference between random allocation of subjects and random sampling. • Often used the word "confounding" in part (a), but did not explain how the treatment results were mixed up with some other variable.

31. Common Student Errors • Seemed to think that a larger sample size would fix any problem in the experiment, rather than recognizing that the major problem of the experiment was that there was no random sampling of employees. • Incorrectly stated that random allocation "eliminates" bias.

32. Released Questions • Work on 2001 Free Response #4 with your partner 2001 Free Response #4

33. 2001 #4

34. Rubric for Part A • Blocking Scheme A is preferable • Creates homogeneous blocks with respect to forest exposure • Plots will have similar forest exposure

35. Rubric for Part B • Randomization within blocks should reduce bias due to the influence of confounding variables • (fertility of soil, moisture, etc.) • on the productivity of the trees. 2001 FR #4 Rubric

36. Extension Questions • How would you randomize trees within blocks? • What other confounding variables might impact the results?