Experimental Design making causal inferences

1. Experimental Design making causal inferences

2. Causal and Effect • The IV precedes the DV in time • The IV and DV are correlated • There are no plausible additional variables that could reasonably explain the correlation

3. The Gold Standard • What are the differences between these types of studies? • True experiments • Quasi experiments • Observational studies • Surveys • Why is it so hard to do in education?

4. The Gold Standard • True experiments versus observational studies, or quasi-experiments • The Randomized Clinical Trial model • Why is it so hard to do in education?

5. The Gold Standard • True experiments • Random assignment to treatment and control conditions • The Gold Standard for causal inferences • The treatment is under the control of the researcher

6. The Gold Standard • Quasi experiments • The treatment is under the control of the researcher • As many of the characteristics of a true experiment as possible • Intact groups, random assignment is not possible

7. The Gold Standard • Observational studies • Intact groups, not randomly assigned • If there is a treatment, it may not be under the control of the researcher • Correlations between variables, not causality

8. The Gold Standard • Surveys • The focus is on estimation of parameters, not the effects of a treatment • Snap shot in time • Self-report, subjective perceptions of respondents, not objective or external • Not designed to provide hard outcomes

9. The Essential Characteristics • Random Selection • Random Assignment • Manipulation of the IV • Control Condition

10. Additional Factors to Consider • Placebo or Comparison Group(s) •   Multiple Measurements Over Time •  Control Over Confounding Variables 

11. Controlling Confounds • The experimental environment • Admissibility criteria • Blocking 

12. Blocking • Creates homogeneous subsets • Builds potential confounds into the design •  Reduces error term • Makes a more sensitive and therefore powerful experiment

13. Additional Design Features • Measurement of Compliance or Implementation Effects • Blinding • Measurement of Outcomes • Integrity of Data Collection, Entry, and Reporting Procedures

14. Ethical Considerations • Informed consent • Privacy • Do no harm • Denying services • Inducements • Dual roles

15. 2003 #4

16. Rubric for Part A • Identify a plausible example of a problem • “Because a deadline has been moved back...” • Relate the identified problem to the change in stress level • “...the stress levels of those working in the department have been lowered...” • State that the problem effects can not be distinguished from the treatment effects • “...which could be mistakenly attributed to the treatment.”

17. Rubric for Part A • Give a reason for the necessity of random assignment. • State that randomization is relied upon to create comparable groups. • State that randomization helps reduce the influence of potential confounding variables.

18. Rubric for Part A • “Without random assignment of volunteers to the two programs, it is possible that the two treatment groups could differ in some way that affects the outcome of the experiment. Randomization “evens out” the possible effects of potentially confounding variables.”

19. Rubric for Part B • Indicate that a control group does provide additional information • Explain that the control group allows the company to determine if either or both treatments are effective in reducing stress • Explain that the control group provides a baseline for comparison, an indication of what might have happened anyway, even without the treatment

20. Rubric for Part B • “Without the control group, the company could compare the two treatments, but would not be able to say whether the observed reduction in stress was attributable to participation in the programs. For example, a change in the work environment during this period might have reduced the stress level of all employees. The addition of a control group would enable the company to assess the magnitude of the mean reduction attributable to each treatment, as opposed to just determining if the two programs differ.”

21. Rubric for Part C • Indicate that one cannot generalize, and give a plausible reason, such as... • The participants were volunteers and volunteers my not be representative of the population • The participants were not randomly selected from the population

22. Rubric for Part C • “No it is not, for this experiment we took volunteers but the problem with it is that the people who volunteered are very likely the ones who needed the stress reduction the most...Therefore, it is not reasonable to generalize because most likely the people who volunteered are not representative of the population.”

23. Common Student Errors • Did not understand the difference between random allocation of subjects and random sampling. • Often used the word "confounding" in part (a), but did not explain how the treatment results were mixed up with some other variable.

24. Common Student Errors • Seemed to think that a larger sample size would fix any problem in the experiment, rather than recognizing that the major problem of the experiment was that there was no random sampling of employees. • Incorrectly stated that random allocation "eliminates" bias.

25. 2001 #4

26. Rubric for Part A • Blocking Scheme A is preferable • Creates homogeneous blocks with respect to forest exposure • Plots will have similar forest exposure

27. Rubric for Part B • Randomization within blocks should reduce bias due to the influence of confounding variables • (fertility of soil, moisture, etc.) • on the productivity of the trees. 2001 FR #4 Rubric

28. Extension Questions • How would you randomize trees within blocks? • What other confounding variables might impact the results?

29. Extension Questions • How does this concept of blocking apply to educational evaluation? • Students • Classrooms • Schools