Quasi-Experiments: Good Enough for Social Science

1. Political Science 30 Political Inquiry Quasi-Experiments: Good Enough for Social Science

2. Quasi-Experiments: Good Enough for Social Science • The philosophy behind “quasi-” and “natural experiments” • A classic example: Cracking down on Connecticut speeders • Strengths and Weaknesses

3. The Philosophy of Quasi-Experimentation • In a randomized experiment, scientists assign cases to control and treatment groups, and apply the treatment. • In a“natural experiment,” Mother Nature assigns cases to control and treatment groups in some nearly random manner. • In a“quasi-experiment,” scientists merely observe two or more groups of cases that have been treated differently.

4. The Philosophy of Quasi-Experimentation • “Perhaps its fundamental credo is that lack of control and lack of randomization are damaging to inferences of cause and effect only to the extent that a systematic consideration of alternative explanations reveals some that are plausible.” – Campbell and Ross, 1968, p. 34. • Translation: If no one can think of a confound, don’t worry, be happy.

5. The Philosophy of Quasi-Experimentation • Step #1. Think hard about whether or not your groups of cases differ in their values of some confounding variable. • Step #2. If there is a difference, try to make another comparison of two or more groups that are similar in every important variable other than the key IV. • Step #3. Measure your DV and make time-series or cross-section comparisons.

6. A Classic Example • In 1955, Connecticut had 324 traffic fatalities on its highways. • Gov. Ribicoff cracked down on speeders by suspending their licenses in 1956. This change in policy is the “treatment.” • In 1956, only 284 Connecticut motorists died, and Ribicoff declared victory.

7. A Classic Example: Time-Series • Campbell and Ross compare CT before crackdown vs. CT after crackdown. • Comparison plagued by differences between these groups (pp. 38-39): • History • Maturation • Testing • Instrumentation, Instability and Regression also at work (pp. 39-40)

8. A Classic Example: Possible Confounds • History: Besides the treatment, other events take place over time. • Maturation: Steady, long-term trends are also at work. • Testing: The very act of measuring cases in a pre-test changes the cases.

9. A Classic Example: Cross-Section • Compare trends in Connecticut with trends in adjacent, similar states. • This eliminates threats to inference brought by many confounds. • Gives us a way to judge instrumentation, instability and regression effects.

10. Strengths and Weaknesses • Internal validity judges how well a research design has tested a causal relationship, in the cases examined. • Quasi- and natural experiments fare worse on this criterion than lab experiments, because assignment is not truly random. • Still, these are better than observational studies that compare very different groups.

11. Strengths and Weaknesses • External validity judges how confident we can be that a causal relationship identified in our cases can be generalized to the outside world. • Quasi- and natural experiments beat lab experiments on this count, because they take place in the real world. • The limitation is that not every situation presents an opportunity for an experiment.