Research Design

1. Research Design • “The program that guides the investigator in the process of collecting, analyzing, and interpreting observations. It is a logical model of proof that allows the researcher to draw inferences concerning causal relations among the variables under investigation” (Nachmias and Nachmias).

2. Research Design and Causality • Relationships between variables: Two variables are related to one another (i.e. are correlated) if one or more values of one variable tend to be associated with one or more values of the other variable. • Causal relationship: A relationship in which one variable directly causes/explains the other variable.

3. 3 Criteria for Establishing Causality • X is correlated with Y (Covariation) • X precedes Y in time (Time Order) • The observed relationship between X and Y is not spurious (Causation) • Spurious Relationship = An observed relationship between X and Y is said to be spurious (or partly spurious) if there exists a third variable Z, which is both a cause of Y AND is correlated with X.

4. Example of Spuriousness

5. Choice in Design • Experimental can be the most reliable and believable • But Researchers must balance what is possible to accomplish with what is ideal • Many phenomena that interest political scientists lie outside of what can be measured experimentally • Experimental Designs not always ethical when dealing with real people.

6. Historical Development of Ethical Standards in Observational Research 1974 - National Research Act National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research identify the basic ethical principles that should underlie the conduct of biomedical and behavioral research involving human subjects and to develop guidelines which should be followed to assure that such research is conducted in accordance with those principles Belmont Report (1979)

7. Milgram’s Obedience Experiments • Experiment advertised as test of memorization training through electric shock • Those being tested were actually actors • Lab assistants hired to administer “shocks” were the true subjects • 65% of lab assistants were willing to administer shocks of up to 450 volts

8. Experimental Designs • Select a sample • Randomly assign subjects into 2 or more groups. • Observe (measure) DV for all groups (if design includes pretest) • Introduce the stimulus (IV) • Observe (measure) DV for each group • If the change in the value of the dependent variable varies significantly across groups, then we conclude that X  Y

9. Simple Experimental Designs 2-Group Pretest - Posttest Design (Classical or “Simple” Experiment) • R Mexp1 X Mexp2 • R Mcontrol1 Mcontrol2

10. Simple Experimental Designs 2-Group Pretest - Posttest Design (Classical or “Simple” Experiment) • If two different treatments (and no pure control group): • R M1A XA M2A • R M1B XB M2B

11. Experimental Designs • Key distinguishing feature of the experimental design: • Randomization (random assignment of subjects to groups)

12. Example: Negative Ads • Hypothesis: Negative Campaign Ads Lead to Lower Voter Turnout. • Sample: University students • Dependent Variable: Response to likelihood to vote. • Independent Variable: 30 second fake negative campaign ad exposure.

13. Research Design (R&M) • 2-group Pretest – Posttest • Generally: • R M1eX M2e • R M1c M2c • R M1e (Will Vote?)XA (Neg. Ad) M2e (Will Vote?) • R M1c (Will Vote?) (Commercial) M2c (Will Vote?)

14. Experiments and Causality • Correlation?

15. Experiments and Causality • Correlation? • Comparison of two or more groups (on dependent variable) experiencing different levels of exposure to the causal (explanatory) variable (X). This establishes correlation.

16. Experiments and Causality • Temporal Precedence?

17. Experiments and Causality • Temporal Precedence? • The introduction of the independent variable (“stimulus”) is manipulated by the researcher to insure temporal precedence.

18. Experiments and Causality • Spuriousness?

19. Experiments and Causality • Spuriousness? • Random assignment insures that rival hypotheses are ruled out, thus eliminating the threat of spuriousness. (How?)

20. Simple Experimental Designs 2-Group Pretest - Posttest Design (Classical or “Simple” Experiment) • R Mexp1 X Mexp2 • R Mcontrol1 Mcontrol2 OR • R M1A XA M2A • R M1B XB M2B

21. Simple Experimental Designs • 2-Group Posttest Only Design • R X M1exp • R M1control OR • R XA M1A • R XB M1B • Why is this acceptable?

22. Other Types of Experimental Designs • Multigroup designs – more than two groups • Multiple Group Pretest - Posttest Design • Multiple Group Posttest Only Design

23. Other Types of Experimental Designs • Time series design • Multiple observations over time

24. Field Experiments • Experiments that occur outside the artificial setting of laboratory (occur in the “real world”) • Example: Malawi – Nambia Example from Class • 1. Cannot have random assignment • 2. Can have manipulation of independent variable.

25. Evaluating Research Designs:Internal Validity Internal Validity - the degree to which we can be sure that the independent variable caused the dependent variable within the current sample

26. Evaluating Research Designs:Internal Validity Experimental designs - randomization of subjects/units across values of the independent variable greatly reduce (eliminate?) the potential for spuriousness to threaten internal validity

27. Evaluating Research Designs:Internal Validity Specific factors threatening internal validity (experimental or nonexperemental) include: History Maturation Experimental mortality Instrumentation Testing Selection Bias

28. Evaluating Research Designs:External Validity External Validity - the degree to which the results of the analysis can be generalized beyond the current sample/study. Can be maximized by: Using subjects (units) that are representative of the population to which one’s theory applies Using a “laboratory” that is as close to “real life” conditions as possible Field experiments

29. PS 372 – Research Design Cont., with maybe a splash of Sampling

30. Quiz – Don’t forget your name! Explain the following problems for research designs: History Maturation Experimental mortality Instrumentation Testing Selection Bias

31. Nonexperimental Designs • Deviate in some important way(s) from true experimental design • All nonexperimental designs lack random assignment of subjects to groups • But some nonexperimental designs may lack other features too

32. Matched Pair Designs (precision matching) Effort to overcome nonrandom assignment of subjects to treatment groups (i.e. to equalize the comparison groups in a research design) 1. Form “matched pairs” – pairs of subjects that are matched based on variable(s) known to affect the DV 2. Assign one member of each pair to treatment and control groups

33. Nonexperimental Designs • Cross-Sectional Designs • No manipulation of IV by researcher • Observations for IV and DV recorded at the same time

34. Example: Wine and Health • Hypothesis: Drinking wine causes individuals to be healthier (esp. heart) • Existing studies: compared the health of wine drinkers to the health of those who do not drink wine: Research design XA (Wine drinkers) M1A (Health) XB (Non-drinkers) M1B (Health)

35. Spurious Results?

36. Controlling for Affluence Research design: XA (Affluent Wine drinkers) M1A (Health) XB (Affluent Non-drinkers) M1B (Health) XC (Poor Wine drinkers) M1C (Health) XD (Poor Non-drinkers) M1D (Health)

37. Causality?

38. Another (Possible?) Example

39. Stack & Gundlach • Hypothesis: There is a positive relationship between exposure to country music and suicide rates • Research design: XA (no country music) MA (suicide rate) XB (1 station) MB (suicide rate) XC (2 stations) MC (suicide rate) XD (3 stations) MD (suicide rate) Xi ( etc.) Mi (suicide rate)

40. Stack & Gundlach • Findings: 51% of the variation in urban white suicide rates can be explained by variation in airtime devoted to country music • Is this result spurious?

41. Time as a guide to cause. • How do you determine whether listening to country music causes suicide rates, suicide rates causes listening to country music, or both result from a common cause? • In the absence of any prior assumptions or knowledge, the covariational data alone are insufficient to answer this question. • Solution?

42. Nonexperimental Designs Time Series Design – repeated observations for X and Y for a single unit Panel Time Series Design – repeated observations for X and Y for a group

43. Nonexperimental Designs Case Study X O1 Inference made by examining one case.