“Dirty Pool” Revisited: When Less is More

1. “Dirty Pool” Revisited: When Less is More Robert S. Erikson, Pablo M. Pinto, Kelly T. Rader Department of Political Science Download paper at: http://ssrn.com/abstract=1450061

2. Do Democracies Trade more with other Democracies? How the literature approaches this question: • Convention: • Measure units as dyad-year • e.g.: Sierra Leone-Ecuador 1997 • Measure democracy as democracy score for the least democratic dyad member • Relate democracy to trade levels while controlling for other stuff • Main issues: • Fixed effects? • Lagged dependent variable on RHS? • Dyads as unit not an issue.

3. Do Democracies Trade more with other Democracies? • Methodological issues discussed in 2001 symposium in IO • 90,000+ dyad-years, but only up to 115 countries over 30 years • We present a cautionary tale about using dyads as units of analysis • Large data set with a massive N can create overconfidence in hypothesis testing

4. Symposium on Research Design and Methods in IR: • Editors' Introduction (pp. 439-440) Peter Gourevitch, David A. Lake • Dirty Pool (pp. 441-468) Donald P. Green, Soo Yeon Kim, David H. Yoon • Clear and Clean: The Fixed Effects of the Liberal Peace (pp. 469-485) John R. Oneal, Bruce Russett • Throwing out the Baby with the Bath Water: A Comment on Green, Kim, and Yoon (pp. 487-495) Nathaniel Beck, Jonathan N. Katz • Proper Nouns and Methodological Propriety: Pooling Dyads in International Relations Data (pp. 497-507) Gary King

5. Our contribution Randomization tests on dyads to infer the correct p-values Switch to nation-years as units of analysis: proportion of trade with democracies as a function of regime Difference-in-difference analysis of change in trading partners after a regime shock Cross-sectional analysis of trade by regime type

6. GKY (current dollars)

7. GKY (current dollars)

8. GKY sample (constant dollars)

9. GKY sample (constant dollars)

10. Expanded Trade Data

11. Expanded Trade Data

12. Randomization Tests! • Simulate the state of the world if the null hypothesis were true • Scramble the key independent variable to break the relationship between it and the outcome • Estimate coefficients and significance tests using preferred model, knowing the “answer” is zero • Repeat at least 1000 times • See if the actual estimated effect is rare compared to distribution of spurious effects • Sample estimate of Fisher’s exact test (1935)

13. Randomization Tests! Drop the country labels on the floor and scramble them. Pick them up randomly and reinsert. Albania might become “Bolivia” for example, in all dyads involving true Albania. Rescramble 1000 times. With random country labels and the equation rerun, the data present a distribution around the null outcome (presumably zero, but with complications). T Where in the distribution of observations does the observed value from the actual equation fall? That position in the cumulative distribution yields the p-value with the randomization test.

14. Advantages of Randomization Tests • Unlike parametric tests… • Don’t rely on distributional assumptions about ij • Don’t rely on distributional assumptions about the test statistic • Standard errors and significance tests determined empirically • Especially useful when theoretical standard errors are hard to derive • Like the democratic trade models! • But, do assume exchangeability of errors

15. Figure 1: Randomization tests results

16. Figure 1 (cont): Randomization tests results

17. Figure 1 (cont): Randomization tests results

18. Figure 1 (cont): Randomization tests results

19. Country-year Models

20. Figure 2: Dem. trade and regime (residuals) Y = .03 + .38 X + ε std. err = .044 p = .000 N= 6459 Adj. r2=0.011 Residual proportion of trade with other democracies by residual regime Data: nation-year; controls: nation and year dummies.

21. Figure 3: Democratic shocks Democratic trade at tn relative to democratic shock at t0

22. Figure 4: Democratic shocks Residualized democratic trade at tn relative to democratic shock at t0

23. Figure 5: Anti-democratic shocks Democratic trade at tn relative to anti-democratic shock at t0

24. Figure 6: Anti-democratic shock Residualized democratic trade at tn relative to anti-democratic shock at t0

25. Figure 7: Effect of regime shocks 2.6 = mean change in democratic trade per year x 6 years Change in democratic trade as a function of democratic shocks

26. Figure 8: Country analysis Y = 1.10 X + ε st. err = .185 p = .000 N= 154 adj. r2=0.183 Residual democratic trade over mean regime score (year adjusted) Data: countries

27. Conclusions • Dyads are inappropriate units of analysis for democratic trade hypothesis: • Randomization tests show hypothesis testing in traditional regression results are too optimistic (p-values up to 41 trillion times larger!) • Randomization tests are useful for testing claims, especially when the appropriate parametric test is difficult to figure out • Move to nation-year as unit of analysis • We find no regime effect • Less is more: • Event and difference in differences analyses of shocks seem to confirm democratic trade hypothesis