Field Research Methods

1. Field Research Methods Lecture 7 Natural Experiments Edmund Malesky, Ph.D., UCSD

2. Organization of Today’s Lecture • What is a natural experiment? • What purposes do they serve? • What issues can confound using this technique? • Case Analysis • Diamond • Posner • Miguel • Galiani and Schargrodsky • Malesky and Samphantrak

3. What is a Natural Experiment? • A natural experiment is a method of case selection in which the researcher exploits a “treatment” caused by nature to study differences in two or more populations. • The researcher does not perform any manipulation on their own. • Instead, the researcher must demonstrate that their assignment to subject and control groups is random or “as if” random.

4. Why exploit a natural experiment? • Natural Experiments help researchers overcome two critical empirical hurdles: • Reverse Causality/Endogeneity – An exogenous shock caused by nature helps us cut-in to circular causal logics. • Unobserved Heterogeneity (OV Bias) • Variation on Independent Variable: Natural experiments give us clear identification of treatment and control groups.

5. Utility of Natural Experiments • Small-n design: In qualitative/case study research, natural experiments can be very helpful in helping us determine which cases to study. • Cross-sectional: Two identical villages, but one is impacted by a natural disaster. • Interrupted time series: The same village before and after an exogenous shock (requires some historical research). • Large-n design: Large groups impacted by a particular exogenous shock allows for pre- and post-treatment analysis. • Not really discussed, but the best instrumental variables are really plausible natural experiments

6. Types of Natural Experiments • Randomization Device • Researcher exploits randomization of a mechanism designed for another purpose (i.e. lottery, VN draft lottery). • Downstream randomization. • Researcher takes advantage of a research design meant for a previous randomization. • For example, a researcher can perform a follow-up study on participants in a randomized education project that provided college scholarships to minorities to see if education affects political attitudes five year later. • Regression Discontinuity • Exploitation of cut-off in acceptance to a particular program. • Most famously performed by Angrist and Lavy (1999) to estimate the effect of class size on student performance.

7. Types of Natural Experiments • Exploitation of Arguably Random Selection Process • Not a lottery, but the intervention can be shown to have been systematically related to the dependent variable of interest. • Blattman and Annan (2006) look at the psychological effect of child soldiering by studying siblings, one kidnapped by a village raid and the other who was spared. • Galiani and Schargrodsky (2006) exploit the random selection of land tracks by squatters. • Exploitation of a Process Occurring in Nature. • Most often used in an instrumental variables design. • Using rainfall as an instrument for voter turnout or economic growth. • Using wind as an instrument to predict colonization

8. Types of Natural Experiments • Exploitation of a non-natural process that can be claimed is unrelated or orthogonal to dependent variable. • Beber (2008) shows that crisis mediation occurs primarily in summer when government officials have more time. • Malesky and Samphantharak (2008) study the influence of new Cambodian governors on the predictability of corruption.

9. Types of Natural Experiments • Exploitation of a long-run historical process that can plausibly be argued is exogenous to the outcome of interest. • Jha (2008) takes advantage of Islamic commercial trade to identify Indian ports that were exposed to Islamic influence. He finds religious violence is much lower in provinces with such ports. • Banarjee and Iyer (2006), for instance, use British patterns of land administration in colonial India as an exogenous treatment for institutional development. Different British colonial administers implemented either a land-lord based system or a land tenure system in different regions of India. They find that that original treatment leads to vast differences in electoral participation, public goods provision, and ultimately economic growth.

10. Types of Natural Experiments • Exogenous geographic delineation • Colonial border demarcation (Posner 2004; Miguel 2004). • Electoral redistricting in California (Ansolabehere, Snyder, and Stewart 2000) • Exogenous policy alteration in one unit and not in another (Card and Krueger 1994) • Exogenous policy alteration in single unit • Interrupted Time Series • Behavior of individual actors in new environment. • Political behavior of congressman who become Senators

11. Pitfalls of the Natural Experiment • “As if” Randomization • When a researcher employs a natural experiment, they must be confident that the treatment approximates a randomized design. This can be easily violated: • The exogenous treatment is not really all that exogenous (i.e. Galiani and Scargrodsky –perhaps land rights went to those with the best land). • The treatment is correlated with an omitted variable and the dependent variable, leading to a biased assessment of the treatment (Nation-building is associated with other features of the communist government in Tanzania that to less ethnic strife).

12. Spectrum of “As If” Assumption. Most Plausible Least Plausible

13. Pitfalls of the Natural Experiment • The less plausible the “as if” assumption the more a researcher must do to address alternative hypotheses and confounders. • This requires a great deal of extra work. • Thus, a good research design, taking advantage of a natural experiment, will need to begin from very good theoretical starting blocks. • The more hard thinking you do before the field, the less work you will have to when you get home.

14. Pitfalls of the Natural Experiment • Local Average Treatment Effect (LATE) • The treatment actually only effects a portion of the total population making it difficult to generalize to other groups. • People who buy lottery tickets. • Those who eligible for the VN draft (but did not enlist or dodge) • LATE is not necessarily a problem, but you should be aware of the LATE is.

15. Toolbox • Once you have identified an appropriate natural experiment, your choice of tools is unlimited. A natural experiment is more a means of case selection than a research design in and of itself. • Hard data collection • Thick-description case studies • Surveys • Semi-structured interviews • Content Analysis

16. Case 1: Diamond • What is the research question? • What factors allow some societies to dominate and colonize others? • What is his hypothesis? • Geography and natural resources play critical roles in shaping the possibilities for human advancement. • What role does the natural experiment play for him? • The natural experiment allows him to set aside cultural factors as the ultimate cause. All of the groups were from the same original ancestry (the Bismark Archipelago north of New Guinea). It also allows him to explore the influence of confounding variables? • What are his units of analysis? • Civilization s on Polynesian Islands

17. Case 1: Diamond • What is the type of experiment? • Exogenous movement of peoples. Fisherman, seafaring peoplesettle outlying islands • Are other factors held constant in order to isolate the impact of the experiment? • To some extent, but there a very large number of variables he hopes to explain. Perhaps, too many to feel comfortable • Does he satisfy the “as if” criterion? • Not really. Clearly, fishermen selected the islands for settlement and presumably had some criteria for settlement which impacted the degree of settlement. • What caution does he take to make sure it is satisfied? • He argues that all islands he is studying were settled at some point and can thus be compared? • Is he careful about LATE when generalizing to the larger population? • “Polynesia offers us a small slice, not the full spectrum of the world’s human diversity.”

18. Case 2:Posner • What is the research question? • Under what conditions do ethnic cleavages become politically salient? • What is his hypothesis? • The most important factor is the relative size of the ethnic populations in the country’s political arena. • What role does the natural experiment play for him? • The natural experiment allows him to compare similar ethnic pairs in two different African countries (Zambia and Malawi) • What are his units of analysis? • 4 ethnically homogenous villages on either side of the border (2 Chewa and 2 Tumbuka). • How does he select them? • Pay attention to how careful he is to select villages to hold other factors constant (location from roads, location from borders… What else?

20. Notice that Posner’s n is really 2: He aggregates from a survey to get the dependent variable.

21. Regression Analysis

22. Case 2: Posner • What is the natural experiment he exploits? • The boundary originally demarcated by the British South African Company in 1891 to distinguish the territories of what were then called Northeastern and Northwestern – drawn purely for administrative purposes, with no attention to the distribution of groups on the ground. • Are other factors held constant in order to isolate the impact of the experiment? • Yes, 90% of the paper is devoted to fleshing out how the countries do not differ in the treatment of minorities in any other way. What are some of the factor Posner explores? • Does he satisfy the “as if” criterion? • Not really. He asserts that the border was drawn for only administrative reasons and uses some historical evidence. The achilles heal of this incredibly tight and well-designed paper is if someone can demonstrate that borders were drawn with reference to ethnic-cleavages (or worse…drawn to marginalize the Eastern ethnicities in Zambia). Posner is confident that they cannot and I suppose he is probably correct. • Is he careful about LATE when generalizing to the larger population? • Yes, within Africa. Posner takes great care to test implications of his theory to the Lakeshore Tonga in Malawi. He is a bit more extravagant in generalizing outside of Africa.

23. Case 3: Miguel • What is the research question? • What factors can ameliorate the deleterious of ethnic diversity on economic development. • What is his hypothesis? • Nation-buildings program leads to higher public goods funding in ethnically fragmented states. • What role does the natural experiment play for him? • The natural experiment allows him to compare similar villages in Kenya and Tanzania and assess the impact on public goods funding. • What are his units of analysis? • Similar districts in Kenya and Tanzania. • How does he select them? • Looks for similar ethnic composition (Bantu-majority), similar geography, economic focus. • He also makes an important assumption: Ethnic relations prior to independence are similar.

26. Case 3: Miguel • What is the source of the natural experiment? • Creation of border • Are other factors held constant in order to isolate the impact of the experiment? • As many as he can, but Miguel is not really satisfied himself with his ability to control for other factors. He refers to other studies and surveys without giving them too much credence. • Does he satisfy the “as if” criterion? • Not really. Why did nation-building occur in Tanzania and not Kenya poses big problems for him? Especially since the administrative apparatus of the two states is v. different. • Is he careful about LATE when generalizing to the larger population? • No

27. Case 4: Galiani and Schargrodsky • What is the research question? • Do property rights impact household development decisions and access to credit? • What is their hypothesis? • Property rights do impact economic behavior of actors. • What role does the natural experiment play? • It resolves the problem of reverse causality – that those wanting to develop their property are more likely to use them for development. • What are the units of analysis? • Parcels of land (Treatment – Owner resolved land dispute with government; Control – Owner has not resolved dispute). • How do they select them? • They survey every household in the development.

28. Case 4: Galiani • What is the exogenous movement? • Squatters all settle land randomly without knowledge of how disputes will be settled. • Are other factors held constant in order to isolate the impact of the experiment? • Yes, the parcels of land are similar in every conceivable way. (Size, quality…) • Does he satisfy the “as if” criterion? • Absolutely. To the point they even reconstruct those who were eligible but did not apply. • Is he careful about LATE when generalizing to the larger population? • Not really