Mixed-Method Evaluation: 5 things ethnographers and statisticians should discuss

1. Mixed-Method Evaluation:5 things ethnographers and statisticians should discuss Thad Dunning University of California, Berkeley DFID/EGAP Learning Event July 10, 2014

2. Mixing methods • Under the right conditions, both qualitative and quantitative methods can contribute critically to inferences about impact. • How should they be combined? • Imagine a research team conducting an RCT, or using a natural experiment to evaluate the impact of a policy. • Many such teams feature (1) experts in research design and (2) experts in case knowledge and qualitative methods • What topics should ethnographers and statisticians discuss?

3. An example on the effect of land titles • In 1981, squatters organized by the Catholic church occupied an urban wasteland in the province of Buenos Aires, dividing the land into similar parcels. • After the return to democracy, a 1984 law expropriated the land, with the intention of transferring title to the squatters. • Some of the original owners challenged the expropriation in court, leading to long delays, while other titles were ceded and transferred to squatters. • The legal action created a “treatment” group—squatters to whom titles were ceded—and a control group, that is, squatters whose titles were not ceded

4. The apparent effects of property rights • Galiani and Schargrodsky (2004, 2010) find significant differences in housing investment, teenage fertility, and educational attainment of children (but not in access to credit markets, contradicting Hernando De Soto) • Squatters who received titles through a stroke of good fortune also believed more in their individual efficacy—e.g. they disproportionately agreed with the statement “people get ahead in life through hard work, rather than luck” (Di Tella et al. 2007) • But are these causal relationships? And if so, what explains the effects? • Answering this suggests five topics the ethnographers and the statisticians should discuss

5. 1. What is the treatment-assignment process? • According to these authors, assignment to titles was like a coin flip: as-if random. • One source of quantitative evidence is the pre-treatment equivalence of treated and untreated units, e.g.: • Titled and untitled parcels are side-by-side; the government offered similar compensation across treatment and control parcels • Pre-treatment characteristics of the parcels(such as distance from polluted creeks) are similar in both groups • Pre-treatment characteristics of squatters (age, sex, etc.) do not predict whether they received title • Thus, quantitative data suggest statistical equivalence on pre-treatment characteristics (those not affected by treatment)—just as we would expect after randomization

6. 1. What is the treatment-assignment process? • But even more important for validating the natural experiment is qualitative evidence: • Squatters and Catholic Church organizers did not appear to know the land was privately owned • They did not anticipate the expropriation of land by the state • They had no basis for predicting which particular plots of land would have been expropriated, and thus for assigning plots to particular squatters • This information does not come in the form of systematic values on independent and dependent variables—but rather from knowledge of context and the process by which treatment assignment took place. • Such qualitative information on the treatment-assignment process is critical, in RCTs as in natural experiments.

7. 2. What are the right modeling assumptions? • For RCTs and many natural experiments, the Neyman urn is a good model of the treatment-assignment process

8. 2. What are the right modeling assumptions? • The Neyman model makes key assumptions, beyond as-if random: • Non-Interference: Each unit's outcome depends only on its treatment assignment, and not on the assignment of other units. (Analogue to regression models: Unit i’s response depends on i’s covariate values and error term). • Exclusion restriction: Treatment assignment only affects outcome through treatment receipt • These are central for the Argentina land-titling study: • One can imagine ready violations of both assumptions (such as…?) • Quantitative and qualitative information can help assess how plausible the violations might be • Practicing the ethnography of spillovers…

9. 3. What is the treatment? • In many studies, understanding the implementation of treatment—and the components it contains—is critical • E.g. what is “community monitoring” • This links to issues of cumulation—implementation of interventions can be very different, even across ostensibly similar studies • This is no less true in the land-titling study: • Are titled squatters aware of titles; do they value them; how do they perceive them? • What components of the treatment do squatters themselves perceive as important? (Connected also to exclusion restrictions) • Both quantitative and qualitative information matter here.

10. 4. What are the auxiliary outcomes? • If a theory of change is true, it often generates observable implications for secondary outcomes • In the land-titling study, greater effects on access to financial markets might imply other uses of land as collateral • Just as important are placebo outcomes—settings where it is “known” that no effect exists • Qualitative knowledge can be critical for finding potential placebo outcomes • E.g., sub-groups for which we might expect no effect; if pre-specified before data analysis, can provide powerful tests

11. 5. What are the mechanisms? • A critical point: identifying mechanisms (like the average effects of treatment) implies knowledge of counterfactuals • Macartan’s discussion (session #2) highlights the difficulties • Even with a randomly manipulated mediator, we don’t know if the types who respond to a manipulated mechanism would respond in the same way if induced by change in treatment status • But there are pragmatic approaches: • At the least, identify counterfactual changes in intermediate attitudes, beliefs, behaviors by comparing to a control group • Such “experimental ethnography” can use qualitative as well as quantitative data • Not sufficient for identification but can sure help a lot.

12. Conclusion: Integration of evidence, in sum • David Collier et al. (2010): • A data-set observation (DSO) is the collection of values on independent and dependent variables for one case—e.g., a row in a rectangular data matrix. Example: in an RCT or natural experiment, the treatment assignment status and the outcome for each unit (plus auxiliary data) • A causal-process observation (CPO) comes in some other format; it is “an insight or piece of data that provides information about context, process, or mechanism" (Collier et al. 2010: 184) and reflects in-depth knowledge of one or more units, and/or the broader context in which DSOs were generated • Both kinds of information are critical in RCTs and natural experiments, as in other studies.