Attribution of cause and effect in small n impact evaluation Howard White and Daniel Phillips

1. ww.3ieimpact.org Attribution of cause and effect in small n impact evaluation Howard White and Daniel Phillips

2. Addressing attribution of cause and effect • Impact evaluations should answer the question, ‘to what extent did the intervention being evaluated alter the state of the world?’ • Outputs and outcomes • Intended and unintended • All project-affected persons (PAPs)

3. Defining small n Data on too few units of assignment to permit tests of statistical significance between treatment and comparison groups

4. Defining small n • Why might this arise? • Small N, therefore small n • Heterogeneity • Some universal interventions • Budget

5. Approaches for small n impact evaluation • General Elimination Methodology (GEM) (a.k.a the Modus Operandi approach) • Realist Evaluation • Contribution Analysis • Process Tracing • Most Significant Change • Success Case Method • Outcome Mapping • Method for impact Assessment of Programs and Projects (MAPP)

6. Approaches for small n impact evaluation • Group I • General Elimination Methodology (GEM) • Realist Evaluation • Contribution Analysis • Process Tracing • Group II • Most Significant Change • Success Case Method • Outcome Mapping • Method for impact Assessment of Programs and Projects (MAPP)

7. Approaches for small n impact evaluation • Group I • General Elimination Methodology (GEM) • Realist Evaluation • Contribution Analysis • Process Tracing

8. Small n approaches: a common methodological core • Explicitly address attribution • Articulation of both a theory of change and alternative causal hypotheses • Consideration of context/external factors • Verification of the actual causal chain

9. Small n attribution: making the causal link • Experiments and quasi-experiments establish causation using statistics to create a proxy ‘closest possible world’ comparison group • The small n approaches outlined here explain social phenomena by examining the underlying mechanisms which lie between cause and effect

10. Mechanism-based causal attribution

11. The counterfactual

12. There are significant sources of bias in both the collection and analysis of qualitative data • Respondent bias • Evaluator bias

13. The Barry Manilow T-shirt experiment

14. There are significant sources of bias in both the collection and analysis of qualitative data • Respondent bias • Evaluator bias

15. A framework for small n analysis • The attribution question(s) • Setting out a programme’s theory of change • Identifying alternative causal explanations • Evaluation plan • Verifying the causal chain

16. ww.3ieimpact.org Attribution of cause and effect in small n impact evaluation Howard White and Daniel Phillips