Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation

1. Are Disasters Any Different?Challenges and Opportunitiesfor Post-Disaster Impact Evaluation Alison Buttenheim, Princeton University Howard White, 3ie Rizwana Siddiqui, PIDE Katie Hsih, Princeton University April 1, 2009 Cairo

2. 3ie post-disaster impact evaluation (PDIE) study Motivation: • Frequency and severity of natural disasters • Quantity of assistance provided in post-disaster settings • Recent interest from humanitarian and development sectors in more and better impact evaluation • Opportunity to use Pakistan ERRA experience as case study

3. 3ie post-disaster impact evaluation (PDIE) study Goals: • Review existing approaches to PDIE • Develop a framework for rigorous PDIE • Apply framework to the 2005 Pakistan earthquake case • Identify a set of principles to guide PDIE

4. Disasters Natural events: 414 reported in 2007 (CRED criteria)

5. Disasters Natural events: 414 reported in 2007 (CRED criteria) Human consequences: 211 million affected 16,847 lives lost USD 100+ billion damages

6. Disasters Natural events: 414 reported in 2007 (CRED criteria) Human consequences: 211 million affected 16,847 lives lost USD 100+ billion damages Institutional responses

7. Post-disaster relief and recovery efforts • USD 5.9 billion (pledged) for 2005 Pakistan earthquake • USD 13.5 billion (pledged) for 2004 Indian Ocean tsunami • Actors: Diverse mix of governments, funders, IFIs, aid agencies, humanitarian agencies, int’l/local NGOs.

8. Extensive process evaluations Multiple levels of analysis (project, agency, sector, disaster) Some joint evaluations (e.g. TEC) Review of ALNAP database, etc. suggests few examples of “rigorous” impact evaluation How does PD assistance get evaluated?

9. Why so little focus on IE in PD settings?

10. Why so little focus on IE in PD settings? “Disasters are different” “Disasters are different” “Disasters are different” “Disasters are different” “Disasters are different” “Disasters are different”

11. Are disasters any different? • Unpredictable, rapid-onset event • Proven life-saving measures cannot be randomized or withheld • Mismatch between resources and need (sometimes) • Absence of baseline data (usually) • Which counterfactual is the right one?

12. Are disasters any different? Maybe not… • Nonrandom exposure to disaster event and consequences • Nonrandom assignment of interventions • Fragile states/vulnerable populations • Multiple concurrent interventions • Which counterfactual is the right one?

13. Lessons learned from other PDIE experiences • Bangladesh floods, 1998 • Hurricane Mitch, 1998 • Indian Ocean tsunami, 2004 • Hurricane Katrina, 2005

14. Disaster-related time periods Emergency Relief Recovery/Reconstruction DISASTER Pre-disaster Immediate post-disaster Post-intervention (1) Post-intervention (2) t-1 t0 t1 t2 14

15. Disaster-related populations * or communities (or other unit of analysis) ‡ or receive them later, or receive different ones † or less-affected households/communities 15

16. Within treatment group, single-difference over time

17. Within treatment group, single-difference over time ERRA: “Build Back Better”

18. Within treatment group, single-difference over time Problems: Recall bias if no baseline; attribution?

19. – – – – Cross-sectional, single-difference over treatment groups (A vs. C)

20. – – – – – – Cross-sectional, single-difference over treatment groups (A vs. C)

21. – – – – – – Cross-sectional, single-difference over treatment groups (A vs. C) Implied counterfactual: What would “A” households look like if there had been no disaster?

22. – – – – – – Cross-sectional, single-difference over treatment groups (A vs. C) Problems: Is there an appropriate “C” group? If so, were they observed? Attribution?

23. Difference-in-difference (A vs. C)

24. Difference-in-difference (A vs. C) Controls time-variant factors that are the same between A & C

25. – – – – – – Cross-sectional, single-difference over treatment groups (A vs. B)

26. – – – – – – Cross-sectional, single-difference over treatment groups (A vs. B)

27. – – – – – – Cross-sectional, single-difference over treatment groups (A vs. B) Implied counterfactual: What would “A” households look like if there had been no intervention?

28. – – – – – – Cross-sectional, single-difference over treatment groups (A vs. B) Problems: How were interventions assigned to A but not to B?

29. Difference-in-difference (A vs. B)

30. Difference-in-difference (A vs. B) Controls time-variant factors that are the same between A & B

31. World Bank impact evaluation of housing and livelihood grants

32. World Bank impact evaluation of housing and livelihood grants • Instrumental variable approach to disaster impact: • Villages at same distance from epicenter, at same elevation and slope had comparable pre-disaster SES • Villages at different distance from fault line experienced different earthquake severity.

33. World Bank impact evaluation of housing and livelihood grants • Instrumental variable approach to disaster impact: • Villages at same distance from epicenter, at same elevation and slope had comparable pre-disaster SES • Villages at different distance from fault line experienced different earthquake severity. A1-C1

34. World Bank impact evaluation of housing and livelihood grants • Instrumental variable approach to disaster impact: • Villages at same distance from epicenter, at same elevation and slope had comparable pre-disaster SES • Villages at different distance from fault line experienced different earthquake severity. • Variation in receipt of relief and recovery funds: • Between-district variation in implementing agency for housing grant • Threshold eligibility for livelihoods grant of 5 dependents/households: regression continuity design. A1-C1

35. World Bank impact evaluation of housing and livelihood grants • Instrumental variable approach to disaster impact: • Villages at same distance from epicenter, at same elevation and slope had comparable pre-disaster SES • Villages at different distance from fault line experienced different earthquake severity. • Variation in receipt of relief and recovery funds: • Between-district variation in implementing agency for housing grant • Threshold eligibility for livelihoods grant of 5 dependents/households: regression continuity design. A1-C1 A1-B1

36. ERRA impact evaluation case study • Evaluation opportunities using existing data & HH sample • Household data collection at t2 • Retrospective household reports of t0 • Use of ongoing government household surveys (e.g., HIES) as baseline • Randomization of some interventions from 2009 

37. ERRA impact evaluation case study • Evaluation opportunities in a future disaster • Maintain surveillance sample in disaster-prone regions • Household-level data collection at t0 • Randomized interventions, e.g, • Timing of interventions: • Group 1: Housing grant first, followed by livelihood cash grant • Group 2: Livelihood cash grant first, followed by housing grant • Conditionality of grants • Types of interventions, e.g, different formats or recipients of livelihoods cash grant

38. PDIE Guiding Principles • PDIE is necessary to ensure that relief and recovery funds are appropriately targeted, effective, and efficient. • Each phase of a disaster (emergency, relief, recovery/reconstruction) presents distinct evaluation challenges and therefore may require a different evaluation approach or methodology. • “Evaluation preparedness” is an important part of disaster preparedness.

39. PDIE Guiding Principles • PDIE should incorporate evaluation of (pre-disaster) investments in disaster mitigation, prevention, and resilience. • Rigorous PDIE requires the tools and perspectives of multiple disciplines and sectors. • Quantitative PDIE can benefit from the qualitative and mixed-methods approaches.

40. PDIE Guiding Principles • Proportionate changes in outcomes over time and over groups can be as instructive as changes in levels. • Change-over-time impact evaluations should recognize two distinct baselines: pre-disaster, and immediately post-disaster.

41. PDIE Guiding Principles (ct’d) • PDIE will be most successful when the goals of the intervention are clearly defined through a logical framework or similar model; when the interventions are appropriately targeted, and when the purpose/use of the evaluation is clear. • Experimental and quasi-experimental approaches are feasible in PDIE if ethical, logistical and “fit” issues are adequately addressed.