What is Impact Evaluation and When and How Should We Do It?

1. What is Impact Evaluation and When and How Should We Do It? Linxiu Zhang Center for Chinese Agricultural Policy Chinese Academy of Sciences Nov.14, 2011, Shanghai

2. Special note • This presentation is heavily based on the materials provide by Howard White, the Executive Director of 3ie and Scott Rozelle, A Stanford Professor, REAP co-director. • So, no specific acknowledgements will be given in presentation.

3. The Overall Agenda • When Will We Ever Learn: general introduction to Impact Evaluation • When Random Assignment is Possible? • Implementing and Evaluating RCTs • When Random Assignment is Not Possible? • Quasi-experimental methods  propensity scores, matching, IV, Regression Discontinuity and DinD.

4. Amazing Ideas • Sleeping Bag Incubator • Treadle Pump Irrigation • Agricultural Price Services through Mobile Phones • Computer Assisted Learning for Remedial Tutoring

5. New programs in China (huge fiscal investment) • Rural Health Insurance (NCMS or hezuo yiliao in China) • New Subsidy Program (liangshi butie) • New Education Programs (e.g., raising teacher salaries … or … eliminating tuition for high school) • Financial Crisis Stimulus Package (investments by central gov’t; investments by localities)

6. How many of the innovations/programs that we heard about on the news … … how many of the new technologies/programs that we have become excited about … … how many have been rigorously evaluated? • Do we have empirical evidence, based on a carefully constructed counterfactuals, that these breakthroughs/programs work, can positively affect the lives of the poor and do so in a cost effective way? Unfortunately, the answer is almost certainly some, but, not many …

7. Huge Global Initiatives • UN Millennium Development Villages • USAID Bilateral Investment Program • World Bank / ADB’s Loan Program

8. Statement of Facts: “Accelerating social progress in low- and middle-income countries requires knowledge about what kinds of social programs are effective. Yet all too often, such basic knowledge is lacking because governments, development agencies, and foundations/NGOs have few incentives to start and sustain the impact evaluations that generate this important information.” (International Evaluation Gap Working Group) “When it comes to attribution, there is shockingly little concrete evidence about what works and what does not” (Author of report: When will we ever learn) I was at a conference about 2 to 3 years ago, where one young researcher claimed (in front of scores of older, experienced development economists  after 40+ years of development economics and we had not learned anything … until his work (of course)

9. The Excuses • We don’t have time • It costs too much to do rigorous impact evaluation • It is unethical • Project implementation is site and context specific

10. The Excuses • We don’t have time • It costs too much to do rigorous impact evaluation • It is unethical • Project implementation is completely site and context specific • We already know!

11. Example: • J. Sachs  we already know ITN’s work (insecticide treated nets) for the prevention of malaria • In fact, this time  rigorous public health trials support it: • 90 villages: give residents ITN’s • 90 villages: give residents “0” • In treatment villages, reductions of malaria, anemia and other benefits • even positive spillovers: villages/hamlet around the treatment villages (within 300 meters) also benefitted through reduction of malaria (although no ITN’s) … miracle?

12. ITN’s(insecticide treated nets)

13. Policy implications • People were not buying them … • Despite people being “very afraid” of malaria … • Why? • Stanford University team’s hypothesis:  one-time cost too high • Leads to a new RCT

14. Micro credit or free • Through an NGO that had “cells of members” [10 to 20 to 30 households per village) in 100s of villages in India, did RCT with two treatment arms: • Treatment village 1: give away for free to all NGO members • Treatment village 2: sell ITN’s to households as part of a Micro credit (peer monitoring) project • Control villages: “0” • What is the outcome?

15. Impact: ZERO[none: for malaria / none: for anemia …NONEnone in treatment village 1 / none in treatment village 2 / none in control villages] • Explanation: • Have no definitive proof (though now we may know why villagers do not buy them … they don’t seem to work … • Theory: • Revisit the original trial … and Revisit and live in own project villages • People do not always use ITN’s … trouble / hard to hang / uncomfortable / too many people, not enough nets

16. An explanation • How is it that if people do not use them (even in the original public health trial treatment villages) that they have an impact in the villages AND on surrounding villages? • Only real difference between original trial (100% of households in trial) and Stanford’s trial (10 to 20% of households in trial)  maybe it is that all mosquitos are killed and populations collapse when all households have ITN’s … this would account for efficacy in trial and the spillover … • However, in the partial roll out villages, the ITN’s not effective!

17. ITN’s do work … but, with a caveat • Current most plausible explanation: • In the large public health trial, when all of the villagers received the ITN’s … and were encouraged to use them (and did, at first) … ALL of the mosquitos died … this reduces malaria in the treatment villages and the surrounding hamlets • Jeffrey’s response? • Of course, that is why we give them to all of the families … of course, he had no idea … • Maybe he did “know” … but, surely he does not understand … [but, then none of us do now]

18. Rest of the session plan • Introduce the concept of IE • Definitions and examples of what is right and what is not right • RCT’s … when possible, sexy gold standards! • When you can’t randomize (still a lot of excitement) • IE is not enough: Supplementary tools • Issues in choosing an IE strategy • Selecting a control group • RCTs or Quasi-experimental approaches?”

19. What is impact? • Impact = the (outcome) Indicator with the intervention compared to what it would have been in the absence of the intervention • Unpacking the definition • Can include unintended outcomes • Can include others not just intended beneficiaries • No reference to time-frame, which is context-specific • At the heart of it is the idea of a attribution – and attribution implies a counterfactual (either implicit or explicit)

20. Defined in this way – UNFORTUNATELY (as discussed above) – we (the international development community) have little evidence on impact of development programs [in other words: we don’t (systematically) know the results of many of the uncountable number of programs that development agencies, gov’ts and other organizations have been implementing in recent years]

21. The attribution problem:factual and counterfactual With project Project impact With project Impact varies over time Impacts also are defined over time … Little attention has been given to the dynamics over time … though people think about this …

22. Change in the CAL program effect on the standardized math test scores over time 0.12 0.11 0 The CAL program effect occurred by the midterm evaluation, less than two months after the start of the program.

23. Change in the CAL program effect on the standardized math test scores over time 0.12 0.11 0 There is no improvement between month two and month three..

24. Impact of nutrition at infancy in Guatemala • After 2 years  greater BMI • After 10 years  higher grades in school • After 15 years  higher school attainment • After 40 years  higher wages / income

25. What has been the impact of the French revolution? And, even longer run … “It is too early to say” Zhou Enlai

26. Lets examine a less grandiose intervention • The venue: Poor areas of South West China … a remote mountainous region … populated by groups of Dai and Dong minorities … • In 1980s and 1990s only small share of girls attended school … most were involved with farming, tending livestock and raising siblings … • An NGO began giving scholarships in the early 1990s … objective: increase attendance of girls … they claim in their very polished promotion material and in the many workshops that they attend that they have been effective in their mission …

27. What do we need to measure impact?Girl’s primary school enrollment NOTE: if you measure this well, what is it? Outcome monitoring The majority of evaluations have just this information … which means we can say absolutely nothing about impact

28. What does 92 percent mean? • Is it high? • Is it low? • What does a single number mean? • What do we compare this to? Even if done well … output monitoring in its simplest form  TELLS US NOTHING about impact

29. “Before versus after” single-difference comparisonsBefore versus after = 92 – 40 = 52 “scholarships have led to rising schooling of young girls in the project villages” This ‘before versus after’ approach is more careful outcome monitoring, which has become popular recently. Outcome monitoring has its place, but: outcome monitoring ≠ impact evaluation

30. The changing macro environment … and rising employment opportunities and wages Percent of cohort Yuan / month Employment in the off farm labor market – 16 to 25 year olds Off farm wage rate

31. Rates of completion of elementary male and female students in all rural China’s poor areas Share of rural children 1993 1993 2008 2008

32. Outcome monitoring does not tell us about effectiveness Results… cannot as a rule be attributed specifically, either wholly or in part, to the intervention

33. An (important) asideCollecting data in order to measure outcomes “before an intervention” • Can we collect data about outcomes before interventions, after the intervention (that is: is recollection data valid?) • No (or be careful): work by economists have shown clearly that there are lots of biases introduced to IE by relying on recollection data (most of them psychological) • If individuals have been given a treatment, they often will selectively remember … they will exaggerate the benefits as a way of showing their gratefulness … • Those in the control groups will often want to show they are less fortunate and understate their condition (or improvement) • Empirically, recollection data have lots of biases … hard to determine the direction … • Best practice (only practice?): collect baseline before the project begins

34. Post-treatment control comparisonsSingle-difference = 92 – 84 = 8 Another common approach (lets compare to another set of villages):

35. But we don’t know if treatment and control groups were similar before… • How often are intervention villages / schools / clinics / etcetera / chosen in a way that make them systematically different than control villages? [either for convenience / political necessity / feasibility / cost considerations / or from leaving it to the local partner who uses who-knows-what-type of selection method] • In the SW China villages, the NGO went to a poor county, but, the local bureau of education chose the villages … and chose them along the road … • Is attendance in elementary school lower in the control villages because the NGO did not pass out scholarships, or because villagers in control villages had less use for education (or the cost of going to school higher)

36. Post-treatment control comparisonsSingle difference = 92 – 84 = 7 Another common approach (lets compare to another set of villages): Main point: Post treatment control comparisons are only valid if treatments and controls were identical at the time the intervention began …

37. Double difference =(92-40)-(84-26) = 52-58 = -6 Therefore: lets collect data for all of the cells? Conclusion: Longitudinal (panel) data, with a control group, allow for the strongest impact evaluation design (BUT: still need matching … if they are different at the start of the project … is there something different in the village which would affect the village’s response to the intervention?)

38. Main points so far • Analysis of impact implies a counterfactual comparison • Outcome monitoring is a factual analysis, and so cannot tell us about impact • The counterfactual is most commonly determined by using a rigorously/carefully chosen control group If you are going to do impact evaluation you need a credible counterfactual using a control group (not necessarily RCT / but, still need control)

39. “Gold Standard” Randomized Control Trials Medical Zero What is the counterfactual?

40. “Gold Standard” Randomized Control Trials Crop field trials Zero What is the counterfactual?

41. Can also do randomized control trials in schools to test the effectiveness of new school program … Social Experimentation … Step 1: choose 50 schools … randomly divide into 2 groups 25 elementary schools in Gansu 25 elementary schools in Gansu

42. Does one egg per day, improve test scores / attendance? One Egg Per Day None 0 25 elementary schools in Gansu 25 elementary schools in Gansu What is counterfactual?

43. Randomized Control Trial[like in agriculture or medicine] our question:Will one egg per day lead to higher test scores? Three Stages treated 2. POLICY EXPERIMENT RCT’s 1. Baseline survey 3. Evaluation survey control 0

44. Results: One Egg vs. “0” Change in test scores between baseline and evaluation surveys Difference statistically significant at 95% level of confidence Control Schools (0) One Egg / Day Schools What is causing the difference between Treatment Schools and Control Schools? What is the counterfactual?

45. Before, we talk more about pros/cons and keys/pitfalls of running large RCTs studies:Broaden our set of definitions about IE • Discussion above was for ‘large n’ interventions • There are a large number of units of intervention, e.g. children, households, firms, schools. • Examples of “small n” are most (but not all) policy reform and many (but not all) capacity building projects. • E.g.: some reforms (e.g. health insurance) can be given large n designs • ‘Small n’ interventions require: • Modelling (computable general equilibrium, CGE, models), e.g. trade and fiscal policy … or role of agriculture in development? • A theory-based analysis (this is what is modeled in a small-n study …) … it is the logic through which the reform / new capacity will drive economic change …

46. In fact, many things can’t be randomized? • Effect of a road on access to off farm employment • An agricultural subsidy program (that already has been rolled out? • Impact of the decisions of migrant families to: leave kids behind (get educated in village’s rural public schools while living with Grandma … or go with Mom and Dad and get educated in the city in a private, unregulated migrant school).

47. How well do students that attend migrant schools perform in standardized tests? Children in migrant schools actually are a bit above those in poor rural schools Standardized math score

48. Control for observable characteristics of students and parents (in both rural schools and migrant schools) … and for length of time that migrant children have been in migrant schools … and using quasi experimental methods (e.g., matching in this case) Standardized math score The argument is that parents bring their children that are better students into urban areas with them … so after controlling for these factors … the difference goes away … AND: If you then compare students in migrant schools that have been in Beijing for > 3 years

49. In fact, many things can’t be randomized? • Effect of a road on access to off farm employment • An agricultural subsidy program (that already has been rolled out? • Impact of the decisions of migrant families to: leave kids behind (get educated in village’s rural public schools while living with Grandma … or go with Mom and Dad and get educated in the city in a private, unregulated migrant school). Should we not work on these questions?

50. If we want to study impact of migrant education