Howard White International Initiative for Impact Evaluation

1. Getting what we pay for: impact evaluation for better planning and budgeting Regional conference on public sector management in support of the MDGsBangkok, June 2012 Howard White International Initiative for Impact Evaluation

2. Impact evaluation: an example The case of the Bangladesh Integrated Nutrition Project (BINP) Why did the Bangladesh Integrated Nutrition Program (BINP) fail?

3. Comparison of impact estimates

4. The theory of change

5. The theory of change Right target group for nutritional counselling PARTICIPATION RATES WERE UP TO 30% LOWER FOR WOMEN LIVING WITH THEIR MOTHER-IN-LAW

6. The theory of change Knowledge acquired and used

7. The theory of change The right children are enrolled in the programme

8. The theory of change Supplementary feeding is supplementary

9. Lessons from BINP • Apparent successes can turn out to be failures • Outcome monitoring does not tell us impact and can be misleading • A theory based impact evaluation shows if something is working and why • Quality of match for rigorous study • Independent study got different findings from project commissioned study

10. Stipends in rural China • Enrolments rose from 40 to 92 percent in project areas • So stipends “caused” growing enrolments amongst girls

11. “Results reporting” Results… cannot as a rule be attributed specifically, either wholly or in part, to the Netherlands. (Results report 2005-06)

12. Development effectiveness = how effective are development programmes = what difference did they make • To measure this we need impact evaluation • Results are what we achieved, not what would have happened anyway • So outcome monitoring is not enough

13. Take away message number 1: Results means impact, so only impact evaluation can tell us if we are achieving results. Results are not captured by outcome monitoring

14. So, what is impact evaluation?

15. What is impact evaluation? Impact evaluations answer the question as to what extent the intervention being evaluated altered the state of the world = the (outcome) indicator with the intervention compared to what it would have been in the absence of the intervention = Yt(1) – Yt(0) We can see this But we can’t see this So we use a comparison group

16. What do we need to measure impact?

17. What do we need to measure impact? Girl’s secondary enrolment in rural China The majority of evaluations have just this information … which means we can say absolutely nothing about impact

18. Before versus after single difference comparisonBefore versus after = 92 – 40 = 52 “scholarships have led to rising schooling of young girls in the project villages” This ‘before versus after’ approach is outcome monitoring, which has become popular recently. Outcome monitoring has its place, but it is not impact evaluation

19. Rates of completion of elementary male and female students in all rural China’s poor areas Share of rural children 1993 1993 2008 2008

20. Post-treatment comparison comparisonSingle difference = 92 – 84 = 8 But we don’t know if they were similar before… though there are ways of doing this (statistical matching = quasi-experimental approaches)

21. Double difference =(92-40)-(84-26) = 52-58 = -6 Conclusion: Longitudinal (panel) data, with a comparison group, allow for the strongest impact evaluation design (though still need matching). SO WE NEED BASELINE DATA FROM PROJECT AND COMPARISON AREAS

22. Take away message number 2: Impact evaluation requires a valid comparison group, and baseline data really help. So ex ante design is best

23. Comparison group: an identical group of individuals, or households, or firms, or sub-districts, but NOT subject to the programme. Where do we get the comparison group from?

24. RANDOMIZATIONRANDOMIZATIONRANDOMIZATION RANDOMIZATIONRANDOMIZATION RANDOMIZATION RANDOMIZATION RANDOMIZATIONRANDOMIZATION RANDOMIZATIONRANDOMIZATIONRANDOMIZATIONRANDOMIZATIONRANDOMIZATION RANDOMIZATION RANDOMIZATION

25. Random assignment of the intervention… Not the same as taking a random sample of the ‘treated’ Some examples….

26. Voter education (Rajasthan and Delhi) • Outcomes: voter turnout, vote share of incumbent, politician behavior, service delivery • Intervention: pre-election voter awareness campaigns (report cards) • Unit of assignment: 375 GPs, half to get intervention

27. Schooling and early marriage • Outcome: marriage, school attendance and attainment • Intervention: in-kind transfer for girl remaining in education and unmarried • Unit of assignment: village

28. Health-based education programs Eyeglasses Vitamin pills

29. Some different ways to randomize Pipeline Raised threshold

30. Overcoming resistance to randomization • There is probably an untreated population anyway • Need not randomly allocate whole programme just a bit • Exploit • Roll out • Raised threshold • Encouragement designs • Don’t need ‘no treatment’ control • RCTs are not unethical, spending money on programmes that don’t work is unethical

31. Take away message number 3: RCTs are possible in a large range of settings… though it is not the only way to conduct IE

32. Well designed IEs lead to more nuanced questions • E.g. conditional cash transfer second generation questions: • Conditions or not? • What sort of conditions? • Who to give money to? • How to give the money? • When and how often to give money?

33. Second generation questions: computer-assisted learning, CAL • Most cost effective number of children per computer? • What sort of software? • How much teacher training required? • What technological back up needed? • What age groups to target?

34. So conduct studies to get inform design to get better results

35. And which policies are most cost effective

36. Take away message number 4: Impact evaluation is not just about what works, but why, where and at what cost, and offers insights on intervention design, and so delivers better results

37. Implications for results-based budgeting In principle can identify priority outcomes, and what interventions are most cost effective in achieving these outcomes, and so allocate budget to things that work This IS being done in some countries…

38. But it’s not happening in most • “Evaluation is not systematically embedded in the GoU’s management practices…Because evaluation addresses issues such as actual progress in attainment of program objectives, cost effectiveness, and value for money, it responds to some of the aspects of Uganda’s M&E system that are most critically lacking.” • “There has been a general tendency to monitor rather than evaluate.” (Sri Lanka) • “…the distortion found in most countries of an excess of monitoring and a dearth of genuine evaluation.” (World Bank)

39. And attribution is not addressed • “…M&E is not geared toward understanding causality and attribution between the stages of development change.” (Uganda) • “Furthermore, while national and provincial treasuries have emphasized an approach to collecting information that is based on logical framework (log-frame) results chain, they have not focused on attribution or causality.” (South Africa)

40. But there are growing cases..

41. Evidence into practice examples

42. Recommendations • Review current M&E systems and how it aligns with requirements for “results” • Identify some priority areas for impact evaluation, and commission a small number of studies (both ex post and ex ante) • Start development of national framework to build systematic impact evaluation into M&E, and budgeting to ‘performance’ meaning results, meaning impact

43. Thank you Visit www.3ieimpact.org