Using Evidence to Inform Policy: Impact evaluation of Human development programs

1. Arielfiszbein Chief economist, human development World bank Using Evidence to Inform Policy:Impact evaluation of Human development programs Overseas Development Institute May 4, 2010

2. Motivation ‘Since it is difficult to distinguish the good from the bad prophet, we must be suspicious of all prophets: it is better to avoid revealed truths, even if we feel exalted by their simplicity and splendor, even if we find them comfortable because they come at no cost. It is better to be content with more modest and less inspiring truths that are laboriously conquered, step by step, with no shortcuts, by studying, discussion and reasoning, and that can be verified and demonstrated'’ Primo Levi

3. How can we learn from experience? • Traditional approach: identify best practice cases and inductively derive conclusions. • But the strength of those conclusions is directly related to the internal and external validity of the underlying evaluation. • We could do better if we plan well and take a more systematic approach to learning from experience.

4. The evaluation problem • Impact: difference in outcome with and without program. Cannot observe simultaneously. • with and without comparisons deceptive if participation depends on individual attributes that influence outcomes • before/after comparisons can be misleading if other things happened during the period • Need to estimate counterfactual (control or comparison) • Treated & control have same characteristics on average, • Only reason for difference in outcomes is the intervention

5. Multiple design options • Experimental (often using phase in) • Quasi-experiments/non-experimental • Regression Discontinuity (RD) • Difference in difference – panel data • Other (Instrumental Variables, matching, etc) • In all cases, these will involve knowing the rule for assigning treatment • Advantages of prospective design

6. Evolution of IE in the World Bank

7. Strong focus on HD areas

8. Strategic themes

9. … and IEs of non-HD projects often measure impact on HD outcomes: • Agricultural Development • Indicators: anthropometrics • Housing • Indicators: morbidity, anthropometrics • Rural Electrification • Indicators: study time, respiratory illnesses • Rural Roads • Indicators: school attendance • Urban Upgrading • Indicators: employment • Water Supply and Sanitation • Indicators: morbidity, anthropometrics

10. Building a global evidence base on the impact of HD programs

11. Priority thematic areas • Conditional Cash Transfers • Paying for Performance in Health (P4P) • School Accountability • Malaria Control • Active Labor Market Programs / Youth Employment • HIV/AIDS Prevention • Early Childhood Development

12. CCTs have become very popular….

13. CCTs have become very popular….

14. First generation evaluation questions • Impact on consumption and poverty • Side effects? • Impacts on service utilization • Impacts on human development outcomes

15. Impacts on consumption and poverty

16. CCT impacts on food consumption

17. Impacts on school enrollment

18. Impacts on health service utilization

19. Impacts on education and health outcomes • Only mixed success in terms of improving final outcomes in education and health: Education: Increases in school enrollment and years of completed schooling have not come hand-in-hand with improved learning outcomes Health: Some programs, but my no means all, have improved child nutrition (as measured by height-for-age, hemoglobin status)

20. Second generation evaluation questions • Role of conditions: Testing CCTs vs. UCTs (Burkina Faso, Malawi, Morocco and Yemen) • Does it matter who receives the cash? Mothers vs. fathers (Burkina Faso, Morocco and Yemen); Girls vs. parents in Malawi • Design of payments: Cambodia; Colombia

21. Priority thematic areas • Conditional Cash Transfers • Paying For Performance in Health • School Accountability • Malaria Control • Active Labor Market Programs / Youth Employment • HIV/AIDS Prevention • Early Childhood Development

22. Results-based financing (RBF) ≈ Pay-for-performance (P4P) Provision of payment for the attainment of well-defined results Transfer of money or material goods conditional on taking a measureable action or achieving a predetermined performance target (CGD, 2009) RBF takes many forms… Payers Payees DonorCentral governmentLocal governmentPrivate insurer Service providersFacilities / NGOsCentral governmentLocal governments $

23. Impact evaluation of P4P in Health • Effects on non-contracted services provided? • Impact on equity of services? • Impact on out-of-pocket expenditures? Will the introduction of pay for performance for patient services impact the quality and performance of hospitals/health centers? Impact Evaluations:

24. P4P in Rwanda National P4P scheme to supplement input-based budgets with bonus payments based on the quantity and quality of maternal and child healthcare (14 output indicators)

25. “Paying Primary Health Care Centers for Performance in Rwanda” (Basinga et al., 2010) Impact on proportion of institutional deliveries

26. “Paying Primary Health Care Centers for Performance in Rwanda” (Basinga et al., 2010) Impact on quality of prenatal care

27. P4P in Rwanda: Lessons • No effect on the number of prenatal care visits or on immunization rates. Need demand side incentives? • Greatest effect on services that had the highest payment rates and are under the provider control. • Financial performance incentives can improve both use of and quality of health services. • An equal amount of financial resources without the incentives would not have achieved the same gain in outcomes.

28. P4P experiment in UgandaLundberg, Marek and Pariyo (2007) Contracting for Primary Health Care in Uganda • Three arms: (1) performance bonus, (2) freedom to allocate base grant, (3) control. • Performance bonus based on meeting targets (pre-natal care, attended births, immunizations, etc.) • Results after 18 months: • bonus facilities perform the same as or worse than others; • freedom-to-allocate facilities perform significantly better than others.

29. P4P experiment in Uganda • This performance bonus didn’t work. • Amounts not large enough? • Scheme too complicated? • Not enough time? (some evidence of learning curve) • Facilities may allocate budgets more effectively than the Ministry of Health

30. Priority thematic areas • Conditional Cash Transfers • Paying For Performance in Health • School Accountability • Malaria Control • Active Labor Market Programs / Youth Employment • HIV/AIDS Prevention • Early Childhood Development

31. Strategies for improving learning outcomes • Supply side policies (classrooms, textbooks, extra tutors, teacher training) • Demand side policies (CCTs?) • Child endowments/readiness to learn -- ECD, de-worming • Accountability reforms: Change the environment in which decisions are made

32. WDR 2004 framework of accountability

33. Rationale for Education Accountability Schemes How do we transform this teacher…. …into this teacher?

34. School Accountability

35. 3 Modalities on School Accountability 1. School-based Management (SBM) 2. Information for accountability projects 3. Teacher contracting and pay for performance projects

36. How can SBM improve outcomes? • Those at the local level have more or better information and different objectives • Key decisions about school personnel • Key decisions about spending • Changes in the educational process • Resource mobilization • Changes in the school climate

37. School Based Management in Mexico(Apoyo a la GestiónEscolar, AGE) • Grants + training to Parent Associations (PA). • AGEs increase school autonomy through improved mechanisms for participation of directors, teachers and PA in the management of the schools. • Effects on: School failure, repetition, drop out • Use phasing in of program for evaluation

38. School Base Management in Mexico Gertler P. et al, 2007. “Empowering Parents to Improve Education: Evidence from Rural Mexico”

39. School Base Management in Mexico Gertler P. et al, 2007. “Empowering Parents to Improve Education: Evidence from Rural Mexico”

40. School Base Management in Mexico Gertler P. et al, 2007. “Empowering Parents to Improve Education: Evidence from Rural Mexico”

41. 3 Modalities on School Accountability 1. School-based Management 2. Information for accountability (IFA)projects 3. Teacher contracting and pay for performance projects

42. How could IFA improve outcomes? • Empower parents and communities vis a vis local & school authorities • Increased importance of education in the political sphere • Increased attention to results valued by parents • Enable better technocratic decision-making • Help guide allocation decisions • Under “strong accountability” systems: • Threat of sanction will spur increased teaching effort • When there is school choice: • “Market-pressure” for performance

43. Newspaper Campaign in Uganda • In 1995 only 13 % of non-wage recurrent spending reached schools. • From 1996-2001, amounts and dates of grant to local governments published in local and national newspapers.

44. Impact of information campaign IMPACT ON LEAKAGE: A school close to a newspaper outlet experienced 25 percentage points less leakage compared to a school one standard-deviation (30 km) further away from a newspaper outlet. IMPACT ON LEARNING: A one standard deviation reduction (30 km) in distance lead to a 0.09 standard deviation increase in average test scores. Source: Reinikka and Svensson (2001), Reinikka and Svensson (2003a)

45. Information Campaigns in India • Village-level education committees (VECs), representing parents and village leaders, created by state education policy • Presumption that information campaigns are needed to empower parents/communities • Two sets of experiments • PRATHAM --Banerjee et al. (2008) “Pitfalls of participatory programs: Evidence from a Randomized Evaluation in Education in India” • Three states in India –WB + Nike Foundation --Pandey et al. (2008) Community Participation in Public Schools: Impact of Information Campaigns in three Indian States.

46. Impact of Information Campaign (Banerjee et al.) • Program: • Advocacy and information about VECs =(1); • (1)+teaching how “learning reports cards” are done=(2); • (2)+ Teaching: Capacity Building to Directly Improve Learning (reading classes)=( 3) • Information campaigns ((1) and (2)) had no effect on VEC activism or on learning outcomes • Trained teaching volunteers (3) did have a significant effect on “reading” learning outcomes

47. Impact of Information Campaign (Pandey et al.) 47 Pandey, Priyanka et al. Community Participation in Public Schools: Impact of Information Campaigns in three Indian States. South Asia Human Development. Presentation. World Bank. 2008.

48. 3 Modalities on School Accountability 1. School-based Management 2. Information for accountability projects 3. Teacher contracting and pay for performance projects

49. How could TI improve outcomes? • Accreditation and merit-based incentives lead to better teaching professionals joining the system. • Decentralized hiring and monitoring of teacher performance leads to higher attendance rates. • Career advancement incentives stimulate sustained effort, improving student performance. • Pay-for-performance incentivizes improved quality and quantity of teaching improving student test scores.

50. Teacher Pay Incentives in AndraPradesh (Muralidharan and Sundararaman, 2008) • Three policies • Group and individual pay incentives based on averageimprovement in test scores of all students • Block grants to schools (extra teachers, more inputs) • Assessments conducted by an independent NGO • Results (two rounds) • Incentives increase student performance by 0.22 SD • 1styear both incentives work well. 2nd year individual incentives work better • Additional inputs improve scores only by 0.08 SD • Teachers liked the program