1 / 30

National Solidarity Programme Phase-II Impact Evaluation

National Solidarity Programme Phase-II Impact Evaluation. Interim Estimates from 1 st Follow-Up Survey. www.nsp-ie.org. Evaluation Team Andrew Beath (Harvard University) Fotini Christia (M.I.T.) Ruben Enikolopov (New Economic School, Moscow) Shahim Kabuli (World Bank)

afia
Download Presentation

National Solidarity Programme Phase-II Impact Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. National Solidarity Programme Phase-II Impact Evaluation Interim Estimates from 1st Follow-Up Survey www.nsp-ie.org

  2. Evaluation Team Andrew Beath (Harvard University) FotiniChristia (M.I.T.) Ruben Enikolopov (New Economic School, Moscow) Shahim Kabuli (World Bank) SakhiFrozish(World Bank) Partners Vulnerability Analysis Unit (MRRD) AfghanAid, CHA, IC, IRC, NPO/RRAA, Oxfam, PiN National Solidarity Program (NSP) Funding National Solidarity Program Food and Agriculture Organisation(FAO) World Bank TFs

  3. Background & Methodology

  4. National Solidarity Programme Executed by Ministry of Rural Rehabilitation and Development (MRRD) Implemented by 28 Facilitating Partners (local and int’l NGOs) Funded by Multilateral,Bilateral Donors Phase-I: 2003-7; Phase-II: 2007-11; Phase-III: 2010+ 2003 – 2010: Covered 22,500 communities at a cost of $929 million

  5. National Solidarity Programme Two Principal Village-Level Interventions: Create Gender-BalancedCommunity Development Councils (CDCs) through Secret Ballot, Universal Suffrage Election Fund Projects Selected by CDCs and Villagers and Managed by CDCs (Average Grant: $33,000; Max.: $60,000) Community Development Council (CDC) Projects 6 months 18-24 months

  6. What areas does NSP potentially impact? Social Cohesion & Conflict Political Attitudes Gender Community Development Council (CDC) Projects NSP Access to Services Local Governance Economic Activity

  7. The evaluation measures impact of NSP-II on . . . The evaluation estimates these impacts by . . . Social Cohesion & Conflict Political Attitudes Gender Access to Services Local Governance Economic Activity

  8. The evaluation uses this data to . . . The evaluation estimates these impacts by . . . collecting data over 3½ years in 500 villages: 250 NSP (treatment) & 250 non-NSP (control) Structure of Evaluation and Data Collection Aug. – Sep. 2007 May – Oct. 2009 Spring 2011 Treatment Villages (NSP) Sub-Projects Finished Create CDCs Select Sub-Projects Implement Sub-Projects 1st Follow-Up Survey Baseline Survey 2nd Follow-Up Survey Control Villages (Non-NSP) Similarity of Treatment and Control Villages Interim Estimates (18% of Projects Complete at Survey) Final Estimates

  9. Differences estimation is accurate because . . . The evaluation uses this data to . . . compare changes in treatment villages(NSP) with changes in control villages(non-NSP) Estimation of Impacts of NSP - Change in Treatment Villages 1st Follow-Up Survey Baseline Survey - Treatment Villages (NSP) = - Impact of NSP 1st Follow-Up Survey - Baseline Survey = = Change in Control Villages 1st Follow-Up Survey Control Villages (Non-NSP) Baseline Survey = If treatment and control villages are identical at baseline . . . Difference Estimates Difference-in-Difference Estimates

  10. Differences estimation is accurate because . . . 250 treatment villages selectedrandomly from 500 surveyed villages - other villages to control group study is a randomized control trial Partitioned Randomization: FPs denoted 15 villages to be excluded from randomization and evaluation Village A Village B Treatment Group (NSP) Randomization embedded in FP contracts Control Group (Non-NSP) Partitioned Randomization & Contractual Embedding Improved Chance of Successful Randomization Due to funding constraints and lack of village-level data, randomization was fairest way to decide which villages received NSP

  11. Interim Estimates of Program Impact • Local Governance • Access to Services • Economic Activity • Social Cohesion & Conflict • Gender • Political Attitudes

  12. Local Governance Creates Functional Councils which Assume Authority over Different Aspects of Local Governance Creates Avenues for Women’s Participation; Increases Responsiveness to Women’s Needs; Increases Satisfaction of Women with Village Leaders Increases Participation by Men in Local Governance; No Evidence of Impact on Men’s Satisfaction with Village Leadership

  13. Access to Services Increases Access to Safe Drinking Water Increases Access to Electricity Weak Impact on Transportation (at this stage) Mixed Evidence of Impact on Medical Care No Evidence of Impact on Boys’ Schooling (but Possible Increase in Girls’ Enrolment Rates)

  14. Economic Activity Improves Perceptions of Economic Situation and Optimism No Evidence (at this stage) of Impact on Household Income, Consumption, Assets or Borrowing Limited Impacts on Production

  15. Social Cohesion & Conflict Decreases Out-Migration of Households No Evidence of Impact on Attacks, but Improves Perceptions of Security No Evidence of Impact on Disputes & Feuds No Evidence of Impact on Trust among Villagers

  16. Gender Increases Involvement in Economic Activities, but Doesn’t Affect Involvement in Decision-Making Increases Inter-Village Connections and Availability of Support Groups Increases Respect for Women and Acceptance of Female Participation in Local Governance Decreases Incidence of Extreme Unhappiness among Women

  17. Political Attitudes Connects Villages with Central Government Improves Perceptions of Government & NGOs No Evidence of Impact on Acceptance of Government Jurisdiction, but Increases Support for Electing Local Leaders

  18. Local Governance Access to Services Economic Activity Social Cohesion Gender Political Attitudes Perceptions of Gov’t, Civil Society, and Military Male Villager Believes Official Works for the Benefit of All Villagers + 4% + 6% + 5% + 4% + 7% + 5% + 5% + 3%

  19. Sub-Treatment Interventions

  20. Sub-Treatment Interventions (STIs) Test of Two Different Types of CDC Elections NSP-II impact evaluation incorporates two STIs which test changes in two program components: STIs test different implementation strategies or changes in program design Community Development Council (CDC) Election Type Method of Sub-Project Selection STIs provide real-time evidence-based feedback on how to improve program effectiveness

  21. Test of Two Different Types of CDC Elections Effect of Different Types of CDC Elections 125 Villages Neighborhood Election Treatment Village A Treatment Village B 250 treatment villages randomly assigned to elect CDC either by neighborhood or village election Neighborhood Election: One male & one female from each neighborhood elected to CDC 250 Treatment Villages → Guarantees representation 125 Villages Village Election Village Election: Highest male & female vote-getters in entire village elected to CDC → Preferred candidates are elected

  22. Effect of Different Types of CDC Elections Test of Two Different Types of Project Selection Neighborhood Election 85% Village Election 60% . . . on Electoral Competitiveness • Village Elections (compared to Neighborhood Elections) increase electoral competitiveness

  23. Test of Two Different Types of Project Selection Impact of Different Types of Project Selection 125 Villages Consultation Meeting 250 treatment villages randomly assigned to select project either by consultation meeting or secret-ballot referendum Consultation Meeting: Villagers hold meeting and decide collectively which project is best 250 Treatment Villages → consensus-based 125 Villages Secret Ballot Referendum Secret Ballot Referendum: Villagers select projects through selecting preferred project → directly democratic

  24. Impact of Different Types of Project Selection Test of Election / Project Selection Interactions Baseline Survey asked male villagers, female villagers, and village leaders which village projects they preferred Preferences of female villages do not influence proposals, selection, or prioritization Obtained information from FPs on which projects were proposed, selected, and prioritized for implementation Preferences of male villagers influence selection and prioritization in both meetings and referenda . . . on Elite Capture of Project Selection Compared projects preferred by different groups with projects proposed, selected, and prioritized to find out who is influential in selection process Preferences of village leaders influence proposals, selection, and prioritization in meetings, but not referenda Preferences of village leaders more able to influence selection in consultation meetings (but influence of villagers in affecting selection is not affected) Compare influence of different groups under two selection processes to find out how selection type affects elite capture

  25. Test of Election / Project Selection Interactions Neighborhood Election & Consultation Meeting 62 Villages 125 Villages Neighborhood Election Randomization of Election Type 250 Treatment Villages Village Election 125 Villages Neighborhood Election & Referendum 63 Villages 250 Treatment Villages Randomization of Election Type and Project Selection Type Done Separately Village Election & Consultation Meeting Randomization of Project Selection Type 63 Villages Consultation Meeting 125 Villages 250 Treatment Villages Village Election & Referendum 125 Villages 62 Villages Secret Ballot Referendum Four Randomly-AssignedCombinations of Election and Selection Type Use to Test Interactions between Election and Selection Type

  26. Impact of Combinations on Selection Outcomes Villager leaders influence proposals, selection, and prioritization when village elections are combined with consultation meetings . . . on Elite Capture of Project Selection Male villagers influence selection and prioritization Combinations of election and referendum type do not affect influence of male villagers Probability of Elite Capture Maximized by Combination of Village Elections with Consultation Meetings

  27. Conclusion / Issues for Discussion

  28. Problems with Solutions CDD is interesting b/c it can change behaviors, attitudes, institutions, social cohesion etc. Very difficult to measure well → invest in extensive pilot-testing and consult widely ‘Parrot Bias’ in Surveys: Does program really change attitudes and behaviors or does it just responses to survey questions? → measure actual behaviors rather than simply asking questions Randomization: theoretically easy, practically difficult → tweak procedure to be resistant to pressures & communicate → successful quasi-experimental eval. better than failed RCT Prospective Evaluations take ages . . . → STIs can provide (quicker) real-time feedback to program → Manage expectations and communicate . . .

  29. Issues That Remain Independence of Researchers vs. Accountability of Evaluations to Programs, Implementers, and Donors Importance of Non-Results vs. Lack of Interest and Lack of Incentives for Researchers or Program to Disseminate Small proportion of programs are subjected to prospective IEs. How do we ensure the evaluated ones aren’t punished for non- or bad results? Who will synthesize IE results for policy-makers and make sure the results are used in policy decisions and program design? Who will coordinateresearchers to make sure evaluations address questions useful for policy decisions and program design?

  30. www.nsp-ie.org

More Related