1 / 12

Doing Impact Evaluations in Argentina and Chile:

Doing Impact Evaluations in Argentina and Chile:. Pragmatism, Opportunities and Partnership. Country Context . Argentina Weaknesses in Survey Data (and worsening) Highly Politicized, Particularly Relations between Central and Lower Levels of Government Lack of institutional capacity Chile

calandra
Download Presentation

Doing Impact Evaluations in Argentina and Chile:

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Doing Impact Evaluations in Argentina and Chile: Pragmatism, Opportunities and Partnership

  2. Country Context Argentina • Weaknesses in Survey Data (and worsening) • Highly Politicized, Particularly Relations between Central and Lower Levels of Government • Lack of institutional capacity Chile • Evaluation Culture Led by Ministry of Economy • Requirement for evaluation in Program law

  3. Impact Evaluations 1997-2006 Argentina • Trabajar (workfare) Program • Workfare Participation and Exit • Private Employment Subsidy • Heads of Household Project • Grant-financed Productive Activities • Future (?) Lifelong Learning and Training and Employment Services Chile • Chile Solidario

  4. Motivations • Serious link with possible follow-up operation (Trabajar) – role of Bank management • Risk mitigation measure (Heads of Household/grants for productive activities) • Government/Bank shared technical interest (private employment subsidy, study of workfare leavers, Chile Solidario)

  5. Pragmatic Design • Used existing surveys (Social Survey, Labor Force Survey, CASEN) • Use of Administrative Data • Partnered with local institutions (Ministry of Labor, SIEMPRO, INDEC, MIDEPLAN) • Partnered with DEC staff (Bank supervision budget and other resources) • Government financed out of loan funds • Cost-effective impact evaluations

  6. Pragmatic Design continued • Argentina productive projects: 3rd best design, rigorously implemented • Chile: Bank came in after program started, based on how program was implemented, recovery from last minute change on sample • Don’t give in on key items – sample size, innovation (psycho-social questions), quality of technical advice on evaluation • Random Assignment sometimes possible

  7. Timing Critical • Trabajar: Board approval (6/97) Prel. Evaluation results (5/98) • HoH: Program start (4/2002) Prel. Evaluation results (7/2003) Chile Solidario: Information for New Government

  8. Devil is in the Details • Be ready for close supervision and lots of nitty-gritty work • Carefully monitor implementation (samples, field work, questionnaires) -even then things will go wrong • Data quality issues • Be ready to explain design and methodology many times • Not a one-shot deal

  9. Pay-off of Sustained Partnership • Building of Relationships (INDEC, DIPRES- Budget Office in Chile) • Building of Relationships with Bank staff (DEC) • Building of Understanding of Methodologies (Ministry of Labor, DIPRES) • Aiming to make it “second nature”

  10. Link with Operations of Programs • What are the key questions? • How is program being implemented? • What administrative data is available? • Knowledge/cooperation of operational staff • Important for following up on results • Add value to evaluation – heterogeneity of impacts • Best partners for impact evaluation are program operators. They do not want everything to focus on methodology.

  11. Use of Results • Trabajar – Justified follow-on operation • Head of Household – Program credibility, provided information on implementation • Chile Solidario – too soon to tell • “Public goods” contribution • Spill-over effects (use of data by others)

  12. …but, Not a Magic Bullet • Decisions on programs based on many factors, not just their performance • Not everyone convinced by evidence-based data vs. anecdotes or ideology or initial views • Need for better dissemination, particularly in-country, and making results understandable • Bank value-added, including for middle-income countries • We need to work more broadly on sustainability issues

More Related