When are Impact Evaluations (IE) Appropriate and Feasible ?

1. When are Impact Evaluations (IE) Appropriate and Feasible? Michele Tarsilla, Ph.D. InterAction IE Workshop May 13, 2013

2. A Few Initial Remarks • IE representonly one of the different types of international developmentevaluation • IE definitionsvary (e.g., OECD-DAC and RandomistasMovement) and clarity of specificationsisoftenlacking • Disagreementwithin the developmentcommunity over the mostadequate design or method to beused in IE • IE sometimes are feasible and appropriate but their questions are not relevant

3. WHEN is IE appropriate? • When there is a real need for original and new information on a certain intervention’s effects Examples: - the extent to which an untested hypothesis holds (e.g. in case of an innovative or pilot projects); -which combination of activities or dosage of intervention is the most effective/contributing to the attainment of the envisaged impacts; • When the time elapsed between baseline and follow-up is sufficient (it does not always need to be 4-5 years) • Not appropriate when essential goods/services are denied to the comparison group; • Data Quality review ensured throughout data collection

4. WHEN are IE appropriate? • When the right sampling frame is in place and the target population has been adequately identified (IE often misses the effects of intervention on the most marginalized, invisible, mobile population) • When a communication strategy AND a dissemination strategy are in place (as IE strive to contribute to public knowledge, effects/results that will need to make sense for the general public -effect size vs. statistical significance) • When the intervention being evaluated is linear and not too complex or multi-level (recursive logic models or multiple causal pathway for each individual)

5. When are IE Appropriate? • When a good monitoring system is in place (monitoring data will allow opening the so-called black box – the how/why’s etc.) • When selection bias and other threats to validity have been adequately addressed • When triangulation is pursued in a systematic fashion in the course of data collection • When no major deviation from the implementation strategy is envisaged; No major adjustments are expected to take place throughout implementation. • When context and other environmental factors influencing the impact are qualified

6. When are IE Appropriate? • Some exceptions to this is the Familias en Accion in Colombia) • When there is sufficient time available for the findings (likely to be yielded by the evaluation) as to inform the decision that will need to be made • When there are rival plausible explanation for the results observed

7. When are IE Feasible? • When you have sufficient resources (money and technical expertise- better if independent); • When you have created sufficient support from local and national authorities; • When a good understanding of the evaluation rationale, timeline and practical implications among implementing partners is there (otherwise, the risk is that the control or comparison group will sabotage the activity) • When you have a baseline or a baseline could be reconstructed (e.g., by exploiting medical records/data collected by other donors in the target areas of interest)

8. When are IE Feasible? • When you have identified and tested for alternative explanations of the differences between the treatment and the comparison/control group • When a clear theory of change is available and agreed upon and understood by the program/project and the evaluation team; • When an evaluability assessment has been conducted • When RFPs for evaluators and implementers clearly call for joint planning/data collection and collaboration on evaluation-related activities; • When incentives for implementers and evaluators are aligned with each other • Not feasible when the intervention is universal and no untreated comparison/control group could be identified • Not feasible when randomization is not possible (e.g. infrastructure)

9. Thankyou! Contact info: • Michele Tarsilla • E-mail: mitarsi@hotmail.com