The art and science of Impact Evaluation

1. The art and science of Impact Evaluation

2. Some definitions ”analyses that measure the net change in outcomes for a particular group of people that can be attributed to a specific program using the best methodology available, feasible and appropriate to the evaluation question that is being investigated and to the specific context” 3Ie “is the systematic identification of the effects positive or negative, intended or not on individual households, institutions, and the environment caused by a given development activity such as a program or project” World Bank IEG

3. More… is a form of evaluation that assesses the net effect of a program by comparing program outcomes with an estimate of what would have happened in the absence of a program. US Environmental Protection Agency “Impact evaluations compare the outcomes of a program against a counterfactual that shows what would have happened to beneficiaries without the program. Unlike other forms of evaluation, they permit the attribution of observed changes in outcomes to the program being evaluated by following experimental and quasi-experimental design WORLD BANK DIME Initiative

4. Essentially…. Whether there is a change? Is it because of the project? Measurement Attribution

5. The issue is…. Confounding Influence Treatment Outcome

6. Evaluation Designs • Experimental • Quasi experimental • Non-experimental

7. Experimental Design Randomized control Trials Confounding Influence Treatment Outcome

8. Experimental Design Randomized control Trials • The “gold standard” in evaluating the effects of interventions • It allows us to form a “treatment” and “control” groups • Identical characteristics • differ only by intervention • Random assignment • Closest approximation to counterfactual Each eligible unit has the same chance of receiving the intervention • Allows us to compare the “treatment” and “control group”

9. Qausi-Experimental Design Difference in Difference • Matched Group

10. Qausi-Experimental Design Propensity Score Matching • Employs a predicted probability of group membership—e.g., treatment vs. control group--based on observed predictors, usually obtained from logistic regression to create a counterfactual group • Propensity scores may be used for matching or as covariates—alone or with other matching variables or covariates • Essentially creating a more matched sub-group using set of variables

11. Qausi-Experimental Design Regression discontinuity • The method is typically used in situations where it is not feasible to conduct an experiment between treatment and non-treatment groups. • Entails • Selection of the Cut Off Point/Threshold • Assigning Participants to Treatment/Control Groups • Assuming that the Outcome Variable is a Continuous and Smooth Function of the Assignment Variable (Especially Near the Threshold)

12. Qausi-Experimental Design Regression discontinuity

13. Non-Experimental Design Intervention interrupted time series Pre-test post-test

14. Participatory Methods PRA Tools • Spatial • Temporal • Relational • Decision-support • Evaluation wheel

15. Participatory Methods Outcome mapping • Developed by IDRC • Focus on institutional outcomes, behavior change • Monitoring and evaluation tool • Planning, Monitoring and evaluation tool

16. Participatory Methods Most-Significant change • Story-based tool • Monitoring without indicators • Developed by Rick Davies and Jessica Dart in 2005 • Identification, collection and analyses of stories of significant changes • Both for Monitoring and evaluation

17. Thank you!!