1 / 15

Different Methods of Impact Evaluation

Different Methods of Impact Evaluation. How to measure impact?. Assessing causality Impact of program = outcome 1 – outcome 2 In practice, we compare two groups, one of which benefited from the program, the other one did not. Event 2 (Effects) Outcome 1. Event 1

kevlyn
Download Presentation

Different Methods of Impact Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Different Methods of Impact Evaluation

  2. How to measure impact? • Assessing causality Impact of program = outcome 1 – outcome 2 In practice, we compare two groups, one of which benefited from the program, the other one did not Event 2 (Effects) Outcome 1 Event 1 e.g. Education program/ treatment Causes Event 2 occurs if and only if Event 1 occurred before Event 1 No Education Program Event 2 Outcome 2 Causes

  3. Constructing the counterfactual • The counterfactual • what would have happened in the absence of the program? … • … for the people who benefitted from the program: we don’t observe it • thus, impact evaluation will have to mimic it • Counterfactual is constructed by selecting a group not affected by the program – this is the main challenge of impact evaluation • Methods differ by the way the counterfactual is constructed

  4. Non-Experimental Methods • Differences between outcomes for treated and non-treated (in our ex: reached / not) are the sum of • Inherent differences: “SELECTION” BIAS • TREATMENT EFFECT: that we want to isolate from 1 • The essence of non-experimental methods is to find a way to redress the selection bias ex-post.

  5. Non-Experimental Methods • Simple Difference • Multivariate (Multiple) Regression • “Difference in difference” (Multiple Regression with Panel Data) • Matching • Randomization/RCT’s (advanced topic covered separately)

  6. Simple difference • Simple difference is a first measure – why is it not sufficient? • There may be differences between the two groups (age, location, gender, initial endowment, bargaining power) • So we may want to control for these differences

  7. Multi-variate regression • Suppose we can observe these differences. • Age composition of the group, initial infrastructure in the school, level of education, … • We can include all these variables in a regression: Y = a.T + b.Age + c.Infr + d.Edu +… • A regression provides the linear combination of observable variables that best “mimics” the outcome • Each coefficient represents the effect of each variable • Will give us the effect of the treatment everything else being equal, or more exactly every other observable characteristics being equal

  8. Multi-variate regression • Problems of the regression • You may want to include many many variables, to control for as many characteristics as possible • Problem of sample size (degrees of freedom) • More important: do you have measures of everything? • Bargaining power, Pro-activeness, Intrinsic motivation, hopelessness • There are unobservables

  9. Panel data • Simple difference: before/after • Counterfactual = same group before the program • Can we trust this? Assumption = would have remained the same • Ex: Police project • Double difference • Control for the situation before the program • Ex: Group 1 = Treatment Group; Group 2 = Control Group • 2006: Group 1 : 30 Group 2 : 60 2009: Group 1 : 50(+66%) Group 2 : 90 (+50%) Effect = +16% • Assumption: they would have grown at the same pace • Not sure…

  10. Matching • We compare pairs of 2 individuals for which the values taken by ALL variables are the same

  11. Matching • Variation: Propensity Score Matching: all the variables do not need to be exactly the same, but you look for individuals which have the same “profile” • Problems • This matching method requires a big dataset: find pairs on a sufficient number of variables • What about unobservables?

  12. An example • Case study: US Congress elections, 2002 • 60 000 phone calls to potential voters to encourage them to vote; 25 000 reached • Outcome: did they actually go vote? • 1st method: compare the 25 000 (reached) vs. the 35 000 (not reached) • 2nd method: introduce co-variates in a regression • 3rd method: introduce baseline data (vote in 1998) • 4th method: do a matching

  13. 1st comparison: we suspect a selection bias Selection Bias: differences in observable / unobservable characteristics → differences in outcome not due to the treatment 13

  14. Non-Experimental Methods

  15. For more details, please read Case Study: “Get out the vote? Do phone calls encourage voting” under References

More Related