Impact Evaluation Methods: Randomization, IV, Regression Discontinuity

1. Africa Program for Education Impact Evaluation Impact Evaluation Methods: Randomization, IV, Regression Discontinuity Muna Meky Impact Evaluation Cluster, AFTRL Slides by Paul J. Gertler & Sebastian Martinez AFRICA IMPACT EVALUATION INITIATIVE, AFTRL

2. Measuring Impact • Randomized Experiments • Quasi-experiments • Randomized Promotion-Instrumental Variables • Regression Discontinuity • Difference in difference – panel data • Matching

3. Randomization • The “gold standard” in evaluating the effects of interventions • It allows us to form a “treatment” and “control” groups • identical characteristics • differ only by intervention • Closest approximation to counterfactual

4. Random Assignment • Each eligible unit has the same chance of receiving the intervention • Mimics chocolate experiment • Allows us to compare the “treatment” and “control group”

5. Random Assignment vs. Random Sample • Random Assignment • Are the observed results due to the intervention rather than other confounding factors? (internal validity) • Random Sample • Do the results found in the sample apply to the general population/are they generalizable? (external validity)

6. Randomization Random Sample (external validity) Randomization Randomization Random Assignment (internal validity)

7. Example of Randomization • What is the impact of providing free books to students on test scores? • Randomly assign a group of school children to either: - TreatmentGroup – receives free books - Control Group – does not receive free books

8. Randomization Random Assignment

9. How Do You Randomize? • At what level? • Individual • Group • School • Community • District

10. When would you use randomization? • Universe of eligible individuals typically larger than available resources at a single point in time • Fair and transparent way to assign benefits • Gives an equal chance to everyone in the sample • Good times to randomize: • Pilot programs • Programs with budget/capacity constraints • Phase in programs

11. Oportunidades Example • Randomized treatment/controls • Community level randomization • 320 treatment communities • 186 control communities • Pre-intervention characteristics well balanced

12. Oportunidades Example

13. Difference in Means

14. Impact Evaluation Example: Randomization vs. Other Methods

15. Other Analyses Often Lack Internal Validity • Enrolled vs non-enrolled • The baseline characteristics will be different because people have chosen which group they want to be in • Before and after • There are often other interventions going on at the same time

16. Measuring Impact • Randomized Experiments • Quasi-experiments • Randomized Promotion-Instrumental Variables • Regression Discontinuity • Difference in difference – panel data • Matching

17. When Do We Use Random Promotion? • Common scenario: • National Program with universal eligibility • Voluntary enrollment in program • Can not control who enrolls and who does not

18. Randomized Promotion • Possible solution: random promotion or incentives into the program • Information • Money • Other help/Incentives • Also called • Encouragement designs • Incentive schemes

19. Study Components • Intervention • Chocolate • Randomized Promotion • Encouragement to take chocolate

20. Example of Promotion Design for SATs • What is the impact of supplementary learning material on student test scores? • Outcome • Student test scores • Intervention • Supplementary learning materials • (all teachers can access these) • Randomized promotion • Letters encouraging students to use materials • (sent to randomly assigned teachers)

21. What Information Does Randomized Promotion Give Us? • How effective is the treatment? • We can analyze the effect the treatment had on the outcome in the sub-group of subjects who would not have used the intervention unless encouragement was present

22. How Effective is the Treatment? • Local Average Treatment Effect • Effect of the intervention on those who would not have enrolled unless encouraged

23. Encouragement Design

24. Example: Community Based School Management • What is the effect of decentralization of school management on learning outcomes? • All communities are eligible • Community management of hiring, budgeting, oversight • Need to write proposal

25. Community Based School Management • 1500 schools in the evaluation • Each community chooses whether to participate in program

26. Promotion Design • Community based school management • Provision of technical assistance and training by NGOs for submission of grant application • Random selection of communities with NGO support

27. Community Based School Management • Outcome – learning outcome • Intervention • decentralization of management to community • 1500 schools • Promotion • NGO support • schools randomized to receive this support

28. Examples – Randomized Promotion • Maternal Child Health Insurance in Argentina • Intensive information campaigns • Employment Program in Argentina • Transport voucher • Community Based School Management in Nepal • Assistance from NGO • Health Risk Funds in India • Assistance from Community Resource Teams

29. Randomized Promotion • Just an example of an Instrumental Variable • A variable correlated with treatment but nothing else (i.e. random promotion) • Again, we really just need to understand how the benefits are assigned • Don’t have to exclude anyone

30. Measuring Impact • Randomized Experiments • Quasi-experiments • Randomized Promotion-Instrumental Variables • Regression Discontinuity • Difference in difference – panel data • Matching

31. Introduction • What is the impact of monetary incentives on test scores to schools below some ranking? • Compare the schools right above the ranking point to schools below the cutoff point

32. Indexes are Common in Targeting of Social Programs • Anti-poverty programs • targeted to households below a given poverty index • Pension programs • targeted to population above a certain age • Scholarships • targeted to students with high scores on standardized test

33. Example: What is the Effect of Cash Transfer on Consumption? • Intervention: • Cash transfer to poor households • Evaluation: • Measure outcomes (i.e. consumption, school attendance rates) before and after transfer, comparing households just above and below the cut-off point.

34. Example: What is the Effect of Cash Transfer on Consumption? • Target poorest households for cash transfer • Method: • construct poverty index from 1 to 100 with pre-intervention characteristics • households with a score <=50 are poor • households with a score >50 are non-poor

35. Non-poor Poor

37. Treatment effect

38. Regression Discontinuity • When to use this method? • the beneficiaries/non-beneficiaries can be ordered • there is a cut-off point for eligibility. • cut-off determines the assignment of a potential beneficiary to the treatment or no-treatment

39. Regression Discontinuity Baseline – No treatment 2

40. Regression Discontinuity Treatment Period

41. Sharp and Fuzzy Discontinuity • Sharp discontinuity • the discontinuity precisely determines treatment • equivalent to random assignment in a neighborhood • e.g. social security payment depend directly and immediately on a person’s age • Fuzzy discontinuity • discontinuity is highly correlated with treatment . • use the assignment as an IV for program participation. • e.g. rules determine eligibility but there is a margin of administrative error.

42. Examples • Effect of class size on scholastic achievement (Angrist and Lavy, 1999) • Effect of transfers on labor supply (Lemieux and Milligan, 2005) • Effect of old age pensions on consumption -BONOSOL in Bolivia (Martinez, 2005)

43. PotentialDisadvantages of RD • We estimate the effect of the program around the cut-off point. • the effect is estimated at the discontinuity • fewer observations than in a randomized experiment with the same sample size • limits generalizability • Make sure the relationship between the assignment variable and the outcome variable is correctly modeled, including: • nonlinear relationships • interactions

44. Advantages of RD for Evaluation • RD allows one to estimate the effect of an intervention at the discontinuity • Can take advantage of a known rule for assigning the benefit that are common in the designs of social policy • No need to “exclude” a group of eligible households/individuals from treatment

45. Summary