Why Randomize?

1. Why Randomize?

2. Course Overview • What is evaluation? • Measuring impacts (outcomes, indicators) • Why randomize? • How to randomize? • Sampling and sample size • Threats and Analysis • Cost-Effectiveness Analysis • Project from Start to Finish

3. What is the most convincing argument you have heard against RCTs? • Too expensive • Takes too long • Unethical • Too difficult to design/implement • Not externally valid (Not generalizable) • Can tell us whether there is impact, and the magnitude of that impact, but not why or how (it is a black box)

4. Impact: What is it? • Positive • Negative • No impact • Don’t Know Intervention Primary Outcome Time

5. Impact: What is it? Intervention Impact Primary Outcome Counterfactual Time

6. Impact: What is it? • Positive • Negative • No impact • Don’t Know Counterfactual Intervention Primary Outcome Time

7. Impact: What is it? Counterfactual Impact Intervention Primary Outcome Time

8. Impact: What is it? Intervention Primary Outcome Impact Counterfactual Time

9. How to Measure Impact? • Impact is defined as a comparison between: • The outcome some time after the program has been introduced • The outcome at that same point in time had the program not been introduced • This is know as the “Counterfactual”

10. Counterfactual • The Counterfactual represents the state of the world that program participants would have experienced in the absence of the program (i.e. had they not participated in the program) • Problem: Counterfactual cannot be observed • Solution: We need to “mimic” or construct the counterfactual

11. Impact Evaluation Methods

12. Impact Evaluation Methods • Randomized Experiments Also known as: • Random Assignment Studies • Randomized Field Trials • Social Experiments • Randomized Controlled Trials (RCTs) • Randomized Controlled Experiments

13. Impact Evaluation Methods 2. Non- or Quasi-Experimental Methods • Pre-Post • Simple Difference • Differences-in-Differences • Multivariate Regression • Statistical Matching • Interrupted Time Series • Instrumental Variables • Regression Discontinuity

14. What is a randomized experiment?

15. The Basics • Start with simple case: • Take a sample of program applicants • Randomlyassign them to either: • TreatmentGroup– is offered treatment • Control Group- not allowed to receive treatment (during the evaluation period)

16. Key Advantage • Because members of the groups (treatment and control) do not differ systematically at the outset of the experiment, • Any difference that subsequently arises between them can be attributed to the program rather than to other factors.

17. Why randomize?

18. Example: Pratham’s Balsakhi Program

19. What was the Problem? • Many children in 3rd and 4th standard were not even at the 1st standard level of competency • Class sizes were large • Social distance between teacher and many of the students was large

20. Context and Partner • 124 Municipal Schools in Vadodara (Western India) • 2002 & 2003:Two academic years • ~ 17,000 children • “Every child in school and learning well” • Works with most states in India reaching millions of children

21. Proposed Solution • Hire local women (Balsakhis) • From the community • Train them to teach remedial competencies • Basic literacy, numeracy • Identify lowest performing 3rd and 4th standard students • Take these students out of class (2 hours/day) • Balsakhi teaches them basic competencies

22. Possible Outcomes Pros Cons Less qualified Teacher resentment Reduced interaction with higher-performing peers Increased gap in learning Reduced test scores for all kids • Reduced social distance • Reduced class size • Teaching at appropriate level • Improved learning for lower-performing students • Improved learning for higher-performers What is the Impact?

23. J-PAL Conducts a Test at the End • Balsakhi students score an average of 51% What can we conclude?

24. 1. Pre-post (Before vs. After) • Look at average changein test scores over the school year for the balsakhi children Average change in the outcome of interest before and after the programme

25. Method 1: Pre vs Post (Before vs. After) 26.42

26. What else can we do to estimate impact? Pre-Post • Limitations of the method • No comparison group, doesn’t take time trend into account

27. Method 2: Simple Difference Measure difference between program participants and non-participants after the program is completed Divide the population into two groups: One group not enrolled in Balsakhi program (Control) One group enrolled in Balsakhi program (Treatment) Compare test score of these two groups at the end of the program.

28. Method 2: Simple Difference -5.05 QUESTION: Under what conditions can the difference of -5.05 be interpreted as the impact of the Balsakhi program?

29. Method 3: Difference-in-difference Measure improvement (change) over time of participants relative to the improvement (change) over time of non-participants • Divide the population into two groups: • One group enrolled in Balsakhi program (Treatment) • One group not enrolled in Balsakhi program (Control) • Compare the change in test scores between Treatment and Control • i.e., difference in differences in test scores • Same thing: compare difference in test scores at post-test with difference in test scores at pretest

30. Method 3: Difference-in-difference

31. Method 3: What would have Happened without Balsakhi? Method 3: Difference-in-differences 75 50 25 0 26.42 2002 2003

32. Method 3: Difference-in-differences

33. Method 3: What would have Happened without Balsakhi? Method 3: Difference-in-differences 75 50 25 00 6.82 points? 26.42 19.60 2002 2003

34. Method 3: Difference-in-Differences • QUESTION: Under what conditions can 6.82 be interpreted as the impact of the balsakhi program? • Issues: • failure of “parallel trend assumption”, i.e. impact of time on both groups is not similar

35. Method 4: Regression Analysis • Divide the population into twogroups: • One group enrolled in Balsakhi program • One group not enrolled in Balsakhi program • Compare test score of these two groups at the start and at the endof the program. • Controlfor additional variables like gender, class-size • Post-test =

36. Method 4: Regression Analysis 1.92 QUESTION: Under what conditions can the coefficient of 1.92be interpreted as the impact of the Balsakhiprogram?

37. Impact of Balsakhi Program

38. Constructing the Counterfactual • Counterfactual is often constructed by selecting a group not affected by the program • Non-randomized: • Argue that a certain excluded group mimics the counterfactual. • Randomized: • Use random assignment of the program to create a control group which mimics the counterfactual.

39. Randomised Evaluations Individuals, villages, or districts are randomly selected to receive the treatment, while other villages serve as a comparison • Groups are Statistically Identical before the Program = Comparison Group Treatment Group Village 1 Village 2 • Any Difference at the Endline can be Attributed to the Program Two groups continue to be identical, except for treatment. Later, compare outcomes (health, test scores) between the two groups. Any differences between the groups can be attributed to the program.

40. Basic Set-up of a Randomized Evaluation Total Population Target Population Not in evaluation Evaluation Sample Treatment Group Random Assignment Control Group

41. Random Sampling and Random Assignment Randomly sample from area of interest

42. Random Sampling and Random Assignment Randomly sample from area of interest Randomly assign to treatment and control Randomly sample from both treatment and control

43. Randomization Design • Population = all schools in case villages • Target population: weakest students in all of these schools • Stratify on three criteria: • Pre-test scores • Gender • Language • Give 50% of them the Balsakhi program

44. Impact of Balsakhi - Summary *: Statistically significant at the 5% level

45. Which of these methods do you think is closest to the truth? • Pre-post • Simple difference • Difference-in-Difference • Regression • Don’t know *: Statistically significant at the 5% level

46. Impact of Balsakhi - Summary *: Statistically significant at the 5% level

47. Example #2 - Pratham’s Read India Program *: Statistically significant at the 5% level

48. Which of these methods do you think is closest to the truth? • Pre-post • Simple difference • Difference-in-Difference • Regression • Don’t know *: Statistically significant at the 5% level

49. Example #2 – Pratham’s Read India Program *: Statistically significant at the 5% level

50. Summary of Methods