WHY RANDOMIZE

1. WHY RANDOMIZE Presented by: Raul Sanchez de la Sierra March 2014

2. Why randomize ?

3. Because correlation is not causation

4. Example: Impact of a medicine Does not take a drug Does the medicine increase weight?? Takes the drug

5. Overview • Causal impact and counterfactual • Comparison groups • Randomization

6. CAUSAL IMPACT andcounterfactual

7. Causal impact of a policy • Change due to the policy

8. Causal impact of a policy What really happened What would have happened without the policy Real world Imaginary world Pay for performance Traditional wage Counterfactual

9. Counterfactual Counterfactual: Would would have happened (imaginary) Problem: Will never observe what would have happened How to measure the causal impact of a policy?

10. Causal impact of a policy What really happened What would have happened without the policy Real world Imaginary world Pay for performance Traditional wage Similar to Counterfactual Counterfactual

11. Comparison Groups

12. Selecting the comparison group • Select a group that is exactly likethe group of participants in all ways except the policy Good Bad

13. Pratham’s Balsakhi program

14. The intervention • Work with Pratham in 124 Municipal Schools • Hire local women (Balsakhis) from the community • Train them to teach basic literacy, numeracy • Identify lowest performing students • Balsakhi teaches them basic competencies

15. Possible comparisons • Compare after to before the policy • Compare policy group to other group • Compare change in policy group to change in other group • Other non-experimental methods • Randomized Experiment

16. Method 1: Compare after to before the policy • Test scores after Balsakhi • Test scores before Balsakhi

17. Method 1: Compare after to before the policy

18. Method 1: Compare after to before the policy Improvement could be due to other reasons related to TIME • Problem with this comparison:

19. Why this may not be a good strategy What would have happened without the policy What really happened True counterfactual After Before Wrong counterfactual Pay for performance Traditional wage

20. Method 2: Compare to other group Compare test scores of… With test scores of… Children who did not get balsakhi Children who got balsakhi

21. Method 2: Compare to other group

22. Method 2: Compare to other group Balsakhi students may be different: Examples: - Poorer - Worst performing • Problem with this comparison:

23. Why this may not be a good strategy What would have happened without the policy What happened to the other group What really happened Wrong counterfactual After True counterfactual Before Pay for performance Counterfactual Comparison group

24. Method 3: Difference-in-Differences Children who got balsakhi Compare gains in test scores of… With gains in test scores of… Children who did not get balsakhi

25. Method 3: Difference-in-Differences

26. Method 3: Difference-in-Differences

27. Method 3: Difference-in-Differences

28. Method 3: Difference-in-Differences Balsakhi students may IMPROVE systematically faster for other reasons than the program: Examples: • They may start from a lower level • Problem with this comparison:

29. Why this may not be a good strategy What would have happened without the policy What happened to the other group What really happened After Before Pay for performance Counterfactual Comparison group

30. Method 4: Regression Analysis • Compare test scores at the startand at the endof the program. • For students of the same gender, age (CONTROL) Children who got balsakhi With Children who did not get balsakhi

31. Method 4: Regression Analysis Balsakhi students may IMPROVE systematically faster than other students of their own gender and age for other reasons than the program: Examples: • They may start from a lower level • Problem with this comparison:

32. Why this may not be a good strategy What would have happened without the policy What happened to the other group What really happened After Before Pay for performance Counterfactual Comparison group

33. Method 4: Regression Analysis • Fundamental problem: • Cannot know what makes Balsakhi student different before the program • Never fully credible

34. Problem with non experiments • SELECTION BIAS

35. Non-experimental Methods • Instrumental Variables • Regression Discontinuity

36. Randomization

37. Non-random assignment HQ Monthly income, per capita 1000 500 0 1457 947 Treatment Control

38. Randomassignment HQ Monthly income, per capita 1000 500 0 1242 1257 Treatment Control

39. How does randomization work? 3. INTERVENTION 1. Baseline 4. Endline 2. RANDOMIZED ASSIGNMENT 3. STATUS QUO TARGET POPULATION

40. Randomization at work: education April 06/07 Tests Tests Aug 05 Initial Test Pay for performance Fixed wage Jun 05

41. One of the two then gets the treatment. Outcomes are then compared between those that did and did not get treatment Randomization at work: medicine In a randomized trial, two groups are formed through a lottery(to make sure that they are comparable) In a randomized trial, two groups are formed through a lottery(to make sure that they are comparable) One of the two then gets the treatment. Outcomes are then compared between those that did and did not get treatment This groups does not This group gets the treatment How they would have looked without treatment How the treated group looks now

42. T T T T T C T T C C C C C C T T C C If project goes to easy places The truth T T C T C C C C T T T C If project goes to hard places If lotteries are used

43. Key advantage of experiments Members of the groups are statistically identical any change can beattributedto the program 43

44. Method 5: Randomized Experiment • Suppose we evaluated the balsakhi program using a randomized experiment

45. Impact of Balsakhi - Summary *: Statistically significant at the 5% level

46. Impact of Balsakhi - Summary *: Statistically significant at the 5% level

48. What is the impact of this program? Program starts Impact Primary Outcome Counterfactual Time

49. Impact: What is it? Program starts Counterfactual Impact Primary Outcome Time

50. Impact: What is it? Program starts Primary Outcome Impact Counterfactual Time