Example of an education intervention

1. Example of an education intervention Girls’ scholarship program

2. What helps improve learning? • Often small/no impacts on actual learning in education research • Inputs (textbooks, flipcharts) little impact on learning • De-worming affected attendance but not test scores • What is often most important in education policies and programs? Incentives • What happens if we offer direct incentives for student learning? • What happens if only offer this for a disadvantaged subgroup? Girls

3. Debate: cash incentives • The debate over cash incentives • “Pros” • Incentives to exert effort • Helps with self-control problems • Externalities to effort • Possible “cons” • Exacerbate inequality • Weaken intrinsic motivation (short or long run) • Gaming the system (cramming, cheating) • Merit awards could affect • Eligible students’ own effort • Other students effort & teacher effort could be either complements or substitutes

4. Research Design • The Girls Scholarship Program • Randomized evaluation in Kenyan primary schools • 63 treatment & 64 comparison schools • Balanced treatment groups • Announced an award for girls in treatment schools • Based on end of year standardized test scores • Top 15% of grade 6 girls in program schools win award • 1000 KSh (US$12.80) for winner and her family • 500 KSh (US$6.40) for school fees • Public recognition at an award ceremony • Two cohorts of scholarship winners, 2001 & 2002 • Survey data on attendance, study habits, attitudes

5. Program Implementation • Program implemented in two districts: Teso & Busia • Randomization and awards stratified by district • Historical and ethnic differences in the two districts • NGOs have poor relations with some Teso communities • Tragic lightning incident early 2001

6. Busia and Teso Districts

7. Differential attrition

8. Sample Attrition: Teso • School attrition: five Teso schools pulled out in 2001 • Test attrition: treatment vs. comparison with complete 2001 data: • Teso 54% vs. 66%, Busia 77% vs. 77% • Differential test attrition: significantly more high-achieving students took the 2001 exam in comparison schools relative to program schools, likely to bias estimated program impacts toward zero in Teso

9. Attrition Bias • How to deal with it? • Make sure that no one drops out from your original treatment and control groups. • If there is still attrition… • Check that it is not different in treatment and control. • Also check that it is not correlated with observables. • If there is differential attrition • Impute outcome variable based on baseline covariates • Bounds: run the analysis under the “best-case” and “worst-case” scenario. Either the best or the worst students are the ones that drop out at a rate that is equal to the rate of differential attrition

10. Year 1 (2001) Test Distribution

11. Estimation Strategy • Estimate total effects and district effects • Estimate effects for treatment schools (T):

13. By baseline test and gender

14. Across different Quartiles

15. Year 2

16. Student Attendance

17. Teacher attendance

18. Evidence on Mechanisms • Cheating is likely not a concern • Evidence of “learning”: consistent effects over two years and two cohorts • No effect on tutoring, household textbook purchases, self-esteem, attitudes toward school, amount of chores at home • Teachers report more parental support in Busia • Student and teacher attendance increased

19. Conclusions • Important to think through programmatic issues when designing interventions – incentives must be aligned (teachers, parents, students…) • Randomizing by school can help to pick up within class/school externalities • Things can go wrong – need to monitor attrition • Large and persistent gains in learning are possible to achieve

21. Teacher incentives • What if instead of linking student performance to students, we made the teachers responsible? • Randomized evaluation in Kenya (Glewwe, Ilias and Kremer (2004)) • Offered teachers prizes based on schools’ average scores • Top scoring schools and most improved schools (relative to baseline) • Each category 1st, 2nd, 3rd and 4th prizes were awarded (21% to 43 % of teacher monthly salary) • Penalized teachers for dropouts by assigning low scores to students who did not take the exam

22. Results • What was affected: • Treatment scores 0.14 sd above control • Strongest for geography, history, and religion (most memorization) • Exam participation rose • Extra-prep sessions • What was NOT affected: • Dropout/ repetition rates • Teacher attendance • Homework assignment or pedagogy • Lasting test score gains • Conclusions: • Teachers’ effort concentrated in improving short-run outcomes, rather than stimulating long-run learning

23. Conclusions • Busia: Overall (0.18 – 0.20 s.d.) • Persistent effect for girls the next year • Spillover effect for boys • Teso: Scholarship less successful: either no significant program effect or unreliable estimates • Merit-based scholarships can motivate students to exert effort • Test score and attendance gains among girls in the medium-run • Positive classroom externalities • Initially low-achieving girls, boys, and teachers • Possible multiple equilibria in classroom culture • Cost-effective way to boost test scores • Equity concerns – may wish to restrict to particular areas or populations

24. Difference between T&C