Closing education gaps: lessons from impact evaluations

1. Closing education gaps: lessons from impact evaluations Deon Filmer Development Research Group The World Bank REGIONAL IMPACT EVALUATION WORKSHOP Evaluating the Impact of Development Programs: Turning Promises into Evidence Seoul, Korea December 6-10, 2010

2. Inequalities related to poverty: Grade 6 completion of 15-19 year olds in the richest and poorest quintiles. Within-country inequalities are as big—if not bigger—than cross-country inequalities All countries in database Source: Filmer (2010)

3. Inequalities related to poverty: Grade 6 completion of 15-19 year olds in the richest and poorest quintiles. Within-country inequalities are as big—if not bigger—than cross-country inequalities East Asia and Pacific, and Latin America and Caribbean Source: Filmer (2010)

4. Why are participation and completion low? • Costs of schooling • Poverty and cash/credit constraints • Opportunity costs • Exclusion • Benefits of schooling • Quality • Perceived returns

5. Addressing inequalities in school participation through demand-side programs • Reducing the costs of schooling: • Conditional Cash Transfer Programs • Scholarship programs  Increasing use of rigorously designed studies to evaluate impact.

6. Impact evaluation evidence from a CCT program: Mexico • Progresa/Oportunidades • Large scale program • Multisectoral • Cash to families on condition that • Children attend school regularly and maintain adequate progress • Children complete high school (for use by the child) • Preventive medical checkups for children • Attendance at health and nutrition lectures • Carefully designed for rigorous impact evaluation: • Phased in with a “Randomized Control Trial” design • Can compare outcomes in villages with the program to villages without the program

7. Impact evaluation evidence from a CCT program: Mexico • Small impact at primary grades • Baseline enrollment grades 0-5: 94% • Impact on enrollment: 1.9 • Large impact at transition to secondary school • Baseline enrollment grade 6: 45% • Impact on subsequent enrollment: 8.7 • Small impact at secondary level: • Baseline enrollment grades 7-9: 43% • Impact on subsequent enrollment: 0.6 Source: Schultz (2004)

8. Impact evaluation evidence from a Scholarship program: Cambodia • Cambodia Secondary School Scholarship Program • Targeted 100 lower secondary schools • Focused on education (and implemented through Ministry of Education) • Cash to families on condition that • Children attend school regularly and maintain adequate progress • Implementation design lends itself to rigorous impact evaluation: • Within all “feeder” Primary schools to program schools students were ranked by a “dropout-risk” score • At the level of each Secondary school applicants with • the highest dropout risk offered $60 scholarship (25 in large schools; 15 in small schools) • somewhat lower dropout risk offered $45 (25 in large schools; 15 in small schools) • and others offered no scholarship

9. Impact evaluation evidence from a Scholarship program: Cambodia Attendance as a function of ranking in terms of dropout risk score Impact estimate as a function of score at school-specific cutoff for scholarship receipt Estimate of impact Impact tends to be larger for the poorest recipients Large positive impact of program on attendance and enrollment Source: Filmer and Schady (2010)

10. Summary of CCT/Scholarship programs evaluated to-date • Impacts tend to … • Be larger when baseline enrollments are lower • Be larger at transition points • Be larger for poorer families Source: Fiszbein and Schady (2009)

11. Converting schoolingintolearning • This work has highlighted the importance of school quality as a mediating factor in converting increased enrollments into increased learning • Ecuador and Mexico: No increase in test scores among recipient students • Cambodia and Mexico: No increase in test scores for those who were offered a transfer (despite increased enrollment)

12. Inequalities in test score outcomes Inequality in TIMSS2007 Grade 8 Mathematicstest scores Source: Analysis of TIMSS 2007 database

13. New generation of impact evaluation work on what works to improve learning outcomes The State • Builds on the World Development Report 2004 Making Services Work for Poor People • Structured around the accountability relationships between actors Politicians Policymakers Citizens/Clients Providers Non-Poor Poor Frontline Organizations Voice / Politics Compact / Incentives Client Power Source: World Bank (2003)

14. A focus on three approaches to increasing accountability • Types of approaches that are being pursued or promoted through World Bank operations • Some evidence of success, or at least potential • School Based Management • The decentralization of school-level decision making to school-level agents. • Information For Accountability • The generation and dissemination of information about inputs, outputs, and/or outcomes. • Teacher Incentives • The linking of teacher pay to measures of effort and performance and/or the hiring of local teachers whose performance is evaluated by the school community

15. Leveraging accountability for results: Some examples of IEs

16. Other active areas of impact evaluation in the education sector at the World Bank • Interventions to promote Early Child Development • Home-based programs • Community Preschools • Formal preschools • Teacher training/upgrading • Incentives for relocation • Incentives for extra training for locally-hired teachers • Leveraging the private sector • Grants to private schools • Vouchers to students (which can be used at any school)

17. References • Filmer, Deon. 2010. “Education Attainment and Enrollment around the World: An International Database.” http://econ.worldbank.org/projects/edattain. • Filmer, Deon and Norbert Schady. 2010. “Does More Cash in Conditional Cash Transfer Programs Always Lead to Larger Impacts on School Attendance?” Journal of Development Economics. doi:10.1016/j.jdeveco.2010.05.006. • Fiszbein, Ariel and Norbert Schady. 2009. Conditional Cash Transfers: Reducing Present and Future Poverty. Washington, D.C., World Bank. • Schultz, T. Paul . 2004. “School Subsidies for the Poor: Evaluating the Mexican Progresa Poverty Program.” Journal of Development Economics. 74(1): 199-250. • World Bank. 2003. World Development Report 2004: Making Services Work for Poor People. Oxford University Press. • More information • The Development Impact Evaluation Initiative (DIME) http://go.worldbank.org/1F1W42VYV0 • Spanish Impact Evaluation Fund (SIEF) http://go.worldbank.org/Q2XYY39FW0