The evaluation of an early intervention policy in poor schools

1. The evaluation of an early intervention policy in poor schools By Roxana Maurizio, Irene Kit, Valeria Lahitte, Martin Scasso, Evelyn Vezza & Paula Inés Giovagnoli Team leader PEP-AusAID Policy Impact Evaluation Research Initiative June 10th, 2008

2. Outline_______________________________________________________________________________________________________________________Outline_______________________________________________________________________________________________________________________ 1. Argentina’s education system 1.a) Motivation 2. Research aim 3. The intervention 4. Data 5. Hypothesis to be tested 6. Empirical methods 7. Collaborators and agreements 8. Capacity building and final remarks

3. 1. Argentina’s education system_______________________________________________________________________________________________________________________ • Most of schools are public: free tuition and free entry • Mandatory education: primary (7 years) + secondary (5/6) Almost full coverage in primary school: non-attendance rates below 2% for 6-12 years of age (Census 2001). Learning results: low performance (ONE - National Standarized Tests and PISA - Program for International Student Assessment) Current Debate: How to improve school quality. • Overage. Retention policy, specially on first grades. • Next graph briefly describes enrolment situation

4. Enrolment by grade and population by age Argentina Source: Kit España and Labate. Based on Population Census and Relevamiento Anual de Matrícula y Cargos, 2001

5. Percentage of Overage in Public Schools by province

6. Percentage of repeaters in Public Schools by province

7. 1.a) Motivation _______________________________________________________________________________________________________________________ • Repetition: child repeats same grade to improve performance • Widely used policy in Argentina • As well as in many other developing countries: Latin American Countries (LAC) - 16.4%. UNESCO (2002) Wolff and Shiefelbein (2002) estimation: 22% • The estimated cost for LAC is 4.6 billion US dollars per year, implying an important resource allocation decision • It is the most powerful predictor of dropout (Rumberger, 1995)

8. 1.a) Motivation (cont.)_______________________________________________________________________________________________________________________ • Key question of whether repetition policy has a positive causal impact on subsequent school progression. • Empirical analysis is extremely scarce... • ...and focused on USA. Results are mixed (Holmes et. al., 1984; Jimerson, 2001) • Not controlled for endogeneity problems Exceptions: Eide & Showalter (2001) and Jacob & Lefgren (2004) • For LAC, we know virtually nothing, albeit its relevance Brazil: Gomez Neto and Hanusheck (1994) Uruguay: Manacorda (2006). Methodologically stronger

9. 1.a) Motivation (cont.)_______________________________________________________________________________________________________________________ • 29 schools • 22 schools MISIONES JUJUY • Since 2003: pilot program “ ” is being implementedin 96 primary schools • 217 sections • 122 sections • 6.380 students • 3.136 students • 211 teachers • 197 teachers • 46 principals • 44 principals TUCUMÁN CHACO • 23 schools • 22 schools • 211 sections • 197 sections • 5.414 students • 5.934 students • 122 teachers • 217 teachers • 44 principals • 58 principals

10. 1.a) Motivation (cont.)_______________________________________________________________________________________________________________________ • The program: • aims at improving the quality of learning in grades 1 to 3 • proposes a new way of teaching in the classroom • prevents repetition and instead provides “Promocion Asistida” (Supportive Promotion) • Designed to be applied in urban primary public schools

11. 1.a) Motivation (cont.)_______________________________________________________________________________________________________________________ • It is based on the belief that: • Poor educational outcomes are mainly related to teaching methods. • The first years of learning are critical for a successful path along the academic track. • It is financed by UNICEF jointly with Provincial Governments and implemented by the NGO “Educacion para Todos” • Only descriptive evaluation results are available

12. 2. Research aim _____________________________________________________________________________________________________________________ • To carry out a proper impact evaluation of this program We will investigate whether children under Supportive Promotion have better educational outcomes than pupils under current policies • In Corrientes province, the program will be implemented and scaled-up by 2011 • We will take advantage of the in-phase implementation design to draw the control group • Primary schools will be randomly divided into equally-sized groups. Each group will receive the treatment at different years • Contributions to educational policy decisions

13. 3. The intervention _______________________________________________________________________________________________________________________ • At school level (starting with first grade) • Treatment components: 1.To teachers: Training on how to develop key pedagogical sequences in Maths and Language to the whole classroom 2.To children with learning difficulties: specific pedagogical action plan 3. To principals: school institutional organisation changes to co-ordinate and to implement the new teaching way and SP

14. 3. The intervention _______________________________________________________________________________________________________________________ • Some clarifications: • The program does not change the curricula • It is a new way of instruction: cycle view - same classroom exposed to the same teacher across grade 1 to 3. • It takes account of materials and books recommended in the Learning Priorities Core (Nucleos de Apreidzajes Prioritarios) • Unlike Automatic Promotion, Supportive Promotion is not an administrative act • Direct beneficiaries: cohorts of students starting first grade attending public urban schools in Corrientes

15. 3. The intervention _______________________________________________________________________________________________________________________ • Discussion of potential pitfalls: • Timing of expansion • Hawthorne effects: Because of our external evaluation, actors might change their behaviour (i.e teachers will work harder) Parents may reallocate their children to a different school • John Henry effects: Control group may change its behaviour as well (substitutes available) • External validity: Do the results hold beyond Corrientes province? • Are the results replicable? • Other issues (i.e differential attritions in T and C groups)

16. 4. Data_______________________________________________________________________________________________________________________ • Administrative data for the universe under analysis provided by the Provincial Ministry of Education Geographic location, proxy to socioeconomic status of students, gender, size of the school and number of grade sections • Baseline and follow up surveys in treated and control schools. Data at individual level Main outcome: children’s learning (standardized evaluations to test literacy and numeracy skills) Additional variables including intermediate variables (such as pupils and teachers attendance rates) • Collection: Ideally during 3 (2) academic years (short and long term effects)

17. 5. Hypothesis to be tested _______________________________________________________________________________________________________________________ • Average effects Null hypothesis Imperfect compliance Impact by subgroup (girls/boys, school features ) • Distributional effects Are post-tests distributions (T-C) equal at the end of each academic year? • Heterogeneous effects Do weaker students benefit more from the program than better students?

18. 6. Empirical Methods _______________________________________________________________________________________________________________________ • Difference-in-difference specification • Inclusion of interactions between treatment dummy and subgroups • Value added specification • Boostrap test of equality of distributions (Benerjee et. al 2006) • Quantile treatment effects

19. 7. Collaborators and agreements_______________________________________________________________________________________________________________________ • UNICEF Argentina • National Ministry of Education and provincial Government of Corrientes • NGO: Educacion para Todos • Faculty of Economic Sciences and Statistics, National University of Rosario

20. 8. Capacity building and final remarks_______________________________________________________________________________________________________________________ • Specific knowledge that will be acquired by researchers directly or indirectly involved in the project • Interdisciplinary team: comprehensive approach Economists, an educational specialist, a social worker, a sociologist and a statistician • Coming from different institutions National universities: UNLP, UNS, UNR; NGOs, international organizations • Positive externalities for future projects Beyond the value of results from this specific evaluation, this work will give the possibility to build an unprecedented network among policy makers, researchers, academics and non-governments organizations