Improving Education through Accountability and Evaluation: Lessons from Around the World

1. Improving Education through Accountability and Evaluation: Lessons from Around the World October 4th, 2012, Rome Escola de Ciências Sociais e Humanas Evaluating the “ProgramaMaisSucesso Escolar”: Lessons learned from evaluating the impact of a Portuguese national educational policy for compulsory education. M. Clara Barata (Maria.clara.barata@iscte.pt) & M. Manuela Calheiros, Joana N. Patrício, João Graça & Luísa Lima Instituto Universitário de Lisboa (ISCTE-IUL), CIS-IUL, Lisboa, Portugal

2. EvaluatingthePortuguese PMSE The team at ISCTE-IUL

3. EvaluatingthePortuguese PMSE

4. Barata, M. C., Calheiros, M. M., Patrício, J., Graça, J., & Lima, M. L. (2012). Avaliação do Programa Mais Sucesso Escolar. Lisboa: Direção-Geral de Estatísticas de Educação e Ciência – Ministério da Educação e Ciência. Disponível em http://www.gepe.min-edu.pt/np4/?newsId=670&fileName=PMSE_Alt_PDF.pdf. EvaluatingthePortuguese PMSE

5. EvaluatingthePortuguese PMSE Roadmap • The challenge of building and interpreting scientific evidence • PMSE and the application process • Prior evidence and background • Sample and measures • Data-analytic strategy • Results • Discussion

6. EvaluatingthePortuguese PMSE Challenge • Buildingandinterpretingscientificevidenceofprogramimpact • Previousstudies: • observational, focusedonoutliers (qualitative), no generalizationpossible, no controlgroup; • Evidencenotused for policydecisions. • Ourstudy: • Full sample, quantitative, diversityofindicators, empirically-matchedcontrolgroup to maximize causal inferences; • Convertedestimatesintoeffectsizes to helpwithinterpretation.

7. EvaluatingthePortuguese PMSE The PMSE Policy • Portugal has one of the highest retention rates in compulsory school among OECD countries (OECD, 2011) • PISA: within OECD average on reading, and progressing in mathematics and science (OECD, 2010); • “Culture of grade retention”. • In 2005, the ME legislated that teachers create an individualized plan of recuperation, supervision and development for all students failing their grade • In 2009, the ME launched PMSE, a school-based four-year initiative to reduce retention and maximize completion ratesin compulsory education.

8. EvaluatingthePortuguese PMSE The application process • Individual schoolshad to apply: Novelty! • detailed plan of strategies to improve the outcomes of students likely to be retained; • commit to lowering retention rates by one third each year, for four years. • In exchange ME provided: • additional teaching time for organizational measures; • autonomy (OECD 2007); • technical and academic experts to advise on aspects of implementation. • 375 schools applied: 123 intervention. • ME selected plans with coherence, intentionality and consistency (interviews). • After being accepted into PMSE, all schools were directed to choose up to four courses and two grade levels in which to invest their credits (nested data within schools).

9. EvaluatingthePortuguese PMSE Class size, class composition and differentiated instruction Class size reduction is one of the most common policies used to address low academic achievement internationally (e.g. Hoxby 2000; Dobbelsteen, Levin and Oosterbeek 2002) • Costlyinvestment(Hoxby2000); hiring of new teachers. • Mixed evidence? • Null effects? (Hanushek 1999); • STAR: increasedperformance on standardized tests; larger effect for minority students and students receiving free lunch (Finn and Achilles 1999; Hanushek 1999; Krueger 1999); • Natural class size reductions of 30 to 15 students did not have a significant impact on student performance in state tests (Hoxby1998, 2000) • teachers were not equipped to deal with natural variations in class size (same instructional strategies; Hoxby, 2000)

10. EvaluatingthePortuguese PMSE Class size, class composition and differentiated instruction Class size reduction involves class composition • achievement level, gender, and race/ethnicity (Hoxby 1998) • maximize or minimize homogeneity in the classroom. • when accounting for class composition, students in smaller (more homogeneous) classes did perform better than students in larger classes (Dobbelsteen, Levin, & Oosterbeek 2002). Differentiated instruction • changes to teaching and learning routines in order to address the academic diversity usually found in classrooms (Heacox 2006; Morgado 2005; Vygotsky 1978)

11. EvaluatingthePortuguese PMSE Class size, class composition and differentiated instruction In PMSE: • Divided students into smaller units; • Used prior achievement level to create more homogeneous groups of students; • Offered professional teacher development in differentiated instruction strategies to take advantage of these class size and class composition changes to help maximize those positive results.

12. EvaluatingthePortuguese PMSE Research Question • What is the impact of the first two years of PMSE on school success, success in high-stakes exams at the school and student level, school completion, and alternative education paths?

13. EvaluatingthePortuguese PMSE Sample Compare PMSE schools to schoolsthatapplied for theprogrambutdidnotgetit • 123 intervention schools and 252 control schools in the first year (2009/2010), and on 115 intervention schools and 248 control schools in the second year (2010/2011); • Intervention and control schools were significantly different on 7 out of 23 school and school district indicators in the first year; and 10 out of 23 indicators in the second year. • At the district level, geographic distribution, and urban-rural qualification; • At the school level, inclusion in large administrative units, including in the same building preschool and/or secondary school; total number of students, total number of faculty, percentage of female faculty, average age of faculty, total number of staff, average years of teaching experience of faculty, number of faculty with masters or higher; and • At the student level, percentage of students receiving government support (SASE A, SASE B and SASE C), percentage of students with computer at home, percentage of students with internet at home, and number of school violence events

14. EvaluatingthePortuguese PMSE Measures School success (also high performance). • percentage of grade-level transitions; • percentage success in Mathematics, Portuguese, and English • percentage of fours and fives in Mathematics, Portuguese, and English Success in high-stakes exams, school level. • percentage of exams per school • percentage of success in the Mathematics and Portuguese exam • percentage of fours and fives in the Mathematics and Portuguese exam • percentage of success in Mathematics and Portuguese sixth-grade exam • percentage of fours and fives in the Mathematics and Portuguese exam Success in high-stakes exams, student level. • performance in the Mathematics and Portuguese ninth-grade exam, per student • difference between school performance and high-stakes performance in Mathematics and Portuguese

15. EvaluatingthePortuguese PMSE Measures School completion. • cohort survival rates on the 2nd and 3rd cycle • average age of graduation at the end of each cycle Alternative education paths. • percentage of enrollments in alternative education and training courses on the 2nd and 3rdcycle School e school district demographics for year of application

16. EvaluatingthePortuguese PMSE Data-analytic strategy • MLM modelswith random effects for schools, adjusted according to the contributions of each grade level and course in the program. • Propensity Scores Estimation (PSE) to reduce the differences in educational achievement between intervention and control groups that are not attributable to PMSE (Rosenbaum and Rubin 1983; Diaz and Handa2006) • inverse propensity score weighting; pscore in stata12 • Adjusted all estimates from the multivariate and multilevel impact models for school and school district characteristics, as well as baseline school averages • Example: individual scores in theexams • Effectsizes(Gormley, Gayer, Phillips and Dawson 2005; Wong, Cook, Barnett and Jung 2008).

17. EvaluatingthePortuguese PMSE Schoolsuccess

18. EvaluatingthePortuguese PMSE High-performance schoolsuccess

19. EvaluatingthePortuguese PMSE Success in high-stakes exams, school level. Smaller samples • No effects on: • percentage of exams per school • percentage of success in the Mathematics (Y1) and Portuguese exam • percentage of fours and fives in the Mathematics and Portuguese exam • percentage of success in Portuguese sixth-grade exam • percentage of fours and fives in the Mathematics and Portuguese exam • Except: • percentage of success in the Mathematics (Y2)

20. EvaluatingthePortuguese PMSE Success in high-stakes exams, student level.

21. EvaluatingthePortuguese PMSE School completion • No effectson: • cohort survival rates on the 2nd and 3rd cycle (Y2) • average age of graduation at the end of each cycle • Except: • cohort survival rates on the 2nd and 3rd cycle (Y1) No effects on recourse to Alternative education paths.

22. EvaluatingthePortuguese PMSE Summary of Findings • PMSE schools had higher percentages of transitions and higher success in Mathematics, Portuguese and English in the first and second years of the program, with effect sizes varying between -0.17 and 0.59. • Main goal of PMSE to reduce student retention was achieved, plus PMSE had an impact on school success. • A non-significant to negative impact on the high-stakes Mathematics and Portuguese exams.

23. EvaluatingthePortuguese PMSE Discussion • PMSE schools seem to be moving a larger number of students with lower achievement through compulsory school, and getting them to final exams. • Undesirable consequence of lowering PMSE schools’ performance on exams • PMSE schools vulnerable to accountability policies based solely on standardized national evaluations, as well as attacks by the media based on national school rankings

24. EvaluatingthePortuguese PMSE Implications • PMSE was successful and should be maintained for 4 years • Examine the full impact of PMSE after four years of implementation • Need new indicators of school efficiency, which combine results on external evaluations with other indicators, such as cohort survival rates • Invest in discussion of what constitutes appropriate evidence of the true impact of an educational program in students’ achievement • Design randomized trials that will improve causal inferences about current and future policies.

25. Improving Education through Accountability and Evaluation: Lessons from Around the World October 4th, 2012, Rome Escola de Ciências Sociais e Humanas Thankyou! Email me: Maria.clara.barata@iscte.pt