Improving Education through Accountability and Evaluation Lessons

1. Planning (and working on) school improvementPreliminary evidence from the Quality and Merit Project in ItalyCaputo A., Mori S., Rastelli V. Improving Education through Accountability and Evaluation Lessons from Around the WorldOctober 3-5, 2012 - Rome

2. Summary PQM Project:Training on diagnostic tools Training on improvement interventions Preliminary evidence on PQM effectiveness on student achievement Aim of the studyAnalysis of School Improvement Plans characteristics/contents and their association with school variables Results and Conclusions

3. PQM Project The context Lower secondary school in the Southern Italian regions having access to the European Union funds for low income EU regions (Campania, Sicily, Calabria and Apulia) In-serviceteacher training The teachers participating in the project are part of a network of schools coordinated by a tutor, who gives them both formal and online polyvalent training, all along the school year The goals Help teachers to set up a school Improvement Plan Provide didactic material and alternative teaching solutions

4. TEACHER TRAINING PQM Project The process Student achievement assessment at the beginning of the school year (INVALSI pre-test) Analysis and diagnosis (expert school teachers) Improvement interventions (expert school teachers) Student achievement assessment at the end of the school year (INVALSI post-test)

5. The core of the project Training on diagnostic tools Drafting a School Improvement Plan Training on improvement interventions Planning didactic activities remedial and extra education for students producing new didactic materials - teacher peer-to-peer laboratory sessions

6. Preliminary evidence Pre-post analysis on math achievement was carried out on over 13.000 students (coming from 248 schools) participating in PQM activities in both the 2009/2010 (6th grade) and 2010/2011 (7th grade) school years. Regardless some limitations (need of anchoring of the tests, need of repeating the analyses on control group, unavailability of more waves of data), some evidence exists on the fact that PQM students are improving. This difference remains significant also controlling for regions, socio-economic status (SES), participation and treatment intensity (number of didactic units, school and class size, percentage of PQM students and classes out the total of the school)

7. . Pre-post analysis reveals an increase in PQM student math scores (p. < 0.01). On average students get 4 percentage points in correct answers from 2009/2010 to 2010/2011 school year.

8. Aim of the study Explore characteristics/contents of School Improvement Plans Assess the association between school planning strategies and some variable at school level, i.e. geographical area, student socio-economic status and student math achievement improvement

9. School Improvement Plan 1) analysis of the educational context. It should be addressed to both school and class levels 2) diagnosis of student needs, based on the INVALSI assessment of student achievement deficits 3) detection of improvement goals, i.e. remedy/empowerment of student education, teacher professional development and parental involving in school activities

10. Method Computer-aided Text Analysis on 248 School Improvement Plans by each section: analysis of the educational context diagnosis of student needs detection of improvement goals Lexicometricanalysis Formalcharacteristicsof the texts in termsoflexicalrichness, variety, originality, levelofdetail ThematicAnalysis Thematicdomains and latentdimensions

11. Results Table 2 - Lexicometric Indexes of School Improvement Plans Sections Calabria is the Region whose School Improvement Plans are generally characterized by greater richness and detail, whilst in Campania they tend to be slightly more stereotyped. SES is not significantly associated to any measure. The Improvement Plans of schools with very high student achievement improvement provide a more accurate analysis of the context and a greater originality and specificity of improvement goals The Improvement Plans of schools with very low student achievement improvement show a more precise and articulated diagnosis of student needs.

12. F1 - School responsibility for student education Low achievement improvement External responsibility F2 - School self-efficacy High SES High self-efficacy Figure 1 - Analysis Of The Context. Factorial Space

13. F1 - Utilization of standardized test Low/very low achievement improvement Multi-focused approach F2 - Integration between internal and external assessment Low/very low achievement improvement Low/very low SES School self-assessment Figure 2 – Diagnosis Of Student Needs. Factorial Space

14. F1 – Specificity of planned goals Very high achievement improvement Specific goals F2 - Originality in School Improvement Plan elaboration Very low achievement improvement Stereotypical elaboration Figure 3 - Detection Of Improvement Goals. Factorial Space

15. The key-factors which seem to promote better student achievements concern the school capabilities of: • Carrying out a careful analysis of the educational context • Prioritizing the various elements already in the diagnostic part of the process • Detecting specific and detailed improvement goals Conclusions

16. The main obstacles to school improvement: • Tendency to attribute responsibility for student education to external socio-educational agencies • Exclusive focus on school self-assessment to detect student needs • Diagnosis of student learning deficits not based on cognitive processes or specific subject areas to enhance • Poor autonomy and originality in setting up improvement activities Conclusions

17. City, Elizabeth, et al. Instructional Rounds in Education: A Network Approach to ImprovingTeaching and Learning. Cambridge, Massachusetts: Harvard Education Press, 2009. Creemers, Bert, and Leonidas Kyriakides. The dynamics of educational effectiveness: A contribution to policy, practice and theory in contemporary schools. London: Routledge, 2008. Creemers, Bert, and Leonidas Kyriakides. “Explaining stability and changes in school effectiveness by looking at changes in the functioning of school factors”, School Effectiveness and School Improvement: An International Journal of Research, Policy and Practice 21 no.4 (2010): 409-427. Creemers, Bert, and Leonidas Kyriakides. Improving Quality in Education: Dynamic Approaches to School Improvement. London: Routledge, 2012. Fullan, Michael. Leadership and sustainability: System thinkers in action. Thousand Oaks, CA: Corwin, Toronto, Canada, Ontario Principals Council, 2005. INVALSI Istituto nazionale per la valutazione del sistema educativo di istruzione e di formazione, Rilevazioni degli apprendimenti. Scuola primaria. Prime analisi. INVALSI, Rome, 2009. INVALSI Istituto nazionale per la valutazione del sistema educativo di istruzione e di formazione, Le prove del Servizio Nazionale di Valutazione 2008-2009 - Analisi Tecnica. INVALSI, Rome, 2010. OECD Organization for Economic Co-operation and Development, Education at a Glance 2010 Revised edn. OECD, 2010. References

18. THANKS FOR YOUR ATTENTION