A Performance-Based Evaluation Model for Rewarding Merit in Italian Schools

1. A Performance-Based Evaluation Model for Rewarding Merit in Italian Schools Donatella Poliandri, INVALSI Paola Muzzioli, INVALSI Isabella Quadrelli, INVALSI Sara Romiti, INVALSI Rome, 3-5 October 2012

2. Evaluation and Development of Quality in schools (VSQ) • VSQ is a pilot study carried out by the Italian Ministry of Education (MIUR), INVALSI and Sapienza University of Rome • Its objectives are: • to design a model for school evaluation; • to acknowledge excellence and promote support for school improvement. • The project integrates two perspectives: • Merit rewarding. Top performing schools receive financial rewards (competition mechanism). • School improvement.low performing schools are supported to start an improvement plan (equal opportunity approach).

3. Aimof this presentation

4. Thetheoreticalframework Sen’s concepts of merit and meritocracy, in Merit and Justice (2000)

5. Merit and Justice • According to Sen, we can scarcely dispense with rewarding actions which produce good results, at least for their incentive effect. • Does rewarding merit necessarily produce inequality? • As merit is a derivative and contingent concept, equality or equal opportunities could be the leading criteria to define merit. “The art of developing an incentive system lies in delineating the content of merit in such a way that it helps to generate valued consequences” (Sen 2000. p. 9)

6. Research questions • Starting from the structure of the VSQ project, based on economic incentives and improvement actions, we explored the following research questions: • Are students’ achievements at SAT sufficient to define the quality of a school? • Which criteria determine whether a school deserve a reward or not? In the light of the foregoing considerations, we have investigated the possible ways of defining a concept of quality for schools.

7. The operationalization of quality • A multi-criteria concept of quality

8. Measurement of students’ achievement • standardized instruments were used to assess students’ learning such as reading understanding, grammar and math tests administered by INVALSI • Results from the INVALSI tests were used to construct a measure of Value-added (developed by Ricci & Falzetti from INVALSI) . • Value-added models allows to compare schools from the same starting point, that is not considering the differences in students’ achievement due to individual, structural and contextual variables. In Value-added models the variables which are not under the direct control of schools are not considered. The Value-added represents the “measure of quality in terms of the extent to which the educational experience enhances the knowledge, abilities and skills of students” (Harvey and Green, 1993).

9. Evaluation of positive actions • Evaluation Teams visited each school and assessed the quality level in several areas Inclusion Enrichment & Remediation programmes School choice guidance Evaluation

10. Scoring Rubrics • The instrument used to evaluate schools’ positive actions is the Scoring rubric • A rubric contains evaluation criteria and a description of the good and less good performances related to different levels of quality (inadequate, acceptable, good, excellent). • Each quality criteria is articulated through a “multi-focus” perspective: that is the same quality criteria is assessed considering several dimensions • For each Scoring rubric it was provided a checklist that contains a list of indicators that were considered relevant and that needed to be observed for reaching a good level of quality

11. Multi-focus perspective • In the Rubric assessing Remediation programmes four dimensions are identified: • needs analysis: in this dimension are considered how schools collect information about, or monitor educational and training needs of students, in order to implement specific actions; • organizational structure: this dimension is related to organizational aspects of the intervention (e.g. Is there a teacher leader for remediation programmes? How students access remediation classes?); • teaching: refers to the specific teaching strategies and actions activated (e.g. grouping strategies, etc.) • satisfaction / effectiveness: considers students’ opinion about remediation classes and the impact of remediation classes attendance on students’ results.

12. Scoring Rubrics and Mean Rubric Score Inclusion of students with disabilities Rubric Score Inclusion Inclusion of foreign students Rubric Score Enrichment programmes Rubric Score Enrichment & Remediation Mean Rubric Score Remediation programmes Rubric Score School choice guidance School choice guidance Rubric Score Internal evaluation / Self-evaluation Rubric Score Evaluation Teachers’ educational planning and evaluation of students Rubric Score

13. The Final Score The final score – that defined each school’s position in the final ranking – was made up from the Value-added score and the Mean rubric score The weight of the Value-added score is 60%: - 35% for Italian - 25% for Math The higher weight given to Value-added in Italian arises from the relation between competences in Italian and competences in other subjects (e.g. the link between scientific processes comprehension and Italian literacy) The weight of the Mean rubric score is 40%

14. Results: quantitative and qualitative tools explore different aspects 77 primary/middle schools from 4 different provinces (Pavia, Mantova, Arezzo, and Siracusa) in Italy have participated to this study. Correlation between Rubric scores and Value-added in Italian and Maths confirm that both quantitative and qualitative instruments contributed to the evaluation of the educational process through the exploration of different aspects. Pearson correlation between Value-added, Mean rubric score and Final score **. Correlation is significant at the 0,01 level (2-tailed)

15. Scoring Rubrics’ validity In Scoring Rubrics there is a high correlation between the number of checks filled by evaluation Teams (that is the number of indicators observed by the Team) and the total quality reached by the school in a specific area. The higher the number of indicators included in the checklist, the higher the level of quality reached by a school in a specific area. Pearson correlation between Rubric score and percentage of checklist filling out

16. Distribution of schools by Value-added levels Almost 50% of schools have an intermediate level of Value-added both in Italian and Maths. This is consistent with the results of other research using Value-added in Italy The number of schools with lower levels of Value-added is higher than schools with higher ones.

17. Good and less good school performances (Rubric Score) Overall Mean rubric score is 2.63 with a standard deviation of 0.60, this confirm that most schools have received acceptable and good scores. The lowest performance was on the Rubric “Internal evaluation / self evaluation”: 30% of schools were assessed as inadequate, less than 10% obtained an excellent score. The best performance was on the Rubric “Inclusion of students with disabilities”: only 1% of schools received an inadequate score.

18. Correlation between Rubric scores and Value-added scores Correlations show a significant positive association between students’ evaluation practices and the value added of Italian and Math. There is a positive significant relation between Value-added scores in Math and effective Enrichment programmes There is a significant association between student evaluation practices and enrichment and remediation programmes. These results show the importance of a good student evaluation system in order to plan effective individualisation interventions (enrichment or remediation programmes). **. Correlation is significant at the 0,01 level (2-tailed). *. Correlation is significant at the 0,05 level (2-tailed).

19. VSQ: next steps • Regional ranking have been produced and 25% of schools placed in the top distribution of the ranking have received a money token • All participating schools have received an individual Evaluation report that highlights strengths and difficulties in the areas under investigation and give suggestion for improvement • Schools also have received financial support for implementing improvement plans: the amount of such support is higher for schools that were in the bottom distribution of the ranking • In 2013, participating schools will be evaluated again, using the same instruments, in order to assess the effect of the competitive mechanism and of the improvement plan on school performance, and to distribute the final part of the money incentive.

20. Thanks for your attention! Contacts: donatella.poliandri@invalsi.it