The literacy divide : territorial differences in the Italian education system

1. Statistical Methods for the analysis of large data-sets University G. d'Annunzio - Chieti-Pescara September 23, 2009 – September 25, 2009 The literacy divide: territorial differences in the Italian education system Claudio QUINTANO, Rosalia CASTELLANO, Sergio LONGOBARDI University of Naples “Parthenope” claudio.quintano@uniparthenope.it lia.castellano@uniparthenope.it sergio.longobardi@uniparthenope.it

2. Overview Italian data from the last PISA (Programme for International Student Assessment) survey edition (2006) DATA • Investigating the determinants of student achievement • Highlighting the influence of the TEST-TAKING • MOTIVATION on territorial differences GOAL A MULTILEVEL REGRESSION MODEL is applied This approach is suggested by the hierarchical structure of the PISA data where students (level-one units) are nested in schools (level-two units) METHOD

3. PISA 2006 The OECD’s PISA “Programme for International Student Assessment” survey is an internationally standardised assessment administered to 15 years old students 57 Countries The survey has involved 400.000 students (21.773 in Italy) 14.300 schools (806 in Italy)

4. PISA 2006 Reading literacy The survey assesses the students’ competencies in three areas COGNITIVE TEST Mathematical literacy Scientific literacy FAMILY ENVIRONMENT OF STUDENT STUDENT QUESTIONNAIRE The OECD collects data on SCHOOL QUESTIONNAIRE SCHOOL CHARACTERISTICS

5. International ranking -Reading literacy- Italy ranked 33th in readingamong 57 countries ITALY: 469 points OECD average: 492 points

6. International ranking -Mathematical literacy- Italy ranked 38th in mathematics among 57 countries ITALY:462 points OECD average: 498 points

7. International ranking -Scientific literacy- Italy ranked 31th in Science among 57 countries ITALY:475 points OECD average: 500 points

8. The literacy divide MATHEMATICS READING SCIENCE

9. Determinants of learning outcomes Many studies (Marks, 2006; Korupp et al., 2002) emphasize the role of socio-economic background for determining learning outcomes and explaining the territorial differences. This work aims to asses how much the DIFFERENCES IN THE TEST-TAKING MOTIVATION “boost” the effect of socio economic background onthe Italian literacy divide

10. Low stake test The PISA test is considered as a low stake test since the students perceive an absence of personal consequences associated with their test performance. Without an adequate effort, test performance is likely to suffer, resulting in the examinee’s test score underestimating his or her actual level of proficiency (Wise and De Mars, 2005; Wolf & Smith, 1995; Wolf, Smith, & Birnbaum, 1995)

11. Index of student effort An index of student effort is computed on the basis of three variables : • the “Test non-response rate” computed on the basis of the number of missing orinvalid answers in the PISA cognitive test 2. the “Questionnaire non-response rate” computed on the basis of the number of missing or invalid answers in the PISA student questionnaire (family environment data) 3. the Students’self-report effort in the PISA questionnaire measured on a 10-point scale

12. Effort and performance Variation of the index of student effort in correspondence of the average performance at macro region level. Italy=100 The correlation coefficient between this index and the science performance is equal to 0,553 at national level

13. Multilevel approach A two-level random intercept regression model is adopted Variables at school level Variables at student level Outcome of ith student of jth school (Science plausible values) Error components εij~ IID-N(0, σ2) U0j~ IID-N(0, τ2) cov(U0j, εij)= 0

14. Causal structure of multilevel model Scholastic context -School mean of ESCS -Parents’pressure -Private vs public LEVEL 2 School j Study programme Technical, Professional, Vocational, Lower secondary vs Classical studies Macro area North East, North West, South, South and Islands vs Centre Scholastic resources -Computers with web -Quality of educational resources -Teacher shoratge Immigration background 1. or 2. generation vs native Student performance (Science Test score) LEVEL 1: Student i within School j Gender Girls vs boys Home educational resources Self confidence in ICT Hours per week spent on homework

15. Estimation strategy A block entryapproachis adopted (Choen and Choen, 1983) which consists to the gradual addition of the first and second level covariates The process starts with the simplest model, denoted the empty model, and then progressively adds complexity introducing school and student variables In the last model the index of student effort is considered with 8 school variables and 5 students variables

16. Block Entry Approach Emptymodel Student variables Scholastic context Study programme School location Scholastic resources Student effort

17. Pre-R2 Proportion reduction in variance or “variance explained” (PRE-R2)

18. Effect of the student variables

19. Effect of the school variables

20. Main findings • The index of student effort allows to explain a larger amount of variance, indeed after controlling for students motivation the accounted total variance total variance among schools increases from 80% to 89% • The analysis confirms that the socio-economic context plays an important role on the student achievement • The North-South divide has been overestimated by the PISA test since the score differences are also influenced by the lower effort and engagement of the Southern students

21. Territorial dummies With effort index:-25.46% With effort index:+0.82% With effort index:-48.47% With effort index:-47.50%

22. Collaboration indexes

23. Transformation steps COLLABORATION INDEX 1) (growing when the non-response rate declines) Rescaling procedure in order to obtain a common scale (0-1) for each indicator 2)