PISA: A Comparative Perspective on Quality and Equity in Education Systems

1. Organisation for Economic Cooperation and Development (OECD) Seeing schools through the prism of PISAAn international comparative perspective on quality and equity in education systems Tokyo 24 June 2005 Andreas Schleicher Head, Indicators and Analysis Division OECD Directorate for Education

3. In the dark, all education systems look the same… 暗がりのなかでは、どの学校も教育システムも同じように見える… だが、少し光を当てると…. But with a little light….

4. だが、少し光を当てると…. But with a little light…. …important differences become apparent… …重要な違いが明らかになってくる….

5. High mathematics performance Average performanceof 15-year-olds in mathematics Low mathematics performance

6. Overview 1. The PISA approach • Measuring the quality of learning outcomes internationally 2. Where we are today • What PISA shows students in different countries can do with what they have learned 3. Where we can be • Examples from the best performing countries 4. How we can get there • Some policy levers that emerge from international comparisons

7. The PISA approach Measuring the quality of learning outcomes

8. Three broad categories of key competencies Using “tools” interactively to engage with the world To analyse, compare, contrast, and evaluate e.g.Using language, symbols and texts Interacting with information Capitalising on the potentialof technologies To think imaginatively PISA concept of literacy Acting autonomously Interacting in diverse groups Accessing, managing, integrating and evaluating written informationin order to develop ones knowledge and potential, and to participate in, and contribute to, society e.g.Acting within the bigger picture Learning strategies Taking responsibility and understanding rights and limits e.g.Relating well to others Co-operating, working in teams Managing and resolving conflicts To apply knowledge in real-life situations To communicate thoughts and ideas effectively

9. Using “tools” interactively to engage with the world To analyse, compare, contrast, and evaluate e.g.Using language, symbols and texts Interacting with information Capitalising on the potentialof technologies To think imaginatively Reading literacy Acting autonomously Interacting in diverse groups Using, interpreting and reflecting on written material e.g.Acting within the bigger picture Forming and conducting life plans Taking responsibility and understanding rights and limits e.g.Relating well to others Co-operating, working in teams Managing and resolving conflicts To apply knowledge in real-life situations To communicate thoughts and ideas effectively

10. Using “tools” interactively to engage with the world To analyse, compare, contrast, and evaluate e.g.Using language, symbols and texts Interacting with information Capitalising on the potentialof technologies To think imaginatively Scientific literacy Acting autonomously Interacting in diverse groups Using scientific knowledge, identifying scientific questions, and drawing evidence-based conclusions to understand and make decisions about the natural world e.g.Acting within the bigger picture Forming and conducting life plans Taking responsibility and understanding rights and limits e.g.Relating well to others Co-operating, working in teams Managing and resolving conflicts To apply knowledge in real-life situations To communicate thoughts and ideas effectively

11. Using “tools” interactively to engage with the world To analyse, compare, contrast, and evaluate e.g.Using language, symbols and texts Interacting with information Capitalising on the potentialof technologies To think imaginatively Mathematical literacy Acting autonomously Interacting in diverse groups Emphasis is on mathematical knowledge put into functional use in a multitude of different situations in varied, reflective and insight-based ways e.g.Acting within the bigger picture Forming and conducting life plans Taking responsibility and understanding rights and limits e.g.Relating well to others Co-operating, working in teams Managing and resolving conflicts To apply knowledge in real-life situations To communicate thoughts and ideas effectively

12. Mathematical literacy in PISA The real world The mathematical World Making the problem amenable to mathematical treatment A mathematical model A model of reality Understanding, structuring and simplifying the situation Using relevant mathematical tools to solve the problem A real situation Validating the results Mathematical results Real results Interpreting the mathematical results

13. Deciding what to assess... looking back at what students were expected to have learned …or… looking ahead to what they can do with what they have learned. For PISA, the OECD countries chose the latter.

14. Development of assessments • Frameworks by international experts • Assessment materials • submitted by countries • developed by research consortium • screened for cultural bias • by countries • by expert, international panel • items with prima facie cultural bias removed at this stage • internationally validated translations • trialled to check items working consistently in all countries • Final tests • items shown in trial to be culturally biased removed • best items chosen for final tests • balanced to reflect framework • range of difficulties • range of item types (constructed response, multiple choice)

15. Key features of PISA 2003 • Information collected • volume of questions • 3½ hours of mathematics assessment • 1 hour for each of reading, science and problem solving • each student • 2 hours on paper-and-pencil tasks (subset of all questions) • ½ hour for questionnaire on background, learning habits, learning environment, engagement and motivation • school principals • questionnaire (school demography, learning environment quality) • Coverage • PISA covers roughly nine tens of the world economy

16. Where we are - and where we can be What PISA shows students can do Examples of the best performing countries

17. PISA provides five key benchmarks for the quality of education systems 1. Overall performance of education systems 2. Equity in the distribution of learning opportunities • Measured by the impact students’ and schools’ socio-economic background has on performance… … not merely by the distribution of learning outcomes 3. Consistency of performance standards across schools 4. Gender differences 5. Foundations for lifelong learning • Learning strategies, motivation and attitudes

18. High mathematics performance Average performanceof 15-year-olds in mathematics High average performance Large socio-economic disparities High average performance High social equity Top-performers • Finland remained first in reading and since 2000 moved further in math and science… … and is now on a par with the East Asian countries that were previously unmatched in math and science • Also the Netherlands is among the top-performers in math … though not in reading and science. • As is the Flemish Community of Belgium A widening gap • More improvement at the top of the scale has widened the gap between the top and bottom performers in the OECD. Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Progress • Other countries with improvements in at least two assessment areas were Belgium, the Czech Republic and Germany … In Belgium and Germany it was the top performers who drove improvements. Progress • Poland raised it’s overall performance in all four assessment areas … thanks to big improvements among lower-performing students in the wake of a major reform in 1999. Low average performance Large socio-economic disparities Low average performance High social equity Low mathematics performance

19. High mathematics performance Durchschnittliche Schülerleistungen im Bereich Mathematik High average performance Large socio-economic disparities High average performance High social equity Differences in socio-economic background pose major challenges for education systems • Students whose parents have better-paid jobs, are better educated or have more “cultural” possessions in their homes tend to perform better… … But the performance advantage varies • Australia, Canada, Finland, Iceland and Japan provide examples showing that it is possible to combine quality and equity • In contrast, results for Belgium, Germany, Hungary and the Slovak Republic reveal large socio-economic inequalities in the distribution of learning opportunities . Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Low average performance Large socio-economic disparities Low average performance High social equity Low mathematics performance

20. Is it all innate ability?Variation in student performance 20 OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.

21. Is it all innate ability?Variation in student performance in mathematics In some countries, parents can rely on high and consistent standards across schools • In Canada, Denmark, Finland, Iceland and Sweden average student performance is high… … and largely unrelated to the individual schools in which students are enrolled. Variation of performance within schools Variation of performance between schools 14 12 1 11 5 OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.

22. School performance and schools’ socio-economic background - Germany Student performance PISA Index of social background Disadvantage Advantage Student performance and student SES within schools School performance and school SES Student performance and student SES School proportional to size Figure 4.13

23. 学校の成績と学校の社会経済的背景（日本） 生徒の成績と生徒の社会経済的地位（SES） 生徒の成績と学校内における生徒のSES 学校の成績と学校のSES 学校の規模 生徒の成績 社会的背景に関するPISA指数 不利 有利

24. School performance and schools’ socio-economic background - Norway Student performance PISA Index of social background Disadvantage Advantage OECD Student performance and student SES within schools OECD School performance and school SES OECD Student performance and student SES School proportional to size Figure 4.13

25. School performance and schools’ socio-economic background - Finland Student performance and student SES Student performance and student SES within schools School performance and school SES School proportional to size Student performance PISA Index of social background Disadvantage Advantage Figure 4.13

26. School performance and schools’ socio-economic background Student performance PISA Index of social background Disadvantage Advantage Universal policies • Increasing educational performance of all children through reforms applied equally across the school system, e.g. • Altering content or pace of curriculum • Improving instructional techniques • Changing the learning environment in schools and classrooms • Standards and accountability • Teacher professional development School proportional to size Figure 4.13

27. School performance and schools’ socio-economic background Student performance PISA Index of social background Disadvantage Advantage Socio-economically targeted policies • Providing a specialised curriculum or additional educational resources to students from disadvantaged backgrounds • Students are often also identified through other risk factors, e.g. immigration, ethnicity, low-income community School proportional to size Figure 4.13

28. School performance and schools’ socio-economic background Student performance PISA Index of social background Disadvantage Advantage Compensatory policies • Providing additional economic resources to students from disadvantaged backgrounds • Different to socio-economically targeted policies, efforts are directed to ameliorating economic circumstances, rather than providing specialised curriculum or additional educational resources School proportional to size Figure 4.13

29. School performance and schools’ socio-economic background Student performance PISA Index of social background Disadvantage Advantage Performance targeted policies • Providing additional economic resources to students based on their academic performance • Early intervention programmes • Remedial and recovery programmes • Performance-based tracking or streaming • Countries with flat gradients • In combination with SES-targeted policies for countries with steeper gradients School proportional to size Figure 4.13

30. Gender differences • In reading, girls are far ahead • In all countries, girls significantly outperform boys in reading • In mathematics, boys tend to be somewhat ahead • In most countries, boys outperform girls … but mostly by modest amounts… … and mainly because boys are overrepresented among top-performers while boys and girls tend to be equally represented in the “at risk” group • Within classrooms and schools, the gender gap is often larger • Strong problem-solving performance for girls suggests… … that it is not the cognitive processes underlying mathematics that give boys an advantage… … but the context in which mathematics appears in school • Gender differences in interest and attitudes towards mathematics are significantly greater than the observed performance gap • Girls report much lower intrinsic (though not instrumental) motivation in mathematics, more negative attitudes and much greater anxiety with mathematics… … and this may well contribute to the significant gender difference in educational and occupational pathways in mathematics-related subjects

31. Interest in and enjoyment of mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.4, p.367 and Figure 3.4, p.126.

32. Instrumental motivation in mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.2a, p.360 and Figure 3.3a, p.122.

33. Anxiety in mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.8, p.374 and Figure 3.8, p.139.

34. Attitudes towards school OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.4, p.367 and Figure 3.4, p.126.

35. HighMathematics performance    High performance Low social equity High performance High social equity       Moderate impact of social background on performance Strong impact of social background on performance   Low performance High social equity Low performance Low social equity • Student anxiety in mathematics • High degree of anxiety • Low degree of anxiety        Low Performance

36. How can we get there? Levers for policy that emerge from international comparisons

37. Domain 1 Individual learner LevelA LevelB Instructional settings LevelC Schools, other institutions Country or system LevelD OECD framework Domain 2 Domain 3 Outputs and Outcomesimpact of learning Policy Leversshape educational outcomes Antecedentscontextualise or constrain ed policy Quality and distribution of knowledge & skills Individ attitudes, engagement and behaviour Socio-economic background of learners Quality of instructional delivery Student learning, teacher working conditions Teaching, learning practices and classroom climate The learning environment at school Community and school characteristics Output and performance of institutions Social & economic outcomes of education National educ, social and economic context Structures, resource alloc and policies

38. Money matters but other things do too Finland Korea Japan Netherlands Belgium Canada Switzerland Czech republic Australia Iceland Denmark Sweden Ireland Germany Austria France Slovak republic Hungary Performance in mathematics Norway Poland Spain United States Italy Portugal • Spending per student is positively associated with average student performance… … but not a guarantee for high outcomes • Australia, Belgium, Canada, the Czech Republic, Finland, Japan, Korea and the Netherlands do well in terms of “value for money”… … while some of the big spenders perform below-average Greece Mexico R2 = 0.28 Cumulative expenditure (US$)

39. Sympathy doesn’t raise standards – aspiration does • In the focus countries • National research teams report a strong “culture of performance” • Which drives students, parents, teachers and the educational administration to high performance standards • PISA suggests… … that students and schools perform better in a climate characterised by high expectations and the readiness to invest effort, the enjoyment of learning, a strong disciplinary climate, and good teacher-student relations • Among these aspects, students’ perception of teacher-student relations and classroom disciplinary climate display the strongest relationships

40. Governance of the school system • Monitoring and equity-related goals • Diverging views how evaluation and assessment can and should be used • Some see them primarily as tools to reveal best practices and identify shared problems in order to encourage teachers and schools to improve and develop more supportive and productive learning environments • Others extend their purpose to support contestability of public services or market-mechanisms in the allocation of resources • e.g. by making comparative results of schools publicly available to facilitate parental choice or by having funds following students • Differences in type of performance benchmarks being used and reported for the various stakeholders involved, including parents, teachers and schools • In the focus countries • Decentralised decision-making is combined with devices to ensure a fair distribution of substantive educational opportunities • The provision of standards and curricula at national/subnational levels is combined with advanced evaluation systems • That are implemented by professional agencies • Process-oriented assessments and/or centralised final examinations are complimented with individual reports and feed-back mechanisms on student learning progress • Standard setting and equity-related goals • Key objectives: • Raise educational aspirations, establish transparency over educational objectives, reference framework for teachers • Approaches range from definition of broad educational goals up to formulation of concise performance expectations • Some countries go beyond establishing educational standards as mere yardsticks and use performance benchmarks that students at particular age or grade levels should reach • Instruments • Minimum standards, targets defining excellence, normative performance benchmarks

41. High mathematics performance Durchschnittliche Schülerleistungen im Bereich Mathematik High average performance Large socio-economic disparities High average performance High social equity Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Low average performance Large socio-economic disparities Low average performance High social equity Low mathematics performance

42. High mathematics performance Durchschnittliche Schülerleistungen im Bereich Mathematik Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities • School with responsibility for deciding which courses are offered • High degree of autonomy • Low degree of autonomy Low mathematics performance

43. Public and private schools Public schools perform better Private schools perform better

44. Organisation of instruction • In the focus countries • Schools and teachers have explicit strategies and approaches for teaching heterogeneous groups of learners • A high degree of individualised learning processes • Disparities related to socio-economic factors and migration are recognised as major challenges • Students are offered a variety of extra-curricular activities • Schools offer differentiated support structures for students • E.g. school psychologists or career counsellors • Institutional differentiation is introduced, if at all, at later stages • Integrated approaches also contributed to reducing the impact of students socio-economic background on outcomes

45. HighMathematics performance High performance Low social equity High performance High social equity Moderate impact of social background on performance Strong impact of social background on performance Low performance High social equity Low performance Low social equity • Early selection and stratification • High degree of stratification • Low degree of stratification Low Performance

46. Support systems and professional teacher development • In the focus countries • Effective support systems are located at individual school level or in specialised support institutions • Teacher training schemes are selective • The training of pre-school personnel is closely integrated with the professional development of teachers • Continuing professional development is a constitutive part of the system • Special attention is paid to the professional development of school management personnel

47. Teacher support in mathematicsStudents’ views OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.1a, p.403 and Figure 5.1, p.213.

48. Student-related factors affecting school climatePrincipals’ views OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.2a, p.406 and Figure 5.2, p.216.

49. Teacher-related factors affecting school climatePrincipals’ views OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.4a, p.410 and Figure 5.4, p.220.

50. Pre-school attendance and performance 38 score points is the average performance difference associated with one school year Difference in performance between those who attended pre-school for more than one year and those with no pre-school Percentage of students who attended pre-school