Quality and equity in educational outcomes Seeing school systems through the prism of PISA

1. Organisation for Economic Cooperation and Development (OECD) Quality and equity in educational outcomesSeeing school systems through the prism of PISA Campbell What Works seminar 9 November 2006 Dr. Karin Zimmer OECD / Directorate for Education

3. In the dark……all students, schools and education systems look the same… But with a little light….

4. In the dark……all students, schools and education systems look the same… But with a little light…. …important differences become apparent….

5. Baseline qualificationsA world of changeApprox. by % of persons with upper secondary qualfications in age groups 55-64, 45-55, 45-44 und 25-34 years 1 9 24 1

6. Overview 1. The PISA approach • Objectives and methods underlying OECD’s Programme for International Student Assessment (PISA) 2. Where we are today - and where we can be • What PISA shows students in different countries can do with what they have learned 3. How we can get there • Some policy levers that emerge from international comparisons

7. The PISA approach Measuring the quality of learning outcomes

8. Key features of PISA 2003 PISA country participation • Information collected • volume of the tests • 3½ hours of mathematics assessment, less than half in multiple-choice format • 1 hour for each of reading, science and problem solving • each student • 2 hours on paper-and-pencil tasks (subset of all questions) • ½ hour for questionnaire on background, learning environment, engagement and motivation • school principals • questionnaire (school demography, learning environment quality) • Coverage • PISA covers roughly nine tens of the world economy • Representative samples of between 3,500 and 50,000 students

9. Deciding whom to assess... grade-based sample OR age-based sample For PISA, the OECD countries chose the latter, selecting 15-year-olds in school as the population.

10. Deciding what to assess... looking back at what students were expected to have learned …or… looking ahead to what they can do with what they have learned. For PISA, the OECD countries chose the latter.

11. Three broad categories of key competencies Using “tools” interactively to engage with the world To analyse, compare, contrast, and evaluate e.g.Using language, symbols and texts Interacting with information Capitalising on the potentialof technologies To think imaginatively PISA concept of literacy Acting autonomously Interacting in diverse groups Accessing, managing, integrating and evaluating written informationin order to develop ones knowledge and potential, and to participate in, and contribute to, society e.g.Acting within the bigger picture Learning strategies Taking responsibility and understanding rights and limits e.g.Relating well to others Co-operating, working in teams Managing and resolving conflicts To apply knowledge in real-life situations To communicate thoughts and ideas effectively

12. Using “tools” interactively to engage with the world To analyse, compare, contrast, and evaluate e.g.Using language, symbols and texts Interacting with information Capitalising on the potentialof technologies To think imaginatively Reading literacy Acting autonomously Interacting in diverse groups Using, interpreting and reflecting on written material e.g.Acting within the bigger picture Forming and conducting life plans Taking responsibility and understanding rights and limits e.g.Relating well to others Co-operating, working in teams Managing and resolving conflicts To apply knowledge in real-life situations To communicate thoughts and ideas effectively

13. Using “tools” interactively to engage with the world To analyse, compare, contrast, and evaluate e.g.Using language, symbols and texts Interacting with information Capitalising on the potentialof technologies To think imaginatively Scientific literacy Acting autonomously Interacting in diverse groups Using scientific knowledge, identifying scientific questions, and drawing evidence-based conclusions to understand and make decisions about the natural world e.g.Acting within the bigger picture Forming and conducting life plans Taking responsibility and understanding rights and limits e.g.Relating well to others Co-operating, working in teams Managing and resolving conflicts To apply knowledge in real-life situations To communicate thoughts and ideas effectively

14. Using “tools” interactively to engage with the world To analyse, compare, contrast, and evaluate e.g.Using language, symbols and texts Interacting with information Capitalising on the potentialof technologies To think imaginatively Mathematical literacy Acting autonomously Interacting in diverse groups Emphasis is on mathematical knowledge put into functional use in a multitude of different situations in varied, reflective and insight-based ways e.g.Acting within the bigger picture Forming and conducting life plans Taking responsibility and understanding rights and limits e.g.Relating well to others Co-operating, working in teams Managing and resolving conflicts To apply knowledge in real-life situations To communicate thoughts and ideas effectively

15. Where we are - and where we can be What PISA shows students can do Examples of the best performing countries

16. High mathematics performance Average performanceof 15-year-olds in mathematics Low mathematics performance

17. Mathematical literacy in PISA The real world The mathematical World Making the problem amenable to mathematical treatment A mathematical model A model of reality Understanding, structuring and simplifying the situation Using relevant mathematical tools to solve the problem A real situation Validating the results Mathematical results Real results Interpreting the mathematical results

18. High mathematics performance Average performanceof 15-year-olds in mathematics High average performance Large socio-economic disparities High average performance High social equity Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Low average performance Large socio-economic disparities Low average performance High social equity Low mathematics performance

19. High mathematics performance Durchschnittliche Schülerleistungen im Bereich Mathematik High average performance Large socio-economic disparities High average performance High social equity Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Low average performance Large socio-economic disparities Low average performance High social equity Low mathematics performance

20. School performance and schools’ socio-economic background - Germany Student performance PISA Index of social background Disadvantage Advantage Student performance and student SES within schools School performance and school SES School proportional to size Figure 4.13

21. School performance and schools’ socio-economic background - Denmark Student performance PISA Index of social background Disadvantage Advantage OECD Student performance and student SES within schools OECD School performance and school SES OECD Student performance and student SES School proportional to size Figure 4.13

22. School performance and schools’ socio-economic background - Finland Student performance and student SES Student performance and student SES within schools School performance and school SES School proportional to size Student performance PISA Index of social background Disadvantage Advantage Figure 4.13

23. Is it all innate ability?Variation in student performance 20 OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.

24. In other countries, large performance differences among schools persist • In Austria, Belgium, Germany, Hungary, Italy, Japan, the Netherlands and Turkey, most of the performance variation among schools lies between schools… … and in some of these countries, most notably those that are highly stratified, a large part of that variation is explained by socio-economic inequalities in learning opportunities Is it all innate ability?Variation in student performance in mathematics In some countries, parents can rely on high and consistent standards across schools • In Canada, Denmark, Finland, Iceland and Sweden average student performance is high… … and largely unrelated to the individual schools in which students are enrolled. Variation of performance within schools Variation of performance between schools OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.

25. How can we get there? Levers for policy that emerge from international comparisons… …and what countries have done with the findings

26. Money matters but other things do too Finland Korea Japan Netherlands Belgium Canada Switzerland Czech republic Australia Iceland Denmark Sweden Ireland Germany Austria France Slovak republic Hungary Performance in mathematics Norway Poland Spain United States Italy Portugal • Spending per student is positively associated with average student performance… … but not a guarantee for high outcomes • Australia, Belgium, Canada, the Czech Republic, Finland, Japan, Korea and the Netherlands do well in terms of “value for money”… … while some of the big spenders perform below-average Greece Mexico R2 = 0.28 Cumulative expenditure (US$)

27. Sympathy doesn’t raise standards – aspiration does • PISA suggests that students and schools perform better in a climate characterised by high expectations and the readiness to invest effort, the enjoyment of learning, a strong disciplinary climate, and good teacher-student relations • Among these aspects, students’ perception of teacher-student relations and classroom disciplinary climate display the strongest relationships High ambitions and clear standards Access to best practice and quality professional development

28. Challenge and support Strong support Poor performance Improvements idiosyncratic Strong performance Systemic improvement Lowchallenge Highchallenge Poor performance Stagnation Conflict Demoralisation Weak support

29. Governance of the school system • In many of the best performing countries • School-based decision-making is combined with devices to ensure a fair distribution of substantive educational opportunities • The provision of standards and curricula at national/subnational levels is combined with advanced evaluation and support systems • That are implemented by professional agencies • Process-oriented assessments and/or centralised final examinations are complimented with individual reports and feed-back mechanisms on student learning progress • Monitoring and equity-related goals • Diverging views how evaluation and assessment can and should be used • Some see them primarily as tools to reveal best practices and identify shared problems in order to encourage teachers and schools to improve and develop more supportive and productive learning environments • Others extend their purpose to support contestability of public services or market-mechanisms in the allocation of resources • e.g. by making comparative results of schools publicly available to facilitate parental choice or by having funds following students • Differences in type of performance benchmarks being used and reported for the various stakeholders involved, including parents, teachers and schools • Standard setting and equity-related goals • Key objectives: • Raise educational aspirations, establish transparency over educational objectives, reference framework for teachers • Approaches range from definition of broad educational goals up to formulation of concise performance expectations • Some countries go beyond establishing educational standards as mere yardsticks and use performance benchmarks that students at particular age or grade levels should reach • Instruments • Minimum standards, targets defining excellence, normative performance benchmarks

30. High ambitions Devolved responsibility,the school as the centre of action Accountability and intervention in inverse proportion to success Access to best practice and quality professional development

31. High mathematics performance Durchschnittliche Schülerleistungen im Bereich Mathematik High average performance Large socio-economic disparities High average performance High social equity Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Low average performance Large socio-economic disparities Low average performance High social equity Low mathematics performance

32. High mathematics performance Durchschnittliche Schülerleistungen im Bereich Mathematik Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities • School with responsibility for deciding which courses are offered • High degree of autonomy • Low degree of autonomy Low mathematics performance

33. High mathematics performance Durchschnittliche Schülerleistungen im Bereich Mathematik Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities • Early selection and institutional differentiation • High degree of stratification • Low degree of stratification Low mathematics performance

34. Strong ambitions Devolvedresponsibility,the school as the centre of action Integrated educational opportunities Accountability Individualisedlearning Access to best practice and quality professional development

35. High ambitions Devolved responsibility, the school as the centre of action Integrated educational opportunities Accountabilityand intervention in inverse proportion to success Individualisedlearning Access to best practice and quality professional development

36.   

37. Creating a knowledge-rich profession in which schools and teachers have the authority to act, the necessary knowledge to do so wisely, and access to effective support systems The future of education systems needs to be “knowledge rich” Informed professional judgement, the teacher as a “knowledge worker” Informed prescription National prescription Professional judgement Uninformed prescription, teachers implement curricula Uninformed professional judgement, teachers working in isolation The tradition of education systems has been “knowledge poor”

38. Further information • www.pisa.oecd.org • All national and international publications • The complete micro-level database • email: pisa@oecd.org