PISA 2003: Insights into Student Learning

1. Organisation for Economic Cooperation and Development (OECD) What light does PISA shed on student learning?Selected results from PISA 2003 Paris 2 February 2005 Andreas Schleicher Head, Indicators and Analysis Division Directorate for Education

3. In the dark, all schools and education systems look the same… But with a little light….

4. But with a little light…. …important differences become apparent….

5. Outline for today 1. Learning for tomorrow’s world • PISA – a new framework for evaluating the preparedness of youths for the knowledge society 2. Where we are today • What PISA 2003 shows about the performance of education systems 3. Where we can be • What the best performing countries show can be achieved 4. How we can get there • Levers for policy that emerge from international comparisons, with focus on assessment methods

6. Learning for Tomorrow’s World PISA – a new framework for evaluating the preparedness of youths for the knowledge society

7. PISA - The most comprehensive international assessment of student competencies • Geographic and economic coverage • 275,000 15-year-old students randomly sampled • 43 countries in 2000, 41 and 2003, 59 in 2006 • Subject matter coverage • Mathematics, Science, Reading, Cross-curricular competencies • Volume of questions • 3½ hours of mathematics assessment • 1 hour for each of reading, science and problem solving • Each student • 2 hours on paper-and-pencil tasks (subset of all questions) • ½ hour for questionnaire on background, learning habits, learning environment, engagement and motivation • Variety of task formats • Open-constructed responses, multiple-choice • Depths • A total of 7 hours of assessment material • Target population: 15-year-olds in school

8. Deciding what to assess... looking back at what students were expected to have learned OR looking ahead to what they can do with what they have learned. For PISA, the OECD countries chose the latter.

9. Knowledge based society What do we expect of key competencies? • Competency • Applying psycho-social resources… • Cognitive, motivational, ethical, volitional, social … to successfully meet complex demands in varied contexts • Key competencies • Apply to multiple areas of life • Lead to important and valued individual and social outcomes • Imply the development of a higher level of reflectivity and mental complexity • Build on a combination of cognitive and non-cognitive psychological resources • Can be learned – and taught • Key competencies operate as constellations

10. Three broad categories of key competencies Using “tools” interactively to engage with the world To analyse, compare, contrast, and evaluate e.g.Using language, symbols and texts Interacting with information Capitalising on the potentialof technologies To think imaginatively PISA concept of literacy Acting autonomously Interacting in diverse groups Accessing, managing, integrating and evaluating written informationin order to develop ones knowledge and potential, and to participate in, and contribute to, society e.g.Acting within the bigger picture Learning strategies Taking responsibility and understanding rights and limits e.g.Relating well to others Co-operating, working in teams Managing and resolving conflicts To apply knowledge in real-life situations To communicate thoughts and ideas effectively

11. Using “tools” interactively to engage with the world To analyse, compare, contrast, and evaluate e.g.Using language, symbols and texts Interacting with information Capitalising on the potentialof technologies To think imaginatively Reading literacy (focus in 2000) Acting autonomously Interacting in diverse groups Using, interpreting and reflecting on written material e.g.Acting within the bigger picture Forming and conducting life plans Taking responsibility and understanding rights and limits e.g.Relating well to others Co-operating, working in teams Managing and resolving conflicts To apply knowledge in real-life situations To communicate thoughts and ideas effectively

12. Using “tools” interactively to engage with the world To analyse, compare, contrast, and evaluate e.g.Using language, symbols and texts Interacting with information Capitalising on the potentialof technologies To think imaginatively Scientific literacy (focus in 2006) Acting autonomously Interacting in diverse groups Using scientific knowledge, identifying scientific questions, and drawing evidence-based conclusions to understand and make decisions about the natural world e.g.Acting within the bigger picture Forming and conducting life plans Taking responsibility and understanding rights and limits e.g.Relating well to others Co-operating, working in teams Managing and resolving conflicts To apply knowledge in real-life situations To communicate thoughts and ideas effectively

13. Using “tools” interactively to engage with the world To analyse, compare, contrast, and evaluate e.g.Using language, symbols and texts Interacting with information Capitalising on the potentialof technologies To think imaginatively Mathematical literacy (focus in 2003) Acting autonomously Interacting in diverse groups Emphasis is on mathematical knowledge put into functional use in a multitude of different situations in varied, reflective and insight-based ways e.g.Acting within the bigger picture Forming and conducting life plans Taking responsibility and understanding rights and limits e.g.Relating well to others Co-operating, working in teams Managing and resolving conflicts To apply knowledge in real-life situations To communicate thoughts and ideas effectively

14. Mathematical literacy in PISA The mathematical Worldas an instrument to understand the real world The real world Making the problem amenable to mathematical treatment A mathematical model A model of reality Understanding, structuring and simplifying the situation Using relevant mathematical tools to solve the problem A real situation Validating the results Mathematical results Real results Interpreting the mathematical results

15. Using “tools” interactively to engage with the world PISA assessment of • Problem-solving competencies • Using cognitive processes to resolve real situations where the solution path is not immediately obvious and where the competencies required are not within a single discipline. • PISA self-reports on: • Dispositions to learning • Learning strategies • Engagement with school • Self-concept Acting autonomously Interacting in diverse groups

16. Development of assessments • Frameworks by international experts • Assessment materials • submitted by countries • developed by research consortium • screened for cultural bias • by countries • by an international expert panel • items with prima facie cultural bias removed at this stage • internationally validated translations • trialled to check items working consistently in all countries • Final tests • items shown in trial to be culturally biased removed • best items chosen for final tests • balanced to reflect framework • range of difficulties • range of item types (constructed response, multiple choice)

17. Securing an equitable distribution of learning opportunities Measured by the impact students’ and schools’ socio-economic background has on performance – not merely by the distribution of learning outcomes

18. High mathematics performance Average performanceof 15-year-olds in mathematics High average performance Large socio-economic disparities High average performance High social equity A widening gap • More improvement at the top of the scale has widened the gap between the top and bottom performers in the OECD. Top-performers • Finland remained first in reading and since 2000 moved further in math and science… … and is now on a par with the East Asian countries that were previously unmatched in math and science • Also the Netherlands is among the top-performers in math … though not in reading and science. • As is the Flemish Community of Belgium There is more to this than national income • Countries with higher national income and better educated adult populations tend to perform better… … but there are exceptions. Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Progress • Other countries with improvements in at least two assessment areas were Belgium, the Czech Republic and Germany … In Belgium and Germany it was the top performers who drove improvements. Progress • Poland raised it’s overall performance in all four assessment areas … thanks to big improvements among lower-performing students in the wake of a major reform in 1999. Low average performance Large socio-economic disparities Low average performance High social equity Low mathematics performance

19. High mathematics performance Durchschnittliche Schülerleistungen im Bereich Mathematik High average performance Large socio-economic disparities High average performance High social equity Differences in socio-economic background pose major challenges for education systems • Students whose parents have better-paid jobs, are better educated or have more “cultural” possessions in their homes tend to perform better… … But the performance advantage varies • Australia, Canada, Finland, Iceland and Japan provide examples showing that it is possible to combine quality and equity • In contrast, results for Belgium, Germany, Hungary, he Slovak Republic and Turkey reveal large socio-economic inequalities in the distribution of learning opportunities . Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Low average performance Large socio-economic disparities Low average performance High social equity Low mathematics performance

20. Ensuring consistent performance standards across schools Between and within-school variation in performance

21. Is it all innate ability?Variation in student performance 20 OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.

22. Is it all innate ability?Variation in student performance in mathematics In some countries, parents can rely on high and consistent standards across schools • In Canada, Denmark, Finland, Iceland and Sweden average student performance is high… … and largely unrelated to the individual schools in which students are enrolled. Variation of performance within schools Variation of performance between schools 14 12 1 11 5 OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.

23. Bridging the gender gap Performance, attitudes and motivation

24. Gender differences • In reading, girls are far ahead • In all countries, girls significantly outperform boys in reading • In mathematics, boys tend to be somewhat ahead • In most countries, boys outperform girls … but mostly by modest amounts… … and mainly because boys are overrepresented among top-performers while boys and girls tend to be equally represented in the “at risk” group • Within classrooms and schools, the gender gap is often larger • Strong problem-solving performance for girls suggests… … that it is not the cognitive processes underlying mathematics that give boys an advantage… … but the context in which mathematics appears in school • Gender differences in interest and attitudes towards mathematics are significantly greater than the observed performance gap • Girls report much lower intrinsic (though not instrumental) motivation in mathematics, more negative attitudes and much greater anxiety with mathematics… … and this may well contribute to the significant gender difference in educational and occupational pathways in mathematics-related subjects

25. Creating strong foundations for lifelong learning Performance, attitudes and motivation

26. Student approaches to learning • The ability to manage one’s learning is both an important outcome of education and a contributor to student literacy skills at school • Learning strategies, motivation, self-related beliefs, preferred learning styles • Different aspects of students’ learning approaches are closely related • Well-motivated and self-confident students tend to invest in effective learning strategies and this contributes to their literacy skills • Immigrant students tend to be weaker performers … but they do not have weaker characteristics as learners • Boys and girls each have distinctive strengths and weaknesses as learners • Girls stronger in relation to motivation and self-confidence in reading • Boys believing more than girls in their own efficacy as learners and in their mathematical abilities

27. Control strategies in mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.9, p.375 and Figure 3.9, p.143.

28. Elaboration strategies in mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.11, p.377 and Figure 3.11, p.146.

29. Percentage of variance in student mathematics performance that is explained by the combined effect of -interest in and enjoyment of mathematics -anxiety in mathematics -control strategies Percentage of variance in student use of control strategies that is explained by the combined effect of -interest in and enjoyment of mathematics -anxiety in mathematics 50 0 10 40 20 30 20 30 40 10 50 0 % Combined explanatory power of student learning characteristics on mathematics performance and control strategies OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Figure 3.13, p.149.

30. How can we get there? Levers for policy that emerge from international comparisons

31. Sympathy doesn’t raise standards – aspiration does • In many of the best performing countries • National research teams report a strong “culture of performance” • Which drives students, parents, teachers and the educational administration to high performance standards • PISA shows that students perceived a high degree of teacher support • Which should not be simply equated with “achievement press”

32. Governance of the school system • In many of the best performing countries • Decentralised decision-making is combined with devices to ensure a fair distribution of substantive educational opportunities • The provision of standards and curricula at national/subnational levels is combined with advanced evaluation systems • That are implemented by professional agencies • Process-oriented assessments and/or centralised final examinations are complimented with individual reports and feed-back mechanisms on student learning progress

33. HighMathematics performance High performance Low social equity High performance High social equity Some notes on Italy: • Only 8% of schools involved in appointing teachers (OECD 64%) • Only 2% of schools involved in determining teacher salary increases (OECD 38%) • All schools involved in establishing student assessment policies (OECD 85%) • 84% of schools involved in determining course content (OECD 67%) Moderate impact of social background on performance Strong impact of social background on performance Low performance High social equity Low performance Low social equity • Formulating the school budget and deciding on budget allocations in schools • High degree of autonomy • Low degree of autonomy Low Performance

34. Organisation of instruction • In many of the best performing countries • Schools and teachers have explicit strategies and approaches for teaching heterogeneous groups of learners • A high degree of individualised learning processes • Disparities related to socio-economic factors and migration are recognised as major challenges • Students are offered a variety of extra-curricular activities • Schools offer differentiated support structures for students • E.g. school psychologists or career counsellors • Institutional differentiation is introduced, if at all, at later stages • Integrated approaches also contributed to reducing the impact of students socio-economic background on outcomes

35. HighMathematics performance High performance Low social equity High performance High social equity Moderate impact of social background on performance Strong impact of social background on performance Low performance High social equity Low performance Low social equity • Early selection and institutional differentiation • High degree of stratification • Low degree of stratification Low Performance

36. Teacher support in mathematicsStudents’ views OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.1a, p.403 and Figure 5.1, p.213.

37. Student-related factors affecting school climatePrincipals’ views OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.2a, p.406 and Figure 5.2, p.216.

38. Teacher-related factors affecting school climatePrincipals’ views OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.4a, p.410 and Figure 5.4, p.220.

39. Assessment methods in the best-performing PISA countriesStandardised tests

40. Assessment methods in the top performing PISA countries Teacher developed tests

41. Assessment methods in the top performing PISA countries Judgemental ratings

42. Assessment methods in the top performing PISA countries Student portfolios

43. Assessment methods in the top performing PISA countriesStudent assignements/projects/homework

44. One challenge – different approaches The future of education systems needs to be “knowledge rich” Informed professional judgement, the teacher as a “knowledge worker” Informed prescription National prescription Professional judgement Uninformed prescription, teachers implement curricula Uninformed professional judgement The tradition of education systems has been “knowledge poor”

45. Further information • www.pisa.oecd.org • All national and international publications • The complete micro-level database • email: pisa@oecd.org • Andreas.Schleicher@OECD.org … and remember: Without data, you are just another person with an opinion

46. PISA country participation