International Benchmarking What it means – what it takes

1. International BenchmarkingWhat it means – what it takes Washington, September 11, 2009 Andreas SchleicherHead, Indicators and Analysis DivisionOECD Directorate for Education

2. Domain 1 Individual learner LevelA LevelB Instructional settings LevelC Schools, other institutions Country or system LevelD Dimensions for educational benchmarking Domain 2 Domain 3 Outputs and Outcomesimpact of learning Policy Leversshape educational outcomes Antecedentscontextualise or constrain ed policy Quality and distribution of knowledge & skills Individ attitudes, engagement and behaviour Socio-economic background of learners Quality of instructional delivery Student learning, teacher working conditions Teaching, learning practices and classroom climate The learning environment at school Community and school characteristics Output and performance of institutions Social & economic outcomes of education National educ, social and economic context Structures, resource alloc and policies

3. Domain 1 Dimensions for educational benchmarking Domain 2 Domain 3 Outputs and Outcomesimpact of learning Policy Leversshape educational outcomes Antecedentscontextualise or constrain ed policy Quality and distribution of knowledge & skills LevelA Individ attitudes, engagement and behaviour Socio-economic background of learners Individual learner LevelB Quality of instructional delivery Student learning, teacher working conditions Instructional settings Teaching, learning practices and classroom climate The learning environment at school Community and school characteristics LevelC Output and performance of institutions Schools, other institutions Social & economic outcomes of education National educ, social and economic context Country or system Structures, resource alloc and policies LevelD

4. A world of change – higher education Expenditure per student at tertiary level (USD) Cost per student Graduate supply Tertiary-type A graduation rate

5. A world of change – higher education Expenditure per student at tertiary level (USD) United States Cost per student Finland Graduate supply Tertiary-type A graduation rate

6. A world of change – higher education Expenditure per student at tertiary level (USD) Australia Finland United Kingdom Tertiary-type A graduation rate

7. A world of change – higher education Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate

8. A world of change – higher education Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate

9. A world of change – higher education Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate

10. A world of change – higher education Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate

11. A world of change – higher education Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate

12. A world of change – higher education Expenditure per student at tertiary level (USD) United States Australia Finland Tertiary-type A graduation rate

13. Domain 1 Dimensions for educational benchmarking Domain 2 Domain 3 Outputs and Outcomesimpact of learning Policy Leversshape educational outcomes Antecedentscontextualise or constrain ed policy Quality and distribution of knowledge & skills LevelA Individ attitudes, engagement and behaviour Socio-economic background of learners Individual learner LevelB Quality of instructional delivery Student learning, teacher working conditions Instructional settings Teaching, learning practices and classroom climate The learning environment at school Community and school characteristics LevelC Output and performance of institutions Schools, other institutions Social & economic outcomes of education National educ, social and economic context Country or system Structures, resource alloc and policies LevelD

14. Contribution of various factors to instructional cost per high school student as a percentage of GDP per capita (2006) Percentage points B7.1

15. Domain 1 Dimensions for educational benchmarking Domain 2 Domain 3 Outputs and Outcomesimpact of learning Policy Leversshape educational outcomes Antecedentscontextualise or constrain ed policy Quality and distribution of knowledge & skills LevelA Individ attitudes, engagement and behaviour Socio-economic background of learners Individual learner LevelB Quality of instructional delivery Student learning, teacher working conditions Instructional settings Teaching, learning practices and classroom climate The learning environment at school Community and school characteristics LevelC Output and performance of institutions Schools, other institutions Social & economic outcomes of education National educ, social and economic context Country or system Structures, resource alloc and policies LevelD

16. High science performance Average performanceof 15-year-olds on PISA in science … 18 countries perform below this line Low science performance

17. Consistency in quality standardsVariation in the performance of 15-year-olds in mathematics 20

18. Consistency in quality standardsVariation in the performance of 15-year-olds in mathematics Variation of performance within schools Variation of performance between schools OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.

19. Strengths and weaknesses of countries in science relative to their overall performanceFrance Science competencies Science knowledge OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Figure 2.13

20. Strengths and weaknesses of countries in science relative to their overall performanceCzech Republic Scientific competencies Scientific knowledge OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Figure 2.13

21. Domain 1 Dimensions for educational benchmarking Domain 2 Domain 3 Outputs and Outcomesimpact of learning Policy Leversshape educational outcomes Antecedentscontextualise or constrain ed policy Quality and distribution of knowledge & skills LevelA Individ attitudes, engagement and behaviour Socio-economic background of learners Individual learner LevelB Quality of instructional delivery Student learning, teacher working conditions Instructional settings Teaching, learning practices and classroom climate The learning environment at school Community and school characteristics LevelC Output and performance of institutions Schools, other institutions Social & economic outcomes of education National educ, social and economic context Country or system Structures, resource alloc and policies LevelD

22. High policy value A real-time assessment environment that bridges the gap between formative and summative assessment . Quick wins Must haves Examine individual, institutional and systemic factors associated with high performance Extending the range of competencies through which outcomes are assessed Monitor educational progress Measuring growth in learning Low feasibility High feasibility Establish the relative standing of states on international standards Assuming that every new skill domain is orthogonal to all others Money pits Low-hanging fruits Low policy value

23. What it takes…

24. OECD’s PISA assessment of the knowledge and skills of 15-year-olds Coverage of world economy 83% 77% 81% 85% 86% 87% Subnational/regional PISA assessments in Country Coordinated by Reported at • Australia National authorities National level • Belgium Regions OECD level • Brazil Regions National level • Canada National authorities National level • Germany National authorities National level • Italy Regions OECD level • Mexico National authorities National level • Spain Regions OECD level • Switzerland Regions National level • United Kingdom Regions OECD level

25. What it takes • Implementing PISA volume of the tests, e.g. • 3½ hours of main assessment area • 1 hour for each of the minor assessment areas each student • 2 hours on paper-and-pencil tasks (subset of all questions) • ½ hour for questionnaire on background, learning habits, learning environment, engagement and motivation school principals • questionnaire (school demography, learning environment) • Alternatives • Suitable if state performance is main interest • Curriculum match / assessment match • Embedding PISA items in state tests • Embedding state items in PISA tests • Requires coherence between assessment frameworks .

26. Test Items • A unit structure • Authentic stimuli/contexts • High proportion of constructed response items • Multiple-choice • Short constructed responses • Open constructed responses .

27. Main products • A set of basic indicators that provide policy makers with a baseline profile of the knowledge, skills and competencies of students in their state relative to those in other countries • A set of contextual indicators that provide insight into how such skills relate to important demographic, social, economic and educational variables • Trend indicators that become available because of the on-going, cyclical nature of the data collections • A knowledge base for further focused policy analysis.

28. Thank you ! www.oecd.org; www.pisa.oecd.org All national and international publications The complete micro-level database email: pisa@oecd.org Andreas.Schleicher@OECD.org … and remember: Without data, you are just another person with an opinion