1 / 59

PISA Mathematics Assessment APEC Tokyo Feb 2010

PISA Mathematics Assessment APEC Tokyo Feb 2010. Kaye Stacey University of Melbourne, Australia. Data and images in this presentation are from OECD websites and official publications and from ACER publications on PISA in Australia.

zenia
Download Presentation

PISA Mathematics Assessment APEC Tokyo Feb 2010

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. PISA Mathematics AssessmentAPEC Tokyo Feb 2010 Kaye Stacey University of Melbourne, Australia Data and images in this presentation are from OECD websites and official publications and from ACER publications on PISA in Australia. The views expressed here are those of the author and do not represent the OECD or associates.

  2. Outline • What is PISA and what does it test? • What is mathematical literacy? • A small sample of results • Country comparisons • Levels of proficiency • Performance of subgroups • Social gradient • Possibilities for Computer-based Assessment of Mathematics

  3. PISA: Programme for International Student Assessment • Test years 2000, 2003, 2006, 2009, 2012, .. • 15 year olds • assesses “the knowledge and skills that students have acquired at school and their ability to use them in everyday tasks and challenges” • reading literacy • scientific literacy • mathematical literacy • statistically rigorous, to ensure that the results are as meaningful as possible, measuring • student performance • data on the student, family, school and system factors

  4. Key features of PISA (from OECD) • policy orientation • major aim is informing educational policy and practice • aim to significantly improve understanding of the outcomes of education • concept of “literacy” (discussed later) • relevance to lifelong learning • motivation to learn, • attitudes towards learning • learning strategies; • surveys to explore features associated with educational success • characteristics of students and schools • trends monitored every 3 years; • breadth • by 2006, around 90% of the world economy • nearly 400 000 students • Recent studies tracking young people in the years after age 15 show PISA measures knowledge and skills relevant to a life success

  5. Participating countries • PISA 2000: 43 • PISA 2003: 41 • PISA 2006: 57 • nearly 400 000 students • PISA 2009: 66 • PISA Plus : +9 • PISA 2012: about 90?

  6. Asia-Pacific 2009 • China • Hong Kong • Macao • Shanghai • Indonesia • New Zealand • Thailand • Japan • Korea • Australia • Singapore • Chinese Taipei • Chile • Peru • Panama • Argentina • Mexico • USA • Canada • +9 PISA Plus • Malaysia, …

  7. Survey methods • Schools randomly selected by PISA (usually 150+) • Random sample of 35+ students per school • between age 15 yrs 3 mths & 16 yrs 2 mths • Strict sampling criteria to be included in reports • e.g. Netherlands in 2000 below required so not in trend data • Some countries oversample for their own purposes • Each student does 2 hour test and 30 min questionnaire • Items are in rotating booklets (about 13) • results of individual students not available/meaningful

  8. TIMSS: Trends in International Mathematics and Science Study • Independent body, not OECD • Tests every 4 years, since 1994/5 • Grade based sample (years 4, 8, 12) • Tests randomly sampled intact classes • hence teacher survey makes sense • Aims to test achievement of curriculum goals • Careful and extensive curriculum comparisons • More Asian countries have participated in TIMSS • Singapore’s high results very famous

  9. Now preparing Schedule of performance measures • Additional cognitive assessments: • 2003: problem solving • 2006: computer based assessment of science • 2009: electronic reading • 2012: problem solving • 2012: computer based assessment of mathematics

  10. 2000 2006 2009 Questionnaire components • School context and attitudes • themselves and their homes • attitudes to learning • School questionnaire, optional teacher questionnaires • TRENDS since 2000 (nearly) 2003 2012

  11. Country rankings are always of interest – statisticsmake comparison complicated

  12. Statistically better than Australia - Maths • PISA 2003 • Hong-Kong-China, Finland, Korea, Netherlands, Liechtenstein, Japan, Canada • PISA 2006 • Chinese Taipei, Finland, Hong-Kong-China, Korea, Netherlands, Switzerland, Canada, Macao-China • Movements: 5 stay above Australia, 2 drop to Australia’s group, 1 rises from Australia’s group, 2 new entrants

  13. What percent of students in top performance bands?

  14. What percent of students in lowest performance bands? OECD: PISA Proficiency Level 2 “a baseline level of proficiency at which students begin to demonstrate skills that enable them to actively use mathematics”

  15. What is mathematical literacy?

  16. What is mathematical literacy? • PISA assesses “the knowledge and skills that students have acquired at school and their ability to use them in everyday tasks and challenges” • Reflect recognition that globalisation and computerisation are changing labour markets and societies, and that a different set of skills is needed • US evidence: • greatest decline in jobs over the past decade has not been in manual labour, but in routine cognitive tasks – those that can easily be done at less cost by computer (Levy & Murnane, 2006).

  17. Mathematical literacy • 2003/2006 "an individual’s capacity to identify and understand the role that mathematics plays in the world, to make well-founded judgments, and to engage in mathematics in ways that meet the needs of that individual’s current and future life as a constructive, concerned and reflective citizen."  • Strong links to other concepts • Mathematical modelling (in PISA framework) • Numeracy (but certainly not just “basic skills”) • Quantitative literacy

  18. Sample domain items • PISA: “Take the test” • Reading – page 13 • Science – page 187 • Mathematics – page 97 • Questionnaires • Also download from “MyPISA”

  19. Reading

  20. Reading % teachers unaware

  21. Reading • Written text followed by questions • This one: answer as graph

  22. A test across countries needs… • cultural breadth and balance in tests • bullying, Ministry of Education, …. • not a question of intersection of school curricula around the world (TIMSS) • school curricula (e.g. reading graphs) influence success rate and hence usefulness of items for constructing a measure • high quality in translations • two “masters” for each item (English and French) • translations from both masters compared • back translation etc • some informal language will not translate • corner vs vertex English: corner or vertex French: vertex

  23. Growing Up • 6.1 A height of female in 1980 (given increase since then) • 6.2 Explain how graph shows growth rate of girls slows down after 12 yrs of age • 6.3 When are females taller than males of same age?

  24. Growing Up (6.2 – growth rate girls) • Classification • Scientific; Change and Relationships; Connections • Difficulty 574 PISA score points. • The question requires students to: • Analyse different growth curves • Evaluate characteristics of data set, represented by graph. • Note and interpret different slopes along graphs. • Reason and communicate the results of this process, within explicit models of growth. • OECD average 45% • Most successful countries: Netherlands (77%), Finland (68%), Belgium (64%), Canada (64%) • Large omission rates: Austria (44%) and Greece (43%).

  25. Scoring for 6.2 (growth rate for girls) • Score (Code) 1 : Response refers to “change” of gradient of female graph, explicitly or implicitly. • Code 11:Refers to reduced steepness, using daily-life language. • It does no longer go up, it straightens out; The curve levels off; It is more flat after 12; The line of the girls’ starts to even out and the boys’ line just gets bigger; It straightens out and the boys’ graph keeps rising. • Code 12: Refers to reduced steepness ,using mathematical language. • You can see the gradient is less; The rate of change of the graph decreases from 12 years on; (uses words like “gradient”, “slope”, or “rate of change”) • Code 13: Compares actual growth (comparison can be implicit). • From 10 to 12 the growth is about 15 cm, but from 12 to 20 the growth is only about 17 cm; The average growth rate from 10 to 12 is about 7.5 cm per year, but about 2 cm per year from 12 to 20 years. • Score (Code) 0 • Code 01: Student indicates that female height drops below male height, but does NOT mention the steepness of the female graph or a comparison of the female growth rate before and after 12 years. • The female line drops below the male line. • Code 02: Other incorrect responses. For example, the response does not refer to the characteristics of the graph, as the question clearly asks about how the GRAPH shows…. • Girls mature early; Girls don’t grow much after 12.

  26. Growing Up (6.2 – growth rate girls) • Answer type: • Daily life language: over 70% of correct answers in 24 countries • Mathematical language : 56% of correct answers in Korea • Comparing actual growth: common in Austria (34%); Mexico (26%), Greece (23%), France and Turkey (19%). • Common errors • Most common error: not referring to graph e.g. “girls don’t grow much after 12”. • Around 40% of incorrect answers in France, Korea and Poland refer to graph, only to show the female height drops below the male height.(concept of gradient??)

  27. Heatbeat (M537) - graph Heartrate = 200 – age Heartrate = 208 – 0.7*age • Newspaper statement in text alerts student to phenomenon • Question 46.1: from which age does the recommendation increase? • Question 46.2: write formula for most effective training heartrate (80% of max) Possible solution to Q46.1

  28. Meaning of “2 out of 3 in next 20 years”

  29. How additional data affects average – complex multiple choice item

  30. Interpreting an unusual representation

  31. Bookshelves • Classification • Quantity; Occupational; Connections • Difficulty rating: 499 PISA score points (Mean is set to 500) • The question requires students to: • Develop a strategy to connect two pieces of information for each component: how many available, how many needed per set • Use logical reasoning to link that analysis across the components to produce the required solution. • Communicate the mathematical answer as a real-world solution (not 5.5 bookshelves) • Most successful: • Finland and Hong Kong-China (74%), • Korea, the Czech Republic, Belgium and Denmark (72%). • OECD average: 61% correct, 29% of students attempted & incorrect and 10% did not attempt.

  32. CBAM 2012

  33. 2012 CBAM: computer based test of mathematics • New opportunities for presentation of items to measure same material better • More attractive presentation • Better presentation (e.g. animation) • Better response formats (e.g. move an animation?) • Able to test some aspects of doing mathematics by computer and so extend notion of literacy to better match world • What are these aspects?

  34. Mathematical Skills in the Workplace Final Report to the Science, Technology and Mathematics Council, UK, 2002 C. Hoyles, A. Wolf, S. Molyneux-Hodgson & P. Kent • Recommendation 1 . Raising Visibility and Awareness of the Importance of Mathematical Literacy in the Workplace • The focus should be: • The nature of mathematical literacy: that it is anchored in real data, in the context of a particular workplace. • That maths used in the workplace has economic benefits in the market-place. • That mathematics may be present quite implicitly in jobs and tasks, which are not obviously mathematical. • Many employees, regardless of their level of employment, are required to use mathematical literacy. • That IT and mathematical skills are interdependent.

  35. What IT-maths skills in ML?

  36. What IT-maths skills in ML? Use 3-D views See GoogleSketchUp demo

  37. What IT-maths skills in ML? Plot graph

  38. What IT-maths skills in ML? Write formula? What tools? (graphics) calculator? computer – what software? mobile phone? Enter data?

  39. MyPISA query How well can you…..use a spreadsheet to plot a graph? • “Do well by myself” • OECD total: 42.09% • Macao: 28.49% • Thailand 25.75% • Australia 58.42%

  40. Open Geogebra demo What IT-maths skills in ML? Explore maths

  41. Studying subgroups of students • Gender • Ethnic and home background • Migration • Language background • Indigenous students • Social gradient

  42. Social gradient

  43. Mathematical literacy by socio-economic background (Australia) • Graphic shows: • mean and confidence interval (white) • 5th, 10th, 25th, 75th, 90th, 95th percentiles

  44. Performance against social index (Science 2006)(Note wide spread)

  45. Social gradient (Science 2006)(Sci-literacy score against social index) From: Thomson & de Bortoli Exploring Scientific Literacy: How Australia Measures Up

More Related