1 / 33

Education for Life: PISA 2003 Symposium

This presentation discusses the PISA 2003 assessment framework, the PISA test-curriculum rating project, associations between PISA and the Junior Certificate, and key issues raised. It also explores the definitions and overarching ideas in PISA 2003 Mathematics, Reading Literacy, and Science, and the interpretation challenges of the PISA approach. The Irish test-curriculum rating project is examined, along with its limitations and potential for international comparisons.

concepciono
Download Presentation

Education for Life: PISA 2003 Symposium

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Education for Life: PISA 2003 Symposium PISA's Approach to Assessing Knowledge and Skills in the Context of Curriculum and Assessment in Ireland Jude Cosgrove, Educational Research Centre Regency Hotel, April 6, 2005

  2. Content of Presentation • Brief overview of PISA 2003 frameworks • The PISA test-curriculum rating project • Purposes • Design • Results • Associations between performance on PISA and on the Junior Certificate • Performance standards for PISA reading and mathematics by Junior Certificate syllabus level • Summing up: Key issues raised

  3. The PISA 2003 Assessment • Earlier studies, e.g., TIMSS, took a ‘common curriculum’ approach • PISA in contrast focuses on knowledge and skills deemed necessary for future participation in society • Position justified on the basis of • broader domain coverage • authenticity • relevance • (but note inherent limitations of assessment mode)

  4. The PISA 2003 Assessment • Assessment framework underpins the three domains • Developed by an international panel of subject matter experts in consultation with PISA countries • Each framework contains • definition (and elaboration) of the subject domain • description of key content and processes with illustrative examples • description of other characteristics • problem-solving framework not discussed here

  5. Definition of PISA 2003 Mathematics • “…an individual’s capacity to identify and understand the role that mathematics plays in the world, to make well-founded judgements and to engage with mathematics in ways that meet the needs of that individual’s life as a constructive, concerned and reflective citizen.” (OECD, 2003, p. 24) • Reporting scales by content area; scales for mathematical processes may be developed in the future (Reproduction, Connections, Reflection)

  6. Overarching ideas in PISA 2003 Mathematics • Shape & Space: geometry and spatial reasoning • Change & Relationships: functions, inequalities • Quantity: number sense, arithmetic and measurement • Uncertainty: statistics and probability • Not the same as, but linked to, curriculum areas • Approximately evenly represented (85 items)

  7. Definition of PISA 2003 Reading Literacy “…understanding, using and reflecting on written texts, in order to achieve one’s goals, to develop one’s knowledge and potential, and to participate in society.” (OECD, 2003, p. 15) • Retrieving information – locating one or more pieces of information in a text • Developing interpretation – constructing meaning and drawing inferences using information from the text • Reflecting on content and form – relating a text to one’s knowledge and ideas to provide a critical evaluation. • Reported as three separate subscales in 2000, only one combined reading scale in 2003. • Subscales by text format were also developed in 2000.

  8. Definition of PISA 2003 Science “…the capacity to use scientific knowledge, to identify questions and to draw evidence-based conclusions, in order to understand and help make decisions about the natural world and the changes made to it through human activity.” (OECD, 2003, p. 133) • Science in life and health • Science in Earth and in the environment • Science in technology • about one-third of items in each topic area • low emphasis on physical sciences • influenced by STS movement • no science subscales in 2000 or 2003

  9. Issues in Interpretation • While giving breadth and relevance to the assessment, the PISA approach can make results difficult to interpret vis a vis national curricula (focus, priorities, development) • Purpose of Irish test-curriculum rating project is to provide a broad description of overlap and divergence between PISA and the Junior Certificate • Junior Certificate rather than Leaving Certificate selected since 95% of Irish students have completed/almost completed JCE, while only 15% or so have just one year of LCE completed • Sample design not conducive to exact linkages between curriculum/assessment and achievements on PISA

  10. PISA Test-Curriculum Rating Projects (2000 and 2003) • Limitations • Is a measure of intended curriculum • Provides only a broad measure • Is a one-way mapping – classifies PISA items in terms of Junior Certificate content but not vice versa • Would be usefully supplemented with qualitative comparison of PISA framework/items with Junior Certificate syllabus/guidelines/examination papers • Other countries have expressed interest (Belgium, Denmark, Germany, Italy, Poland, Portugal, Switzerland, UK, Uruguay, USA) but no international comparative framework

  11. PISA Test-Curriculum Rating Projects (2000 and 2003) • Framework • Ratings attempt to measure the extent and type of similarities and differences between PISA test items in reading, mathematics and science, and the types of questions students in Third year of the Junior Cycle are exposed to, based on an examination of the intended curriculum at each level of the syllabus • Intended curriculum = instruction and learning goals in mathematics as defined at the system level (cf. TIMSS)

  12. PISA Test-Curriculum Rating Projects (2000 and 2003) • Framework • Each test item is rated with regard to each syllabus level • A 3-point rating scale is used (not/somewhat/very familiar) • Ratings for mathematics: concept, context of application, item format • Ratings for reading: process, context of application, item format • Ratings for science: concept, process, context of application, item format • Each item thus receives 9 or 12 ratings • These can be examined at both the item level and the student level

  13. PISA Test-Curriculum Rating Projects (2000 and 2003) • Method • Three individuals with extensive experience in their subject area in Ireland (ranging from curriculum development and examination setting to teaching) carried out the curriculum rating exercise for each domain • Initially each item was rated independently • This was followed by a meeting of the raters to obtain consensus on items on which there was ‘disagreement’ • 1, 1, 1; 2, 2, 3 : OK – choose modal rating • 1, 2, 3; 1, 3, 3: Disagreement – needs discussion

  14. PISA Test-Curriculum Rating Projects (2000 and 2003) • Results • Reading and Science: from 2000 • few or no new items • no change in curriculum • minor domains • science to be examined in detail in 2006 • Mathematics: from 2003 • in addition to familiarity ratings, mathematics items were classified by Junior Certificate syllabus topic area

  15. READING: PISA 2000

  16. SCIENCE: PISA 2000

  17. MATHEMATICS: PISA 2003

  18. Mathematics: JC Curriculum Area

  19. Associations Between Curriculum Familiarity and Performance • Student ‘familiarity’ scores were computed for each scale depending on • Syllabus level taken • The combination of items attempted • NB interdependence of scales • In the case of reading in particular, possible confounding with difficulty level of text • In the case of science and reading in particular, students may learn concepts/processes in other contexts

  20. Associations Between Curriculum Familiarity and Performance • Reading (2000): • Process = .55 • Context = .54 • Format = .46 • Science (2000): • Concept = .19 • Process = .05 • Context = -.01 • Format = .06 • Mathematics (2003): • Concept = .37 • Context = .21 • Format = .28

  21. Associations Between Performance on the Junior Cert. and on PISA 2003 • Students taking the JCE in 2002 or 2003 had their results matched to the PISA 2003 dataset (about 94% of the cohort) • Students’ Junior Certificate achievements were mapped onto a 12-point JCPS (9-point in the case of science) • NB the scale is not of equal intervals – in particular, there is evidence of clustering at the lower end

  22. JC Performance Scale

  23. Associations Between Performance on the Junior Cert. and on PISA 2003 • Results • PISA maths/JCE maths = .75 • PISA reading/JCE English = .67 • PISA science/JCE science = .67 • Indicates an appreciable degree of overlap in achievements, despite disparities in test content and format • BUT correlations between achievements on the PISA domains themselves are higher (.80 to .85)

  24. Performance Standards by Syllabus Level • According to the OECD (2004), students scoring at or below Level 1 on the mathematics and reading proficiency scales have only very basic, or inadequate, literacy skills and may be at risk in their future occupational, educational and personal lives. • NB PISA is not designed to measure the literacy skills of students below Level 1 with any confidence.

  25. Performance Standards by Syllabus Level: Reading

  26. Performance Standards by Syllabus Level: Mathematics

  27. Summing Up • PISA’s real-life literacy approach raises issues for interpreting achievements, particularly relating results to curriculum and instruction. • The test-curriculum rating project, unique to Ireland, was developed to facilitate interpretation and stimulate discussion around such issues.

  28. Summing Up • Ratings for reading and science were reviewed for 2000, and for 2003, mathematics ratings were reviewed; an in-depth analysis for science in 2006 is planned. • In interpreting outcomes, it might be borne in mind that students may have learned skills/concepts outside the classroom or in other subject areas (cf. the Uncertainty conundrum). Other limitations… • In all three domains, divergence between PISA and the Junior Certificate syllabus/examinations was noted; areas of overlap are also in evidence. • It appears that PISA reading taps many of the processes associated with Junior Certificate English.

  29. Summing Up • The particularly strong performance of Irish students on the Reflect subscale in PISA 2000, supports this observation (critical evaluation and audience awareness are often required in both JCE and Reflect tasks). • The contexts in which science and mathematics concepts are assessed in PISA are largely not familiar. • The low level of familiarity with context in mathematics and science suggests that a more in-depth examination of how PISA embeds problems in context is merited.

  30. Summing Up • Support for the applied nature of PISA mathematics comes from the finding that concepts appear to be located mainly in Junior Certificate mathematics topic areas of applied arithmetic & measure, and to a lesser degree, in statistics and number systems. • Up to half of the concepts underlying PISA mathematics and science items are not taught for JCE. The relevance of such concepts for students in Ireland should be considered. • Analysis of content assessed for JCE but not on PISA also merit consideration with the issue of relevance in mind…

  31. Summing Up • The lack of PISA items located in JC maths topic areas of geometry, trigonometry, functions, and algebra is noteworthy and suggests that items on the Space & Shape, and Change & Relationships scales in particular that are not taught for the Junior Certificate merit further examination. • Relating expected familiarity of various aspects of the items to student performance, the relationship is appreciable in the case of PISA reading (although possible confounding with text was noted). • For mathematics and science, concept familiarity is most strongly predictive of achievement (although the association is weak in the case of science). • Refinements to the methodology of the test-curriculum rating project may be required for 2006.

  32. Summing Up • The divergence between what PISA and the Junior Certificate assess is further supported by the observation that, although associations between PISA and Junior Certificate are substantial, achievements amongst the PISA domains are more closely associated with one another. • A cross-tabulation of PISA proficiency levels and Junior Certificate syllabus levels in the areas of English/reading and mathematics suggest a further area of divergence between the two assessments (although the purposes of the two assessments of course are very different; moreover, the Junior Certificate is not intended to be a terminal examination).

  33. Summing Up • Nonetheless, the relatively high proportions of students at Ordinary and Foundation levels at or below the lowest proficiency levels in both English/reading and mathematics suggests that many students at the end of lower secondary education lack basic literacy skills in reading and mathematics. • Further, the decline in the percentages of Higher level students achieving Level 5 on the PISA reading scale from 2000 to 2003 suggests that performance standards at Higher level in particular merit careful monitoring.

More Related