Assessing Reading Literacy in the Language of Instruction

1. Assessing Reading Literacy in the Language of Instruction John H.A.L. de Jong Intergovernmental Conference: Languages of Schooling towards a Framework for Europe Strasbourg, 16-18 October 2006

2. Overview • Background of the PISA studies • Design of the PISA studies • PISA approach to Reading Literacy • Some central findings of PISA 2000 • Extensions for PISA 2009 • Relationships between PISA & CEF • Applying CEF in Upper Secondary Vocational

3. Background of PISA study OECD: Organisation for Economic Co-operation and Development • 30 member countries committed to democracy and market economy • Provide comparative data and analyses • For countries • to compare policy experience; • seek answers to common problems; • identify good practice; • co-ordinate policies. • To support economic growth, boost employment, raise living standards.

4. History OECD • 1948 Organisation for European Economic Co-operation (OEEC), set up with support from the United States and Canada to co-ordinate the Marshall Plan for the reconstruction of Europe after World War II (www.oecd.org/history). • 1961 Created as an economic counterpart to NATO, the OECD took over from the OEEC • Early recognition of importance of education of its Citizens • 1985  Gather indicators on investment in education • Education at a glance yearly publication • 2000 first PISA

5. PISA Goals After studying resources & investments in education for a number of consecutive years: • Wish to know the yield of education: at the end of compulsory education how well are students prepared to continue further education or to start working life?

6. PISA assessment schedule

7. PISA assessment schedule

8. Participation in OECD/PISA

9. Definition Reading literacy is understanding, using and reflecting on written texts, in order to achieve one’s goals, to develop one’s knowledge and potential and to participate in society.

10. Retrieving information Interpreting text Reflection & Evaluation

11. Let’s look at Reading In the activity of reading we can distinguish four main elements: 1 there is a reader 2 there is a text (the author is part of the text, but not present in the activity) Both of these are observable. 3 the reader has a goal (a self-chosen reason for reading the text) 4 the reading activity will have a result (a change in the reader: more knowledge, satisfaction, etc ) These are not observable.

12. - Task Conform to the task Not chosen, but imposed by test Text - Test taker - Response Must be observable Reflects difference between task and observed response Reading in ‘real’ life Goal - Result Text - Reader - End point of process Assessing Reading: - Scoring rules - Score

13. These elements should be relevant to the intended trait These elements should interfere the least possible Only then are score differences likely to reflect differences in true ability on the intended trait: i.e. reliable and valid A Reading item • refers to (part of) a text • sets task pertaining to the text, requiring particular subskill(s) • may require to link information from the text to prior knowledge and experience • specifies format for overt response demonstrating application of subskill • specifies rules for scoring response

14. + + Reader Evaluate / Hypothesize Find / Connect /Assess – Personal / Private – Formal / Public Comprehension ▪Complexity Scoring Guide Specificity ▪Ling.Characteristics Response Format ▪ # of elements ▪ Familiarity Aspects of item difficulty Difficulty of task depends on prior knowledge & experience of candidate Difficulty of item depends on scoring rules Difficulty of item depends on response format Difficulty of item depends on skills required to perform task Difficulty of comprehension depends on text characteristics Comprehension is the interaction between text & reader Text

17. PISA 2000: Relation SES & Reading

19. Mean PISA 2003 scores per educational track in the Netherlands

20. Mean Reading Literacy score for Dutch-born students and for immigrant students

21. Language proficiency in upper secondary vocational education CINOP Study using CEF • what is the actual language proficiency of students? • is there a discrepancy between actual and required language proficiency for further education and professional occupation?

22. Research design - written questionnaires for language teachers and vocational teachers, N = 210 - self assessment on language proficiency for students, N= 345 - followed by a test for reading, N= 328 - oral interviews with teachers, staff members and management, N= 40

23. Teachers’ views • 60 to 70 % estimate language proficiency (reading A2-B1) insufficient for school • 10 % estimate the language proficiency on a higher level than needed for school • 80 % estimate the language proficiency insufficient for professional occupation

24. Self assessment by students > 2/3 of students estimate their reading proficiency level at B2 In reality: 7 % at A1 24 % at A2 52 % at A2-B1 17 % at B1-B2

25. CEF appropriate for description of language of instruction • Coherent and workable structure for teachers (contributes to improvement of their professional competences) • Competence based descriptions more suitable for language of instruction than linguistic descriptions of L1 • Descriptors applicable • More transparency and motivating for students (specially in combination with a language portfolio) • Offers opportunity for reflection on language learning by students

26. Shortcomings of CEF in describing language of instruction in vocational education • Can do statements are geared at academic learning routes: not well adjusted to describing specific tasks in vocational education and specific tasks in the workplace • More situations and examples have need to be offered for a range of vocational sectors

27. “Milestones” • 2000 1st PISA Cycle: Reading Major Domain • 2002 Publication Reading for Change • 2004 CINOP Dutch in Vocational Education • 2006 ALARM: Language skills at risk