Reading literacy of Austrian school leavers: between PISA and " Matura”

1. Reading literacy of Austrian school leavers: between PISA and "Matura” Irene Thelen-Schaefer, BIFIE Wien

2. Presentation overview • The study • Theoretical background • The research questions • The instruments • Operationalisation • Results • Findings • Limitations • Future research

3. Background to the study • An interest in researching the predictive potential in PISA reading tests

4. Cross-cultural assessment and translation PISA = Programme for international student assessment • In order to make a comparison of competence possible • Adaptation of tests across borders and cultures • Set the specific psychological construct into the right context • Exclude cultural bias

5. The Study • Background test: Reading in German (L1) • German translation of PISA reading literacy tasks 2009 • Reading in English (L2) • English version PISA reading literacy tasks 2009 • Austrian standardised school leaving exam • (“Matura”) • Research focus • the comparability of items • the predictability of L2 reading proficiency

6. Theoretical background 1: The skill reading L1 reading and L2 reading • Are we testing the same thing? • Same basic cognitive comprehension process across L1 and L2 contexts BUT distinctive features in L2 reading: • Language proficiency (“language threshold“) • L2 processing skills • Background knowledge

7. Theoretical background 2: The learnerLearning to read: L1/L2 differences L1 Learner • Young learner • Oral skills developed before starting reading • Learning to read and write at the same time • Large knowledge of linguistic structures and vocabulary L2 Learner • Cognitively mature • They are L1 literate, but: they have not developed L2 oral comprehension yet  different starting point in L2 reading • L2 reading processes occur in a dual-language system

8. Theoretical background 3: The processes Lower level processes Higher level processes Motivation / purpose of reading Two levels of understanding “a text model of comprehension“ “a situation model of reader interpretation“; (Grabe, 2009) Strategies Background knowledge (Schema theory) Automatisation is essential • Automatisation of word recognition • Speed of fixation • Working memory • Rauding (Reading + auding; Carver, 1984)

9. The Studyꞌs Research Questions Can L1 reading results predict L2 reading results? Can reading results from PISA English predict results of the “Matura” English L2?

10. The Instruments Student feedback questionnaire

11. Operationalisation Participating schools • Allgemeinbildende Höhere Schulen • (AHS; grammar schools) • Urban and rural areas • (Vienna, Linz, Admont, Hallein, Reutte, Villach…) • Public and private schools • Different types of AHS • BORG, BG, BRG, WIKU • Different branches (foreign languages, science, arts, technology…)

12. Operationalisation Test takers (TT) • 18-year-old students • 50 females, 51 males • 101 TT completed all three tests • 96 questionnaires returned Teachers as test administrators • trained test administrators • precise instructions given

13. (Provisional) Results Tests compared

14. (Provisional) Results All three tests show that • The test takers did well on the tests (FV, skew...) • The sample is relatively homogeneous (SD) • Normal distribution on the PISA English test and the “Matura”

15. (Provisional) Results - Facility values The PISA tasks Findings FV equal or slightly better in PISA German than in PISA English Exceptions: items 4 and 5

16. (Provisional) Results – Item analysisItem 4 • Possible reasons: technical language • Frequent reasoning for test difficulty given in questionnaires by TT: (lack of) vocabulary

17. (Provisional) Results - Item analysis Item 5 Possible reasons: • Test method • Aspect targeted (“access and retrieve”) – answer to item 5 is after answer to item 6

18. (Provisional) Results – Item analysisItem 10 • Possible reasons: • Partial credit in English only • Possible answers for partial credit are included in the “incorrect”-answer key in German • Item 10 is the same in both languages re translation, but: it is the best item in PISA German and the worst in PISA English

19. (Provisional) Results - Reliability and discrimination PISA tasks • Low coefficients on both reliability and discrimination • German: eight items discriminate positively • English: all items discriminate positively; but only item 4 is above .25 (at .279) • These plus the number of items might be the reasons for the low Cronbach’s alpha coefficients on the PISA tasks

20. (Provisional) Findings - Item analysis The “Matura” • Problematic items mainly in one test format: T/F/NG Possible reasons: • Difficult for TT to distinguish between F and NG • This test format has now been replaced by T/F justification.

21. (Provisional) Results - Correlations **. Correlation is significant at the 0.01 level (2-tailed). Findings Empirical data show that research question 1 on a low possible overlap between reading in L1 and L2 can be stated. Surprisingly, TT did nearly as well on the PISA English tasks than on the PISA German tasks.

22. Possible impact on the results • Motivation • Unknown test methods in the PISA tasks • Order of items • Partial credit in PISA English (item 10) • Tests target different purposes or aspects • Instructions • Examples

23. Further steps to be conducted • Further item analysis, incl. factor analysis PISA English – “Matura” • Correlations of test results and questionnaire items, e.g. Do avid readers have higher scores?

24. Limitations • Only 11 items in the PISA tasks • Items low reliability values • Possible memory effect PISA English – PISA German • Due to time pressure no counter-balanced design possible

25. Ideas for further research Longitudinal study as a diagnostic instrument: Can reading results from the PISA reading literacy test taken by 16-year-olds in their L2 predict the ”Matura“ reading results? • Diagnostic purpose • Advice for students and parents

26. List of references Alderson, J.C. (1984). Reading in a foreign language: a reading problem or a language problem? In J. C. Alderson and A. H. Urquhart (eds.). Reading in a foreign language. London: Longman. Alderson, J.C. (2000). Assessing reading. Cambridge, UK: Cambridge University Press. Bernhardt, E. (2005). Progress and procrastination in second language reading. Annual Review of Applied Linguistics 25, 133-150. Carver, R. (1984). Rauding theory predictions of amount comprehended under different purposes and speed reading conditions. Reading Research Quarterly 19 (2), 205-218. Grabe, W. (2009). Reading in a second language. Cambridge, UK: Cambridge University Press. Koda, K. (2005). Insights into second language reading. Cambridge, UK: Cambridge University Press. Stanovich, K. (1980). Toward an interactive-compensatory model of individual differences in the development of reading fluency. Reading Research Quarterly 16 (1), 32-71. Walter, C. (2008). Phonology in second language reading: not an optional extra. TESOL QUARTERLY 42 (3), 455-474.

27. Thank you for your attention!Contact address i.thelen-schaefer@bifie.at