NWO / PROO Dutch Programme Council for Educational Research

1. NWO / PROODutch Programme Council for Educational Research Instruments for measuring literacy development of at-risk adolescents Ron Oostdam, Amos van Gelderen, Roel van Steensel University of Amsterdam Netherlands Organisation for Scientific Research

2. Literacy development of at-risk students in multilingual contexts: A tale of three cities • International project in cooperation with the Ontario Institute for Studies in Education (OISE) and Service de la recherche en éducation (SRED) • Main research questions: • What are the most important socio-cultural, educational and individual variables associated with literacy development of at-risk adolecents (grades 7 to 9) in multilingual contexts? • Do the different variables play similar roles in predicting at-risk and non-at-risk adolescents’ literacy? Netherlands Organisation for Scientific Research

3. Factors influencing students’ literacy Netherlands Organisation for Scientific Research

4. Design of the research project Netherlands Organisation for Scientific Research

5. Integrative part project 4 • Part project 4 is directed at the measurement of literacy • The whole research project consists of a longitudinal part (3 years) and a large scale cross-sectional part • Longitudinal approach consists of an in-depth study of a focused sample of 40 at-risk adolescents • The part-projects monitor the participants on the different groups of variables • The cross-sectional study takes place in part-project 4 only • Results from the part-studies 1-3 are integrated • In the cross-sectional study predictions about the relations between the variables and literacy are tested Netherlands Organisation for Scientific Research

6. cross-sectional study • Allows testing with substantial statistical power for hypotheses derived from the multiple case-studies • Allows analyzing interactions between socio-cultural, educational and individual variables in a statistically reliable way • Allows comparison of literacy skills between at-risk and non-at-risk students Netherlands Organisation for Scientific Research

7. International Literacy Test • Literacy skills defined in a broad way: the degree in which students are able to read and write in order to participate in all kinds of functional and cultural exchange in the surrounding society • The PISA definition of reading is adopted (the ability to understand, use and reflect on written tasks in order to participate effectively in life) and extended to writing • Reading and writing tasks were constructed in close cooperation with Toronto and Geneva Netherlands Organisation for Scientific Research

8. Criteria for measuring literacy • Criterium of socio-cultural validity: measurement of literacy takes place with reading and writing tasks grounded in the lives of students • Test has to take into account differences in literacy skills • For international comparison the tasks need to be designed in such a way that they approach international, socio-cultural equivalence Netherlands Organisation for Scientific Research

9. Taxonomy literacy test • Four types of text types: narrative, expository, argumentative, instructive (reading and writing) • Four types of media: (school)books, newspapers and magazines, ‘official’ documents, internet (reading) • Distinction made between continuous and discontinuous texts (reading) • Three different types of questions: retrieving, interpreting, reflecting (reading) • Two answer formats: multiple choice and open (reading) Netherlands Organisation for Scientific Research

10. Reading tasks pilot study(* = discontinuous) Netherlands Organisation for Scientific Research

11. Example reading task HOUSE RULES • Smoking and lighting fires are strictly prohibited on our premises! • For safety and hygienic reasons hot dishes and drinks must not be prepared in living rooms or bedrooms. • During the night (from 10:00 p.m. until 6:00 a.m.) respect the rights of other guests and residents to a good night’s sleep. Showers should not be used and noise should be avoided at these times. We expressly point out that in the event of disturbance caused by noise, measures can be taken against the person(s) who caused the noise, including the termination of the contract for accommodation. • Guests are requested to avoid waste and to be economical in using electricity, heating, and water. Kindly deposit any garbage that cannot be avoided in the appropriate containers for recycling or disposal. • No visitors from outside are allowed in the bedrooms. Visitors can be received in the cafeteria daily from 10:00 a.m. until 10:00 p.m. Persons who are not occupants of our hostel cannot stay overnight in the hostel. Netherlands Organisation for Scientific Research

12. Retrieving 5. No visitors from outside are allowed in the bedrooms. Visitors can be received in the cafeteria daily from 10:00 a.m. until 10:00 p.m. Persons who are not occupants of our hostel cannot stay overnight in the hostel. What do the rules say about visitors from outside? • There are no restrictions with respect to receiving visitors from outside. • Visitors from outside are not allowed. • Visitors from outside are only allowed at certain times and in a certain place. • Visitors from outside are only allowed after the hotel management has given permission. Netherlands Organisation for Scientific Research

13. Interpreting 3. During the night (from 10:00 p.m. until 6:00 a.m.) respect the rights of other guests and residents to a good night’s sleep. Showers should not be used and noise should be avoided at these times. We expressly point out that in the event of disturbance caused by noise, measures can be taken against the person(s) who caused the noise, including the termination of the contract for accommodation. Why can’t guests take showers during the night? • Because of safety and hygienic reasons. • Because the hostel wants guests to be economical in the use of water. • Because the hostel does not provide hot water at night. • Because taking a shower could disturb other guests. Netherlands Organisation for Scientific Research

14. Reflecting 9. The hostel depends on a helping hand from our guests. We need your assistance with cleaning and tidying up the equipment, rooms and objects you have used during your stay as well as setting and clearing tables in the cafeteria. What is the purpose of rule 9? • To offer a complaint. • To give an order. • To give advice. • To make a request. Netherlands Organisation for Scientific Research

15. Writing tasks pilot study Netherlands Organisation for Scientific Research

16. Example writing task SAVE FOR TWO FREE CINEMA TICKETS!!! This is what you must do: On each Yummy Yummy candy bar wrapper there is 1 point. Collect 10 points and send these in a sufficiently stamped envelope to: Yummy Yummy Candy Bars Points Offer, PO Box 3333, 1273 KB Etten-Leur, the Netherlands. Also include € 0.39 in stamps to cover postage. Clearly write your name, address and postal code, and the free (FREE!) cinema tickets will be sent to your home as soon as possible. This offer is open until April 15. It is April 7. You have collected a total of 8 points, but you cannot find anymore bars with points. The bars in the shops have no points on the wrappers, but it is still not April 15. Thus, you cannot get your 10 points together. Nevertheless, you wish to receive the two cinema tickets. Therefore, you send your 8 points along with two complete wrappers without points. Write a letter to enclose with the points and wrappers. Explain why you are unable to send ten points. Convince Yummy Yummy Candy Bars that you want to receive the two cinema tickets and that there was nothing you could do to get ten points. Make sure that they send you the cinema tickets! Then address the envelope. Netherlands Organisation for Scientific Research

17. Pilot study Pilot of reading tasks • 14 reading tasks (internet tasks not admistered in Switserland) • 408 Dutch students (grade 7 and 9) • 298 Swiss students (grade 7, 8 and 9) • Interviews with 40 Dutch students Pilot of writing tasks • 6 writing tasks • 80 Dutch students (grade 7 and 9) • 24 Swiss students (grade 7, 8 and 9) • Interviews with 4 Dutch students Netherlands Organisation for Scientific Research

18. Final selection tasks Reading tasks • Contribution of items to the overall reliability of the literacy test • Difficulty level of the items • Distribution of questions over formats (retrieving, interpreting, reflecting; open or closed) • Distribution of tasks over text types and media • Differences between grades • Opinion of students about the tasks Writing tasks • Global scoring on text quality and text length • Global scoring of written texts on text type and content • Opinion of students about the tasks Practical constraints • Maximum administration time (about 3 hours) • Internet tasks (and writing tasks) demand computer facilities Netherlands Organisation for Scientific Research

19. Results reading tasks • Overall test reliability for reading tasks sufficient: .75 (Geneva, without internet tasks) and .77 (Amsterdam, including internet tasks) • Total number of items: 102: Retrieving (45), Interpreting (31), Reflecting(26) • One task showed a ceiling effect in Geneva as well as Amsterdam • A relatively small number of items with a item rest correlation below .10: Geneva (13); Amsterdam (19) • All tasks, except for one, show a sufficient difference between grades • Most students evaluate the tasks as attractive and enjoyable to read, with the exception of ‘House Rules’ and ‘TV guide’ Netherlands Organisation for Scientific Research

20. Results writing tasks • The narrative assignments seem to work quite well • variability in content, quality and text length • In one task students ‘squeeze in’ the elements from the final sentence right at the end of their texts • The expository assignment is problematic • Requested text type (a letter) often not delivered • Little variability in content • The argumentative assignments differ • Yummi Yummi candy bar evokes good writing products • Other task evokes texts without relevant content • The instructive assignment works well, although students find it too much work to process all the given information Netherlands Organisation for Scientific Research

21. Final remarks • In the next 2-3 weeks a final selection made for 8 reading and 3-4 writing tasks • Items with a low item rest correlation removed or adjusted • Final data collection (first measurent) done in april-may Netherlands Organisation for Scientific Research

22. For further information:Dr. R.J. OostdamFaculty of Social and Behavioral Sciences SCO Kohnstamm Institute for Educational ResearchUniversity of AmsterdamP.O. Box 94208NL-1090 GE Amsterdamphone: ** 31 (0)20 525 1330fax: ** 31 (0)20 525 1200E-mail: R.J.Oostdam@uva.nl Netherlands Organisation for Scientific Research

23. Effect sizes comparison 7th and 9th grade Netherlands Organisation for Scientific Research