1 / 23

What do the kids think ? A quantitative analysis of feedback questionnaires in standardised reading tests

What do the kids think ? A quantitative analysis of feedback questionnaires in standardised reading tests. Eva Konrad & Annabell Marinell. Outline. Research questions Secondary school types in Austria The research instrument Results Conclusions. Research questions.

kale
Download Presentation

What do the kids think ? A quantitative analysis of feedback questionnaires in standardised reading tests

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. What do thekidsthink?A quantitative analysis of feedback questionnaires in standardised reading tests Eva Konrad & Annabell Marinell

  2. Outline • Research questions • Secondary school types in Austria • The research instrument • Results • Conclusions A. Marinell & E. Konrad IATEFL TEASIG 2011

  3. Research questions • Is there a connection between • the test takers’ familiaritywith the topics • their interestin the texts used • how difficult the test takers felt the texts to be • their familiarity withthe test methods and how well they felt it tested their reading ability? • Is there a connection between the test takers’ total scores andhow well they felt the standardised reading test tested their reading ability? • Is there a difference in test takers’ responses from different school types? A. Marinell & E. Konrad IATEFL TEASIG 2011

  4. Secondary school types in Austria AHS BHS Classical secondary academic school Secondary academic school emphasizing mathematics and science Secondary academic school emphasizing economics Upper-secondary academic school specializing in instrumental music Upper-secondary academic school specializing in fine arts HUM: comprise business studies, theory in the respective area as well as compulsory work placements HAK: secondary schools for occupations in the business sector HTL: secondary schools for engineering BAKIP: nursery schools BASOP: colleges for social education HLFS: colleges for agriculture and forestry A. Marinell & E. Konrad IATEFL TEASIG 2011

  5. Research instrument: test booklet • 4 independent reading tasks • Test formats: • Multiple Choice (MCQ) • Multiple Matching (MM) • Sequencing (SEQ) • Note Form (NF) • True/False/Justification (TFJ) (dropped: True/False/Not given) • Answer sheet • Questionnaire A. Marinell & E. Konrad IATEFL TEASIG 2011

  6. Research instrument: questionnaire Questionnaire excerpt A. Marinell & E. Konrad IATEFL TEASIG 2011

  7. Selected questions for research • How familiar were you with the topic of the texts? • How interesting did you find the texts? • How difficult did you find the texts? • How familiar were you with the test methods? • How well could you show your reading competency in this test? A. Marinell & E. Konrad IATEFL TEASIG 2011

  8. Face validity “The extent to which a test meets the expectations of those involved in its use, e.g. administrators, teachers, candidates and test score users; the acceptability of a test to its stakeholders.” (McNamara 2000) For our research:How well did the test takers feel the standardised reading test was able to measure their reading competency? A. Marinell & E. Konrad IATEFL TEASIG 2011

  9. Test population A. Marinell & E. Konrad IATEFL TEASIG 2011

  10. Method Programme • MSB statistics programme • Developed by Gerhard Marinell and Gabriela Steckenberger Method A. Marinell & E. Konrad IATEFL TEASIG 2011

  11. MSB statistics interface

  12. Data matrix Significance level as quoted in Henning (1987): α=0.05 A. Marinell & E. Konrad IATEFL TEASIG 2011

  13. Question 1 Is there a connection between • the test takers’ familiarity with the topics • interest in the texts used • how difficultthey felt the texts to be and how well they felt their reading ability was tested? Is there a connection between these factors and the test takers’ total score? A. Marinell & E. Konrad IATEFL TEASIG 2011

  14. Values in % A. Marinell & E. Konrad IATEFL TEASIG 2011

  15. Values in % A. Marinell & E. Konrad IATEFL TEASIG 2011

  16. Values in %

  17. Question 2 Is there a connection between the test takers’ familiarity with the test methods and how well they felt their reading ability was being tested? A. Marinell & E. Konrad IATEFL TEASIG 2011

  18. Values in % A. Marinell & E. Konrad IATEFL TEASIG 2011

  19. Question 3 Is there a connection between the test takers‘ perception of how well their reading competence was tested and their total scores? A. Marinell & E. Konrad IATEFL TEASIG 2011

  20. Values in %

  21. Conclusions • We could observe certain tendencies: • Correlations in all of the factors were more frequent in 2011 than in 2008 • Correlations with face validity were more frequent than with total score • Further analysis of feedback questionnaires is needed • We will be able to say more about BHS after more trials • We still have several questions: • Are there differences between the different BHS school types? • What further information might we gain from the open questions? A. Marinell & E. Konrad IATEFL TEASIG 2011

  22. Bibliography • Alderson, C. J., Clapham, C., & Wall, D. (2005). Language testconstructionandevaluation. Cambridge: CUP. • Bachman, L. F. (2004). Statistical analysesforlanguageassessment. Cambridge: CUP. • Henning, G. (1987). A guidetolanguagetesting: development, evaluation, research. Cambridge, Mass: Newbury House Publishers. • McNamara, T. (2000). Language testing. Oxford: OUP.

  23. Thank you for your attention and enjoy the rest of the conference! A. Marinell & E. Konrad IATEFL TEASIG 2011

More Related