1 / 28

Why bother?

Curricular Objectives – Test Constructs – Test Items – Test Tasks … ao.Univ.Prof. Mag. Dr. Guenther Sigott Language Testing Centre Alpen-Adria-Universität Klagenfurt http://www.uni-klu.ac.at/ltc. Why bother?.

bridgit
Download Presentation

Why bother?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Curricular Objectives – Test Constructs – Test Items – Test Tasks …ao.Univ.Prof. Mag. Dr. Guenther SigottLanguage Testing CentreAlpen-Adria-Universität Klagenfurthttp://www.uni-klu.ac.at/ltc

  2. Why bother? Important to understand what we want to teach and what pupils are supposed to learn (curricular objectives). Important to understand what we want to test (test constructs).

  3. BHS Matura • Is it clear what the exam should test? • Is it clear what the exam should test in the different school types? • Does the exam in fact test what it should test? • Does the exam test the same things in all • schools of the same type so that the results are comparable? • Does the exam test the same things every year so that results are comparable? • Does the exam relate to internationally accepted descriptions of educational standards such as the CEFR?

  4. Q1: Is it clear what the exam should test? WHAT: curricular objectives / test construct Models of language competence Curriculum CEF HOW: test method

  5. Origins of the WHAT

  6. L.F. Bachman, Fundamental Considerations in Language Testing, OUP 1990, p. 87.

  7. E8 Reading Strategies (extract) • Careful Reading • Understanding the Text • understanding explicitly stated main idea(s) and / or distinguishing that from supporting details • locating, identifying, understanding and comparing facts, opinions, definitions (this includes search reading strategies) • understanding the logical organisation of the text, e.g. understanding relationships among ideas in a text (problem – solution, cause – effect, temporal sequence, etc.) • understanding cohesive relationships (reference, ellipsis, substitution, conjunction, lexical cohesion) • making propositional inferences (deducing information that is not explicitly stated from information that is explicitly stated) • Adapted from Urquhart & Weir 1998

  8. CEF Overall Written Production C1 Can write clear, well-structured texts of complex subjects, underlining the relevant salient issues, expanding and supporting points of view at some length with subsidiary points, reasons and relevant examples, and rounding off with an appropriate conclusion. B2 Can write clear, detailed texts on a variety of subjects related to his / her field of interest, synthesising and evaluating information and arguments from a number of sources.

  9. HOW? What do we do in order to make our students display their competences?

  10. Item 1 You are asking your friend a lot of questions about him and his family. Choose the right answer for each of the questions (there is only one correct answer per question!) and fill in the empty boxes with the correct number.

  11. Item01 Test format: Grammar, Matching activity (2nd grade AHS) You are asking your friend a lot of questions about him and his family. Choose the right answer for each of the questions (there is only one correct answer per question!) and fill in the empty boxes with the correct number.

  12. Item 1 Tick the answer which is grammatically correct: Is your brother good at swimming? Yes, he can. Yes, you are. Yes, he does. No, he isn’t.

  13. You will have 10 minutes to do this task. Read the instructions carefully and then write your text on the separate sheet. Your short letter or e-mail should be 40 to 60 words long. Your American aunt has sent you a birthday present. Write a short letter or an e-mail to say thank you. Tell your aunt you really liked your present and why it is useful for you. You can also ask her if you could visit her in your summer vacation. You could start your letter like this: Test Method in Writing and SpeakingResist the temptation: keep method distinct from construct Dear Aunt Lizzy: .................................................. If you rate the resulting text for Grammar, what is construct, what method?

  14. Test Method in Writing and SpeakingResist the temptation: keep method distinct from construct Du hast für diese Aufgabe 10 Minuten Zeit. Lies die Anleitung gut durch und schreib Deinen Text auf das beiliegende Blatt. Dein kurzer Brief oder Deine Email soll 40 bis 60 Wörter lang sein. Deine Tante in Amerika hat Dir ein Geburtstagsgeschenk geschickt. Schreib einen kurzen Brief oder eine Email, um dich zu bedanken. Sag Deiner Tante, dass Dir das Geschenk sehr gut gefällt und warum Du es gut gebrauchen kannst. Du kannst sie auch fragen, ob Du sie in den Sommerferien vielleicht besuchen kannst. Du könntest Deinen Brief so beginnen: Dear Aunt Lizzy: .................................................. If you rate the resulting text for Grammar, what is construct, what method?

  15. Test Method ctd. If you rate the resulting text for Vocabulary, what is construct, what method? If you rate the resulting text for Cohesion and Coherence, what is construct, what method?

  16. Test Method in Writing and SpeakingResist the temptation: keep method distinct from construct If you rate the resulting text for Vocabulary, what is construct, what method? If you rate the resulting text for Cohesion and Coherence, what is construct, what method?

  17. So? To state what the exam should test, we must be able to say what is the construct and what the method. We want to test construct, not method.

  18. Q2: Is it clear what the exam should test in the different school types? Once we are clear about the distinction between construct and method, we are in a good position to discuss this question. Which areas of the construct are relevant for all school types, which only for particular school types?

  19. Q3: Does the exam in fact test what it should test? • Construct + method will yield an exam which we believe tests what we are interested in. It is our obligation to demonstrate that this exam indeed tests what we claim it tests. • Are we measuring anything at all or are we just ascribing figures to performances on a random basis? – reliability • Scoring keys must be available. • Rating scales have to be put in place. • Raters need to be trained in using the rating scales. • It should not matter who conducts the exam. • It should not matter who rates / scores it. • Are we measuring the things we set out to measure? Accompanying research is needed: • Content validation • Construct validation

  20. Q4: Does the exam test the same things in all schools of the same type so that the results are comparable? Use the same exam in all the schools of the same type at the same time. Develop parallel exams which test the same things and which are equally difficult.

  21. Q5: Does the exam test the same things every year so that the results are comparable? Use the same exam every year (a theoretical possibility). Develop parallel exams which test the same things and which are equally difficult.

  22. Parallel forms are needed. This requires identity of construct plus constancy / comparability of difficulty. This is related to setting cutoff scores for pass / fail decisions. This leads to the issue of how these cutoff scores relate to the CEF (B2?) Moreover, there is a need to determine cutoff scores for the individual grades on the national 5-point scale (multiple cutoffs). This calls for standard-setting procedures to be put in place. Q6: Does the exam relate to internationally used descriptions of educational standards such as the CEF?

  23. Standard Setting Problem: item-banking not practical in the Austrian Matura context (item security). Consequence: standard setting required for each parallel form. Needed: descriptions of minimally acceptable competences for Matura (4-3-2-1). CEF comes in handy but is not sufficient.

  24. Bottom Line For all of these things to work, we need items and tasks based on a clearly defined construct with a clearly defined test method.

  25. Item vs Task A cline which is defined by length of response. Tick: item (closed format) Long open-ended response: task Receptive skills lend themselves to assessment by closed item formats; productive skills to assessment by open-ended formats.

  26. Thank you!

  27. Developing Reading Items Read CEF descriptors. Read through the input texts and identify sub-competences in these descriptions for which you then try to write reading comprehension items. Fill in the Item Characterisation Form after you have finished an item.

  28. Developing a Writing Task Develop a writing prompt plus a scale for assessing the student performances originating from the prompt. Prompt Decide on a topic area. Decide on a text type and addressee. Formulate a concrete task. Decide on required length in number of words. Scale Decide on the aspects (dimensions) which determine the quality of the performance. Describe two levels on each dimension: bare pass and excellent. Finally, describe the test construct (The ability to ...).

More Related