1 / 32

SHORTCOMINGS OF ITEM MODERATION

SHORTCOMINGS OF ITEM MODERATION. Croatia July, 07-09 2009. MAIN OBJECTIVES. OVERVIEW OF SHORTCOMINGS. ADVANTAGES (REMEDY AND IMPROVEMENT OF ITEMS, SUCCESSFUL MODIFICATION, ETC.) AND DISADVANTAGES (MISCALCULATIONS, TIME AND RESOURCE CONSUMPTION, etc.) WARM-UP FOR THE FOLLOWING SESSIONS .

pdenney
Download Presentation

SHORTCOMINGS OF ITEM MODERATION

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SHORTCOMINGS OF ITEM MODERATION Croatia July, 07-09 2009

  2. MAIN OBJECTIVES • OVERVIEW OF SHORTCOMINGS. • ADVANTAGES (REMEDY AND IMPROVEMENT OF ITEMS, SUCCESSFUL MODIFICATION, ETC.) AND DISADVANTAGES (MISCALCULATIONS, TIME AND RESOURCE CONSUMPTION, etc.) • WARM-UP FOR THE FOLLOWING SESSIONS 

  3. INTRODUCTION • GENERAL GUIDELINES • PROBLEMS WITH SPECIFIC ITEMS (reading/listening) a. shortcomings b. examples c. ways out • PROBLEMS IN WRITING AND SPEAKING • CONCLUSION

  4. GENERAL GUIDELINES

  5. TESTING WHAT IS NOT INTENDED • A. TESTING INTELLIGENCE Eg. Rearrange the following letters to make English words ICBURR METS TTSE RIEWT LOOFWL PAHYP

  6. TESTING WHAT IS NOT INTENDED • B. BACKGROUND KNOWLEDGE eg. Answer the questions after listening to a text about Elvis Presley. 1. Where was he born? ___________ 2. What was his first job? ___________ 3. How did he die? ___________

  7. TEXT SELECTION 1 • a. The target language must be correct and authentic, preferably written by a native speaker for native speakers. • b. Attention must be paid to texts which we edit while we design the task

  8. TEXT SELECTION 2 • c. The text has to be realistically long for the level we want to test and for the time we are assigning to the task. Eg. A 200 word text for only one short answer question is of course exaggerated

  9. TEXT SELECTION 3 • c. We must also consider the most probable pool of test takers, their age, interest and life experience. Eg. The latest pop concert of some teen age pop idol is not very appealing to a group of 45-55 year olds if they are the ones most likely to take the test you are designing

  10. TEXT SELECTION 4 • Using the same text twice is time efficient for item writers • …but it might penalize some candidates who did not understand the text the first time… (…open to discussion)

  11. RUBRICS • Unclear and confusing • Don’t specify the number of words in certain task types eg “Actions speak louder than words” Discuss and illustrate your answer. - Simpler than the item - Format (bold letters, underline, italics) - Native language

  12. WORKED EXAMPLE • Confusing • Not of the same nature as the rest • Misleading eg. 0. Do you want to sell your house? Look at ad 6. 1. If you call +33(0) 4 93 22 1000, you want to _____.

  13. SAME TASK TYPES vs. DIFFERENT TASK TYPES • Same task type leads into building up strategies • In listening, different task types are confusing

  14. Shifting levels • Item more/less difficult than text • Pace not natural (listening)( too slow even by natives when recording texts) • Too much text in the alternatives (not enough time to read)

  15. PROBLEMS WITH PARTICULAR ITEM TYPES

  16. MCQ 1. MORE THAN ONE CORRECT ANSWER/ DUBIOUS ANSWERS eg. Which is the odd one out? A rabbit B hare C bunny D deer • CHECK WITH COLLEAGUES.

  17. MCQ • 2. TOO GRAMMATICALLY RIGOROUS eg. ‘Why hasn’t your mother come?’ ‘Well, she said she ________ leave the baby.’ A can’t B won’t C couldn’t D mayn’t • CHECK WITH NATIVE SPEAKERS

  18. MCQ • 3. NOT ALL DISTRACTORS EQUALLY ATTRACTIVE • eg. Woman: Will this be enough? Man: Yes, two slices are enough for me. They are talking about _______________. a. water, b. fruit, c. sausages, d. bread -THREE ALTERNATIVES FOR SOME ITEMS

  19. MCQ • NOT ENOUGH CONTEXT IN THE STEM • eg. Come back soon. a)shortly b)later c)today d)tomorrow GIVE CONTEXT, REARRANGE THE ITEM eg. Visitor: Thank you very much for such a wonderful visit. Hostess: We were so glad you could come. Come back __________ a)Soon b)later c)today d)tomorrow

  20. MCQ • Options do not fit equally well into the stem eg. Someone who designs houses is a ____ a) designer b) builder c) architect d) plumber • Put a/an

  21. TRUE/FALSE • 50% CHANCE OF GETTING IT RIGHT • NOT GOOD FOR LISTENING Eg…. ADD. NG/ Does not say

  22. MATCHING • LAST CORRECT BY DEFAULT eg. 1 car A room 2 cup B pet 3 bed C dress 4 night D board GIVE MORE CHOICES IN ONE OF THE COLUMNS

  23. INFO TRANSFER • TOO COMPLICATED / • What should go where?!?!?! EG. Following directions; listening to instructions Moderation????

  24. ORDERING TASKS • More than one right way. • Not clear what is being tested and problematic with marking the answers – not independent eg. a) it was called ‘The Last Waltz’ 1 _D_ b) the street was in total darkness 2 ____ (g) c) because it was one he and Richard had learnt at school 3 ____ (e) d) Peter looked outside 4 ____ (c) e) he recognized the tune 5 ____ (a) f) and it seemed deserted 6 ____ (b) g) he thought he heard someone whistling 7 ____ (f) BUT 1:D, 2:B, 3:F, 4:G, 5:E, 6:C, 7:A . - improve the item by adding ‘but’ to the beginning of phrase (g) - Marking scheme wholly right or wholly wrong (but still not worth the effort)

  25. GAP FILL • The gap should lead to the expected word • More than one correct answer • Does not really test language-the word simply does not spring • Sentence equally good without the deleted word - eg. It happened that the man ______ I was chasing was my boyfriend. - construct a banked gap-filling task - a list with more words than there are gaps

  26. CLOZE • DIFFERENT INITIAL GAPS ON THE SAME TEXT MAKE VALIDITY AND RELIABILITY VARY • AMENDMENTS ARE DIFFICULT -changing the nth word - principle not preserved - rewriting the text - no longer authentic

  27. SHORT ANSWER QUESTIONS • Sometimes too evasive (ss come up with endless versions not anticipated) • Answers tend to be very long (although the rubric might specify the number of words)- leading to problems with marking schemes

  28. WRITING • problems with prompts for writing - easy for the test writer…difficult for the test taker - not appealing enough - too technical for some (civ. in mil.)

  29. SPEAKING • Wrongfully considered as the simplest way to assess language ability - superficial topics → limited vocab/structures - complex language → betrays candidates’ inadequacies • Carefully structured interview-important aspects covered with each candidate

  30. Provide clear instructions (clear and precise) Give example Order items info appears in the text Spread the items evenly throughout the script Restrict the number of possible answers Avoid ambiguity Avoid predictable answers Write good distractors Provide instructions in bold Assure good quality of recordings Test language not the knowledge Be aware of the level Think about marking Avoid abbreviations Cue the listeners in Use clear visuals Avoid distractors that give away the correct answer CONCLUSION - DO’S

  31. CONCLUSION – DONT’S • Don’t test memory • Don’t test background knowledge • Don’t use culturally biased items • Don’t use unfamiliar topics, genres • Don’t write items dependant on each other • Don’t write item on the first sentence • Don’t write items on proper names • Don’t introduce too much of unnecessary information • Don’t write too many items on the same thing e.g. numbers,

  32. Final comments - questions??????

More Related