320 likes | 336 Views
This conference in Croatia, July 07-09, 2009, focuses on the shortcomings, advantages, and disadvantages of item moderation in language testing. It aims to offer remedies, successful modification, and warm-up activities for future sessions. Topics include testing intelligence, background knowledge, text selection, rubrics, worked examples, task types, and common problems like MCQ and gap fill.
E N D
SHORTCOMINGS OF ITEM MODERATION Croatia July, 07-09 2009
MAIN OBJECTIVES • OVERVIEW OF SHORTCOMINGS. • ADVANTAGES (REMEDY AND IMPROVEMENT OF ITEMS, SUCCESSFUL MODIFICATION, ETC.) AND DISADVANTAGES (MISCALCULATIONS, TIME AND RESOURCE CONSUMPTION, etc.) • WARM-UP FOR THE FOLLOWING SESSIONS
INTRODUCTION • GENERAL GUIDELINES • PROBLEMS WITH SPECIFIC ITEMS (reading/listening) a. shortcomings b. examples c. ways out • PROBLEMS IN WRITING AND SPEAKING • CONCLUSION
TESTING WHAT IS NOT INTENDED • A. TESTING INTELLIGENCE Eg. Rearrange the following letters to make English words ICBURR METS TTSE RIEWT LOOFWL PAHYP
TESTING WHAT IS NOT INTENDED • B. BACKGROUND KNOWLEDGE eg. Answer the questions after listening to a text about Elvis Presley. 1. Where was he born? ___________ 2. What was his first job? ___________ 3. How did he die? ___________
TEXT SELECTION 1 • a. The target language must be correct and authentic, preferably written by a native speaker for native speakers. • b. Attention must be paid to texts which we edit while we design the task
TEXT SELECTION 2 • c. The text has to be realistically long for the level we want to test and for the time we are assigning to the task. Eg. A 200 word text for only one short answer question is of course exaggerated
TEXT SELECTION 3 • c. We must also consider the most probable pool of test takers, their age, interest and life experience. Eg. The latest pop concert of some teen age pop idol is not very appealing to a group of 45-55 year olds if they are the ones most likely to take the test you are designing
TEXT SELECTION 4 • Using the same text twice is time efficient for item writers • …but it might penalize some candidates who did not understand the text the first time… (…open to discussion)
RUBRICS • Unclear and confusing • Don’t specify the number of words in certain task types eg “Actions speak louder than words” Discuss and illustrate your answer. - Simpler than the item - Format (bold letters, underline, italics) - Native language
WORKED EXAMPLE • Confusing • Not of the same nature as the rest • Misleading eg. 0. Do you want to sell your house? Look at ad 6. 1. If you call +33(0) 4 93 22 1000, you want to _____.
SAME TASK TYPES vs. DIFFERENT TASK TYPES • Same task type leads into building up strategies • In listening, different task types are confusing
Shifting levels • Item more/less difficult than text • Pace not natural (listening)( too slow even by natives when recording texts) • Too much text in the alternatives (not enough time to read)
MCQ 1. MORE THAN ONE CORRECT ANSWER/ DUBIOUS ANSWERS eg. Which is the odd one out? A rabbit B hare C bunny D deer • CHECK WITH COLLEAGUES.
MCQ • 2. TOO GRAMMATICALLY RIGOROUS eg. ‘Why hasn’t your mother come?’ ‘Well, she said she ________ leave the baby.’ A can’t B won’t C couldn’t D mayn’t • CHECK WITH NATIVE SPEAKERS
MCQ • 3. NOT ALL DISTRACTORS EQUALLY ATTRACTIVE • eg. Woman: Will this be enough? Man: Yes, two slices are enough for me. They are talking about _______________. a. water, b. fruit, c. sausages, d. bread -THREE ALTERNATIVES FOR SOME ITEMS
MCQ • NOT ENOUGH CONTEXT IN THE STEM • eg. Come back soon. a)shortly b)later c)today d)tomorrow GIVE CONTEXT, REARRANGE THE ITEM eg. Visitor: Thank you very much for such a wonderful visit. Hostess: We were so glad you could come. Come back __________ a)Soon b)later c)today d)tomorrow
MCQ • Options do not fit equally well into the stem eg. Someone who designs houses is a ____ a) designer b) builder c) architect d) plumber • Put a/an
TRUE/FALSE • 50% CHANCE OF GETTING IT RIGHT • NOT GOOD FOR LISTENING Eg…. ADD. NG/ Does not say
MATCHING • LAST CORRECT BY DEFAULT eg. 1 car A room 2 cup B pet 3 bed C dress 4 night D board GIVE MORE CHOICES IN ONE OF THE COLUMNS
INFO TRANSFER • TOO COMPLICATED / • What should go where?!?!?! EG. Following directions; listening to instructions Moderation????
ORDERING TASKS • More than one right way. • Not clear what is being tested and problematic with marking the answers – not independent eg. a) it was called ‘The Last Waltz’ 1 _D_ b) the street was in total darkness 2 ____ (g) c) because it was one he and Richard had learnt at school 3 ____ (e) d) Peter looked outside 4 ____ (c) e) he recognized the tune 5 ____ (a) f) and it seemed deserted 6 ____ (b) g) he thought he heard someone whistling 7 ____ (f) BUT 1:D, 2:B, 3:F, 4:G, 5:E, 6:C, 7:A . - improve the item by adding ‘but’ to the beginning of phrase (g) - Marking scheme wholly right or wholly wrong (but still not worth the effort)
GAP FILL • The gap should lead to the expected word • More than one correct answer • Does not really test language-the word simply does not spring • Sentence equally good without the deleted word - eg. It happened that the man ______ I was chasing was my boyfriend. - construct a banked gap-filling task - a list with more words than there are gaps
CLOZE • DIFFERENT INITIAL GAPS ON THE SAME TEXT MAKE VALIDITY AND RELIABILITY VARY • AMENDMENTS ARE DIFFICULT -changing the nth word - principle not preserved - rewriting the text - no longer authentic
SHORT ANSWER QUESTIONS • Sometimes too evasive (ss come up with endless versions not anticipated) • Answers tend to be very long (although the rubric might specify the number of words)- leading to problems with marking schemes
WRITING • problems with prompts for writing - easy for the test writer…difficult for the test taker - not appealing enough - too technical for some (civ. in mil.)
SPEAKING • Wrongfully considered as the simplest way to assess language ability - superficial topics → limited vocab/structures - complex language → betrays candidates’ inadequacies • Carefully structured interview-important aspects covered with each candidate
Provide clear instructions (clear and precise) Give example Order items info appears in the text Spread the items evenly throughout the script Restrict the number of possible answers Avoid ambiguity Avoid predictable answers Write good distractors Provide instructions in bold Assure good quality of recordings Test language not the knowledge Be aware of the level Think about marking Avoid abbreviations Cue the listeners in Use clear visuals Avoid distractors that give away the correct answer CONCLUSION - DO’S
CONCLUSION – DONT’S • Don’t test memory • Don’t test background knowledge • Don’t use culturally biased items • Don’t use unfamiliar topics, genres • Don’t write items dependant on each other • Don’t write item on the first sentence • Don’t write items on proper names • Don’t introduce too much of unnecessary information • Don’t write too many items on the same thing e.g. numbers,