MCQs Made Easy

1. MCQs Made Easy Christine Coombe Dubai Men’s College

2. Introduction • Why is this topic important for us? • Good tests work for us • Tests underlie decision making • Writing Objective Items • Guidelines and pitfalls • Editing Objective Items • Identifying and solving problems

3. Overview of Workshop • Objective vs. Subjective test formats • Guidelines for common formats • Critiquing a “bad test”

4. Objective short answer closed response mostly recognition, limited production difficult to write well quick and easy to grade reliable workload “up front” Subjective long answer open response emphasis on production relatively easy to write difficult to grade time-consuming inter-rater reliability not as reliable workload post test Objective vs. Subjective formats

5. Thinking about test formats • Cornerstones of testing • problems w/ validity: test what and how you teach • be clear about what and how you assess • reliability, consistency of test scores high • objective formats often practical for administration • washback can be negative without planning • authenticity often lacking, context requires effort • transparencyalways important • Ss need to know what to expect • security enhanced with multiple versions, CBT

6. Why Use MCQs? • If written well, they are reliable. • Only one answer possible • They are cost effective. • Can be easily computer scored • Ss are familiar with the item type • Directions are easy to understand • They are useful at various levels • Does not require extensive writing ability, but training needed

7. MCQ Terminology: How to Sound like an Expert • Item: a question on a test • Stimulus: material used as basis for testing (i.e. Reading passage, sentence etc.) • Stem: the question part of the item • Response Options: all the choices in an item (a, b, c or d) • Key: the correct answer • Distractors: the incorrect response options

8. Multiple Choice Questions (MCQs) • Question should be clearly in stem • Four options optimal, one must be unambiguous answer. Others = distractors which attract Ss • unless using best answer format • All response options are parallel (same length, difficulty level) • Correct answer should require processing, not just “selective verbatim” • Answers should appear equally in all positions • Avoid: “all of the above”, never, always, absurd or giveaway distractors, extraneous cues, items that presuppose correct answer to previous item. • Supporting information and examples • How it relates to your audience

9. Qualities of Good MCQs • Validity: items measure only what they are supposed to measure • Fairness: items do not unfairly advantage or disadvantage certain groups or people • Sensitivity: items avoid offensive or upsetting language, situations or actions • Difficulty: items are written at appropriate levels of difficulty

10. Cueing 3 for 1 Split Impurity Subsuming Apples and Oranges Unparallel options Gender representation Stereotypes Inflammatory tone Double keys No key Implausibility Outside or Background knowledge Trickiness Insensitivity Inappropriate terminology Common Item Problems

11. Cueing • Can occur in two places; sometimes called the extraneous clue • within an item or within the test • When you write “for deposit only” on a check, you are probably going to _______________. • a. cash a check • b. deposit money • c. transfer funds • d. open an account

12. 3 for 1 Split • 3 distractors are parallel; the 4th is not (a.k.a. “odd man out”) • The hotel _______ an ad in the newspaper to recruit new employees. • a. placing • b. placed • c. to place • d. placement

13. Impurity • Items that test more than one thing • I never knew ______________. • a. had the boys gone • b. the boys have gone • c. have the boys gone • d. the boys had gone

14. Subsuming • The key is subsumed into another answer that could be assumed as correct on a different level • John: We need to buy some new office furniture. • Mike: Ok, I’ll call the supply company. • John: Just don’t get the metal ones, please! • What will Mike order? • a. metal desks • b. comfortable chairs • c. office furniture

15. Apples and Oranges • Response options have no relation to one another or are 2 for 2 split • Nowadays people use mobile phones ______________. • a. frequently • b. seldom • c. in their cars • d. for emergency purposes

16. Unparallel Options • Response options are not parallel • length • grammar • How do most Russians go to work? • a. By bus. • b. They go by taxi. • c. Most of them drive. • d. Walk.

17. Gender Representation • Care should be taken with vocabulary relating to gender • The mailman ______ many packages despite the rain. • to deliver • delivering • delivered • deliver

18. Stereotypes • Test materials should avoid language that fosters stereotypes. • Latifa went downtown ____ pick up some soul food. • so • to • for • ---

19. Inflammatory Tone • Material that may have a negative emotional impact on Ss should be avoided. • _______ many American soldiers in the Arabian Gulf right now. • There are • There is • There were • There have been

20. Double Keys • More than one response option is keyable. • The teacher waited in the classroom until the students ____. • a. had come • b. would come • c. came • d. has come

21. No Key • Care should be taken to include a correct answer in response options. • This is the student _____ I told you about yesterday. • what • which • where • how

22. Implausibility • Ss can improve ‘monkey score’ by eliminating absurd or giveaway distractors • The Olympics were held in ______ in 2004. • a. Athens • b. Beijing • c. Seoul • d. Disneyland

23. Outside/Background Knowledge • Ss are able to answer the question based on outside or background knowledge. • The weather in Siberia is _____ hot and sunny. • a. often • b. never • c. sometimes • d. rarely

24. Trickiness • Focus on points that are notoriously confusing even for native speakers. • Smoking has a bad________ on health. • a. effect • b. affect • c. affliction • d. effectiveness

25. Insensitivity • Items should not be shocking or upsetting to test takers. • Four people ______ in three serious car accidents on Sheikh Zayed Road yesterday. • a. were killed • b. killed • c. are killed • d. to kill

26. Inappropriate Terminology • Avoid politically incorrect terminology • Crippled children can study a variety of different _____ at the new school. • teachers • students • subjects • methods

27. Special Case: Reading MCQs • Standard items in Reading Comprehension • What is the main idea of this passage? • What would be the best title for this text? • What is the topic of paragraph 3? • “Goldilocks” formula • JR = just right (best answer) • TG = too general, broad • TS = too specific, focuses on one detail • OT = off topic although somehow related

28. Disadvantages of MCQs • Not an effective format for testing productive skills • Time consuming to produce good items • Encourages guessing which can skew results • Does not lend itself to the testing of language as communication

29. True/False/Not Enough Information • TFNs often used to test reading comprehension, but N not appropriate for listening • Effective, and easier, quicker to write than MCQs • Guidelines for good TFNs: • Qs should be at lower level of language difficulty than the text • Qs should appear in same order as text • Have Ss circle TFNs, never write letters • Without N option, 50% guessing factor • Paraphrase questions, avoid verbatim matching

30. Matching formats • Matching ranges from mindless POE to intellectually challenging “making meaning” • Ss have more distractors than with MCQs • Format somewhat easier to write • Pointers for writing matching items: • Never write direct 1:1 matching; use extra choices • Use with cloze paragraphs for meaning in context • In 2 column format, use numbers and letters • Use 2 columns sparingly for word association; be sure of syntax between the columns

31. Sentence Completion and Chart Fills • These formats between objective and subjective • Example of objective: extract info to fill chart • Example of subjective: summarize material • How much processing should Ss need to do? • Think of relationship to course objectives and adequate practice opportunities for Ss • In rubric, be clear about expectations for task • Example: for short answers, specify # of words • In preparing answer key, consider alternatives and credit for incorrect transcription, spelling

32. Error Detection/Correction • Popular format for some standardized testing because it emphasizes monitoring, checking • Distinguish between detection and correction • Ss often identify/fix correct items and overlook incorrect ones, leading to scoring problems • If correction required, what level of accuracy? • Error correction more useful for classroom teaching than as a testing format

33. On to the “bad test” • In real life, assessment instruments are often imperfect • It’s often easier to fix a problem than start from scratch • Active way to put concepts into practice • Look for problems and think how to rework them • There are additional problems we haven’t had time to mention