"The Design and Use of Multiple Choice Tests"

"The Design and Use of Multiple Choice Tests"

Multiple Choice Exams “ Written properly, multiple choice exams correlate strongly with assessments by descriptive tests ” Brown, Robert, “Multiple Choice Versus Descriptive Tests”, Frontiers in Education Conference, Reno NC 2001.

Use of Multiple Choice Exams Not limited to Factual Knowledge • Making inferences • Application of models • Evaluation

Multiple Choice vs. Descriptive Examinations

Structure of a Multiple Choice Question Stem Answers

Formulating GoodMultiple Choice Questions • Wording the stem so that there is a single correct answer • Finding plausible wrong answers • Sensitivity to ESL students • Negative voice in the stem • Conditional “if… would” structures • Cultural contexts for questions

Stems Statements “A river that flows through a desert is called..” Questions “ Which of the following descriptions best fits the character of Alsatian wines?” Negative voice “ Which of the following grapes is not used to make Amarone?”

Stems Conditional Voice “ If you want a VQA Niagara red wine that was very fruity and required only a short ageing period you would select: ”

Answer Options • Single correct answer options • Collective answer options: “ (a) and (c) are both correct ” “ all of the above are correct ” “ none of the above are correct ”

Plausible Answer Strategies • Good questions contain several plausible answer options • Grey area between designing “tricky” questions and providing answer options that test knowledge thoroughly. • Inadvertently giving away the answers

Giving away the answer… A verbal question: The forbidden fruit that Eve offered Adam was an: (a) apple (b) pear (c) banana (d) melon (e) carrot

Giving away the answer i.e. a numerical question: It costs a vendor $1.00 each time s/he sells a hot dog to a customer. If the price of the hot dog is $3.00, what is the contribution margin percentage? (a) 17.5% (b) 22.5% (c) 47.5% (d) 66.7% (e) 82.0%

“Plausible” Answer Strategy It costs a vendor $1.00 each time s/he sells a hot dog to a customer. If the price of the hot dog is $3.00, what is the contribution margin percentage? (a) 300% (b) 200% (c) 67.7% (d) 50% (e) 33.3%

Plausible Answer Strategies Use Nonsense Words : Colio Wines in Ontario produce an interesting product classified as Vin de Curé in much the same way as Amarone is made in Veneto. The Italian word describing the drying process used to desiccate the grapes is: a) Ripasso b) Appasimento c) Raisinato d) Pommace e) Prunello

Plausible Answer Strategies Use related, but different terms…. The word that refers to the addition of sugar to a finished wine is: (a) Amelioration (b) Chaptalization (c) Edulcoration (d) Elaboration (e) Oxidation

Adding Uncertainty.. The term "fortified wine" means: a) the bottle had been reinforced b) the wine has had tannins and acid added c) the wine has had sugar added d) the wine has had SO2 added e) none of the foregoing

Adding difficulty.. Choose the best answer. Rootham’s, a well-known local manufacturer of brandied fruit preserves sells their product through gourmet food shops and at craft fairs throughout Canada. Their recipes use genuine brandy and all natural ingredients. They are able to sell these legally because: (a) the alcohol used is produced by fermentation (b) the product is made with ethyl alcohol (c) the alcohol used is purchased from the LCBO (d) the alcohol concentration is less than 1.2% (e) the label says “made with genuine brandy”

Making Multiple Choice Questions Interesting… A gracious, but parsimonious hostess wants to receive guests in the private dining room of your restaurant with a glass of champagne just to tantalise the taste buds and set the mood for the evening. 1.You suggest that if the wine is being poured at the bar, she should consider serving a Canadian sparkling wine that is very similar to Champagne in quality but considerably less expensive. Such a wine would be a:

2. The cheapest sparkling wine would be obvious to sophisticated guests because they would know that in the best sparkling wines… 3. If the gracious hostess insists on having “French” Champagne and you are left to rationalise costs, the most cost effective solution is to serve Champagne from..

4. Occasionally, a patron asks you to do something that will distinguish their event from others. One client wants to obtain a large bottle of Champagne to celebrate a particular event. Which is the largest bottle of champagne on the list below? 5. If someone wanted to serve a "glass of bubbly" during a reception, which of the following would be the most inappropriate type of wine to select as the pre-prandial (before dinner) libation?

Evaluating Multiple Choice Questions Difficulty Factor = R/N R = # of times the correct answer is selected N = # of people taking the exam Too Easy Too Difficult 0 0.5 + 1

Difficulty Factor Values greater than 0.8: • The question is too easy • Stem wording gives away the answer • Not enough plausible incorrect answer options Values less than 0.2: • Guessing effect can be a factor • The question is defective • Material wasn’t covered • Stem wording is unclear, confusing • Answer options need work

Evaluating Multiple Choice Questions Discrimination Index = Proportion of capable students that select an answer option – proportion of weak students that select the answer option i.e. D = Ftop quarter – Fbottom quarter Good Discrimination Reverse Discrimination - 1 0 + 1

Discrimination Index For the Correct Answer Option Values ~ 0.5 give good discrimination < 0.2 consider dropping negative… drop the question For Incorrect Answer Options Values should be negative.

Knowledge or Successful Guessing? Multiple Choice Exam Strategies • Improve odds by eliminating 1 or more infeasible or unlikely answer options Descriptive Exam Strategies • Brain dumping • Part marks • Consideration for perfect answers to questions that were not asked

Possibility of a “Random Pass” Depends on the number of answer options per question and the number of questions!

from Brown, 2001

Adjustment for Guessing Negative Marking… • Elimination strategy reduces odds of wrong answer penalty • Subtracting a percentage of the number of wrong answers obtained from the final grade • Give a grade of 4 for a correct answer and a score of -1 for a wrong answer on a 4 choice question

Negative Marking.. • A score of less than zero is possible • Students hate negative marking • Negative marking is not practised in descriptive examinations • A poor substitute for a test that is too short with too few answer options

Teaching WithMultiple Choice Questions • Prepare lectures incorporating all material covered in the questions to be used • After each lecture, check coverage by reviewing questions • Provide students with multiple choice questions for self-evaluation at the end of readings

Teaching WithMultiple Choice Examinations Use stem questions to teach additional or ancillary terms not covered in the course: i.e. …. For post-prandial (after dinner) drinks you want to stock a supply of fortified wines and brandy. The term "fortified wine" means….

Provide Students the Stem Questions Prior to the Exam! • Exam questions should cover all the course material being tested • Provides more detail than a course outline • Students focus their study effort on the topics they understand the least • Increases learning • Reduces stress

Thanks for your interest in my topic! I look forward to your questions and ideas. J.E. (Joe) Barth School of Hospitality and Tourism Management University of Guelph, Ontario CANADA

"The Design and Use of Multiple Choice Tests"