Writing Effective Multiple Choice Questions Carlinda D’Alimonte Thanks to Alice Aspinall for her help with preparing th

1. Writing Effective Multiple Choice Questions Carlinda D’AlimonteThanks to Alice Aspinall for her help with preparing this presentation.

2. Rationale for Training EQAO Test Questions Poor performance of students on MCQs Incongruence Practice for standardized tests An opportunity to respond to well constructed multiple-choice questions improves performance 2

3. Our Focus • Student’s Process • Structure of the Question • Cognitive Process

4. Process It is helpful to consider how students should approach multiple choice questions on a test/exam.

5. Three-Step Process • Before they attempt any of these, read the passage for basic understanding. Step One

6. Step Two Read the first question. Does the student know the answer? If yes, read through all the choices and make sure they select the the most correct answer. Do not choose the first correct answer they see.

7. Step Three If they do not know the answer right away: • read the four choices • cross out obviously wrong answers • highlight key words in the question and scan the text for those words to try to find the right answer Be aware that some answers may seem correct, but there is only one best answer.

8. Further Instructions • Now go back to any questions that they circled and left unanswered. • If time permits, repeat the steps to answering multiple choice questions and review the passage. • Always make a selection. Never leave an answer blank.

9. Repeat this three-step process with each question in the passage.

10. Review – Answering MC Questions

11. Structure

12. Anatomy of a Multiple-Choice Question

13. Varieties of MC Questions • Single Correct Answer • Single Best Answer – Our Focus Today • Negative – Which of the following is NOT … • Multiple Response

14. Structure: How the Question is Constructed • Use questions instead of incomplete statements • Base each item on a specific problem/issue stated clearly in the stem. The student should be able to answer the question before reading the possible answers or have a clear, specific idea of what is being asked • Include as much of the item as possible in the stem, but do not include irrelevant material • Use plausible distracters and word them clearly and concisely • Keep the alternatives mutually exclusive • Keep alternatives homogeneous in content • Keep options at a similar length • Be grammatically correct • Avoid clues to the answer • Use only “most correct” answer • Do not use “all of the above,” “none of the above,” or negative questions or answers • Give clear instructions

15. Multiple Choice Questions Think of two types of questions. 1. Questions whose answers are easily determined. In the case of a reading passage the answer is found right in the text. - easiest to write - measure knowledge only 2. Questions whose answers require students to infer – that is, read between the lines or extend beyond what is written in the passage - take time to construct - measure higher level thinking

16. 16

17. Making an Inference Means….. • Drawing conclusions • Making deductions • Making conjectures • Speculating • Making presumptions • … and more 17

18. Cognitive Process Emphasize Higher-Level Thinking Think of Bloom’s Taxonomy

19. Bloom’s Taxonomy "Big Dog & Little Dog's Performance Juxaposition." Bloom's Taxonomy of Learning Domains. N.p., 26/05/2009. Web. 16 Apr 2010. <http://nwlink.com/~Donclark/hrd/bloom.html>. "Bloom's Taxonomy." Univeristy of Victoria Student Services. University of Victoria, 17/12/2009. Web. 16 Apr 2010. <http://www.coun.uvic.ca/learning/exams/blooms-taxonomy.html>.

20. What MCQ’s Can Assess 20

21. Generalizations:A General Thought, Idea or Premise Arrived at as a Result of or Based on Specific Information • Which word best describes the main character? the central conflict? the initial problem? 2. What is most likely the theme of the story? • Nice guys often finish last • Kindness is not always its own reward • Kindness is its own reward • Time really does heal 3. What are paragraphs 3 and 4 mainly about? or What is the passage/selection/paragraph mainly about? 4. Which statement best describes the message of the article? • The Iraqi people are upset that the American soldiers are here. • The situation in Baghdad has resulted in violence erupting between Iraqi’s and US soldiers. • Leadership in the capital has disappeared. • The situation in Baghdad is still not secure.

22. Summarizations:A Statement Which Specifies the Main Idea or Main Points___________________________ • Which is the best summary of the article/paragraph/excerpt? • There are many types of jobs and many people looking for work. It is important to find the best person suited to the job. • Unemployment is on the rise. There are many unemployed people who are applying for different types of jobs. • Job openings are on the rise. It is important that the unemployed work hard to try to find a job. • Unemployment is on the rise. Job openings are on the decline due to many economic factors • Which statement best summarizes the main character’s thoughts/actions/problem-solving processes, etc. when ___________? • What is this passage mostly about?

23. Conclusions:A Judgement Formed After Critical Thought___________________________ • What might the reader conclude based on John’s actions? • John is unable to control his anger • John is very passionate • John cannot accept the loss of his brother • John needs the support of his family. • Which author/playwright would be most likely to make this statement about his text? “I believe that the common man is as apt a subject for tragedy in its highest sense as kings were.” • Miller • Orwell • Shakespeare • Checkov • Based on the data in the chart/graph/data set, what might one conclude?

24. Motives or Reasons:Consider Actions, Ideas, Principles, etc.___________________________ • What is the most likely reason Tom walked out of the meeting? • He was afraid of being confronted about his previous actions • He did not feel prepared and needed time to consider how he would handle his presentation • He was angry about what had happened in his boss’s office that morning • He remembered that he needed to make an urgent phone call • Which of the following best justifies Fred Scott’s decision to sue his employer? • According to the article, why are teens texting more and blogging less? • Teens like texting better • There is an explosion in social networking • In 2006, 28% of teen internet users blogged • Older people are becoming more comfortable with blogging • Which statement best describes the role a circle plays in being able to find the area of a minor segment?

25. Judgements/Predictions:Determinations Related to What Might Happen or Occur Next___________________________ • What will most likely be the effect of Fortinbras’ leadership in Norway? • He will be a stabilizing influence • He will involve Norway in more wars • He will be greedy in his approach to leading • He will have a destabilizing influence 2. Based on Anne’s past actions what is she most likely to do if she …

26. Relevance or Significance:___________________________ • What does the robe symbolize in the passage? a. Love b. Greed c. Charity e. Hope • How is the river important to the story? • How has postmodernism shaped film over the past twenty years?

27. Justification/Evaluation:Examination of Methods, Procedures, Solutions___________________________ • Which statement best supports the ideas contain in paragraph 9? • Which statement is the best assessment of …? • How would you evaluate the strength of a thesis statement? • It is meaningful, defensible, and controversial • It is meaningful, direct, and broad in its scope • It is meaningful, specific, and phrased as a question • It is meaningful, defensible, and specific • Which of the following is the best procedure to use to determine the effectiveness of supporting points?

28. Support:Reasoning or Evidence Behind Ideas___________________________ • Which statement best supports the idea that…? • Which of the following actions best supports the idea of John’s impulsiveness? a. He skipped school often b. He ignored his responsibilities c. He was led to believe he could trust James d. He often regretted his choices • What statement best illustrates ideas explored in Theatre of the Absurd? a. Life is meaningless b. People avoid taking responsibility for their lives c. Events are essentially out of our control d. Life’s absurdity if fun

29. Authorial Choices/Text Features:Ideas About the Writer’s Purpose, Techniques or Use of Format___________________________ • What is the most likely reason the author uses slang/a diary format/subtitles/italics/graphs/etc? • In Macbeth, what purpose is best served by the repetition of the word tyrant in Act 5? • It expresses the sentiments of those under Macbeth’s power • It emphasizes the transformation in Macbeth’s character • It indicates Macbeth’s pending demise • It supports the idea that, like Macbeth, we are all tyrants • What purpose is best served by the semicolon/colon/brackets/bold dash/quotation marks/etc. in paragraph 5? • What is the most likely reason that the author included paragraph 4 in this passage? • It provides a counter argument • It provides more background information • It shows that she is being fair • It creates doubt in the reader’s mind

30. Application:The Use of Previously Learned Information___________________________ 1. Which set of questions would best guide your writing of a character analysis? 2. Which of the following statements uses the verb to lie correctly? a. I’m going to lay down for a while. b. She likes to lay around all day. c. Let’s lie in the grass all afternoon. d. I’m going to lie this book on the table for now. 3. Which of the following shows the best use of a semi-colon? • By telling the story from the perspective of the protagonist, Walker create4s empathy for the main character; we empathize with the narrator’s history, weaknesses, and intentions. • By telling the story from the perspective of the protagonist, Walker creates empathy for the main character; she also sets the story in a run down house in southern United States. • By telling the story from the perspective of the protagonist, Walker creates empathy for the main character; the main character struggles with her decision at the climax of the story. • By telling the story from the perspective of the protagonist, Walker creates empathy for the main character; the tone is both bitter and nostalgic, creating a tense mood.

31. Comparisons:___________________________ • To what popular children’s story might Hamlet best be compared? a. Little Red Riding Hood b. Cinderella c. The Lion King d. Chicken Little • To what might _________ best be compared? • Which of these characters best contrasts Hamlet? • Julius Caesar • King Lear • Romeo • Macbeth

32. Interpretations/Explanations of Cause and Effect Relationships:Describing Outcomes or Impact of Events or Situations ___________________________ • Which of the following best describes the effect of _________on _________? • What was the cause of Hamlet’s inability to kill Claudius in the church? a. Over-thinking b. Cowardice c. Anger d. Grief • Based on the context of the word “shenanigans” in the article, what would be its best definition? a. Tricky or questionable conduct b. Athletic ability c. High energy level d. Fair and generous practices

33. Improving Questions • What marks the climax of a Shakespearean tragedy? • When the tragic hero commits an error of judgement • When the tragic hero recognizes his error • When the tragic hero first responds to the generating incident • When the tragic hero dies Suggestions for improvement: Include as much information in the stem so as to avoid unnecessary repetition in the answers A better way to phrase this question might be as follows: It is the behaviour of the tragic hero that marks the climax in a Shakespearean tragedy. Which of the following best describes the point at which the hero’s fortunes begin to decline?

34. Improving Questions 2. A word used to describe a noun is called an • Adjective • Conjunction • Pronoun • Verb The word “an” gives the answer away. Also it would be better for the student to apply knowledge of adjectives to a passage. This question could be more effectively phrased in this way: In the following sentence the words that add descriptive detail to the images are what part of speech? He remembered the quaint East India Tea House at the fair, the polished sandalwood, the tightly draped turbans, and the loose robes, the cool interior and the sweet smell of India tea.

35. Improving Questions • Hamlet • Is the Thane of Denmark • Is awkward when he rehearses with the player • Never demonstrates his leadership abilities • Is clearly angry with Ophelia Suggestions for improvement: The stem is vague and broad. The answers are not homogenous. The student should know what specifically is being asked after reading the stem. A way to improve this question would be as follows: Hamlet is one of Shakespeare’s most complex characters. Which of the following statements best reflects his complexity? • He is indecisive when he rehearses with the players, yet he is decisive with the pirates who take him captive • He is confident and forthright when he rehearses with the actors, yet he is indecisive with the pirates take him captive • He is accepting of his mother’s marriage, yet he cannot accept Claudius as his stepfather • He is confident and forthright with the players and with the pirates, yet he is indecisive about killing Claudius

36. Your Task Work in a small group to write two higher-level multiple-choice questions that will measure a part of any two objectives. Use the Planning Chart to write your questions and determine which higher order thinking task you are tapping. Refer to the sample questions. Write structurally solid questions that force the student to use higher order thinking .

37. Worksheet 37

38. Example • Which of the following best describes the reason it is important to follow the directions listed in the passage? • To be able to accurately complete income tax forms • To be able to learn to file taxes before deadline • To ensure the government will not do an audit • To ensure the correct information is on all forms x X X 38

39. Share Your Favorite Question Tell …. What higher order thinking skills you are tapping, your question and answer choices. 39

40. ClassMarker • http://classweb101.pbworks.com/Online-Testing-Sites • http://www.classmarker.com/ • http://walkliteracy.pbworks.com (Download this and other material here.)

41. Works Cited Burton, Steven. Brigham Young University, 1991. Web. 16 Apr 2010. <http://testing.byu.edu/info/handbooks/betteritems.pdf>. Scott, Jana L. Making Inferences. University of Missouri, n.d. Web. 16 Feb 2010. <www.av.k12.mo.us/main/files/Making_an_Inference.ppt >. 14 Rules for Writing Multiple-Choice Questions. Brigham Young University. Web. 12 Jan. 2010. “Bloom's Taxonomy of Learning Domains.” Big Dog & Little Dog's Performance Juxaposition.". N.p., 26/05/2009. Web. 16 Apr 2010. <http://nwlink.com/~Donclark/hrd/bloom.html>. "Bloom's Taxonomy." Univeristy of Victoria Student Services. University of Victoria, 17/12/2009. Web. 16 Apr 2010. <http://www.coun.uvic.ca/learning/exams/blooms-taxonomy.html>.