The Exam Learning Through Questions

1. The Exam Learning Through Questions by: Anthony Herben Training Design Manager Air Canada Cargo

2. The Exam – Learning Through Questions • This presentation will accomplish three outcomes: • Instructors will know and follow appropriate principles for developing and using questions for their exams and avoid common pitfalls. • Instructors will be able to identify and accommodate the limitations of different question methods. • Instructors will gain an awareness that certain exam questions can be incompatible with certain instructional goals.

3. The Exam – Learning Through Questions Developing an Exam • Most classroom exams are developed for one or more of the following purposes: • To establish a basis for assigning grades. • To determine how well each student has achieved the course objectives. • To determine where lessons need improvement.

4. The Exam – Learning Through Questions • Developing a good exam is like target shooting. Hitting the target requires planning. You must choose a target, select an appropriate arrow and take careful aim. • Developing a good exam also requires planning. You must determinethe purpose for the exam and carefully write appropriate exam questions to achieve that purpose.

5. The Exam – Learning Through Questions What can you do to increase the likelihood of obtaining more valid exam scores? • You should make sure your instructional objectives and the exam questions are congruent. For example, if one of your instructional objectives is for students to apply principles to new dangerous goods packaging, then your exam questions should match your objectives by providing opportunities for students to exhibit how to package dangerous goods items.

6. The Exam – Learning Through Questions How can you arrange the exam question to facilitate scoring? • Space exam questions so they can be read, answered and scored with the least amount of difficulty. Double-space between questions. • Place answer spaces for questions in vertical columns for easy scoring with each answer space clearly associated with the corresponding question. • Provide adequate space for students to supply answers to questions.

7. The Exam – Learning Through Questions Length of Exams • In theory, the more questions an exam has, the more reliable it is. • On a short exam a few wrong answers can have a great effect on the overall results. On a long exam, a few wrong answers will not influence the results as much. • If an exam is too long, students may get tired and not respond accurately or seriously. If an exam needs to be lengthy, divide it into sections with different kinds of questions to maintain the student’s interest.

8. The Exam – Learning Through Questions Proofread the Exam • It is important to proofread exams carefully and when possible, have another person proofread them. • Tiny mistakes, such as miss-numbering the response can cause big problems later. • Collation should also be checked carefully, since missing pages can cause a great deal of trouble.

9. The Exam – Learning Through Questions One Wrong Answer • It is wise to avoid having questions depend upon answers required in previous questions. • A student’s initial mistake will be perpetuated over the course of succeeding questions, penalizing the student repeatedly for one error.

10. The Exam – Learning Through Questions A Little Humour • Using a little humor or placing less difficult questions at the beginning of an exam can help students with exam anxiety reduce their preliminary tension.

11. The Exam – Multiple Choice Anatomy of a Multiple Choice Question • A standard multiple choice question consists of two parts: a problem (stem) and a list of suggested solutions (alternatives). • The stem may be in the form of either a question or an incomplete statement. The list of alternatives contains one correct or best alternative (answer) and a number of incorrect or inferior alternatives (distractors). • The purpose of the distractors is to appear as plausible solutions to the problem for those students who have not achieved the objective being measured by the exam question.

12. Multiple ChoiceQuestions

13. The Exam – Multiple Choice Questions Strengths of Multiple-Choice Questions • Versatility in measuring all levels of cognitive skills. • Permit a wide sampling of content and objectives. • Provide high reliable exam scores. • Can be machine-scored quickly and accurately. • Reduced guessing factor compared with true-false questions.

14. The Exam – Multiple Choice Limitations Versatility • Since the student selects a response from a list of alternatives rather than supplying or constructing a response, multiple choice questions are not adaptable to measuring certain learning outcomes, such as the student’s ability to: • display thought processes • furnish information • produce original ideas • provide examples

15. The Exam – Multiple Choice Difficulty of Construction • Good multiple choice questions are generally more difficult and time-consuming to write than other types of questions. • Coming up with plausible distractors requires a certain amount of skill. • This skill may be increased through study, practice and experience.

16. The Exam – Multiple Choice • One of the reasons why some Instructors dislike multiple choice questions is that they believe these are only good for measuring simple recall facts. • This misconception is understandable because multiple choice questions are frequently used to measure lower-level objective, such as those based on knowledge of terms, facts, methods and principles. • The real value of multiple choice questions is their applicability in measuring higher-level objectives, such as those based in comprehension, application and analysis.

17. The Exam – Multiple Choice Questions Guidelines for Writing Questions The challenge is to write questions that examines a significant concept that are unambiguous. • Constructing good multiple choice questions requires plenty of time for writing, review, and revision.

18. The Exam – Multiple Choice Questions Writing the Stem • The “stem” of a multiple-choice question poses a problem or states a question. The basic rule for stem-writing is that students should be able to understand the question without reading it several times. • Write the stem as a single, clearly-stated problem. Direct questions are best • State the question as briefly as possible, avoiding wordiness and undue complexity • State the question in positive form because students often misread negatively phrased questions

19. The Exam – Multiple Choice Questions • The stem should fully state the problem and all qualifications. To make sure that the stem presents a problem, always include a verb in the statement • Write questions that measure the students’ ability to comprehend, apply, analyze and evaluate as well as recall

20. The Exam – Multiple Choice Questions Writing the Response • Multiple-choice questions usually have four or five options to make it difficult for students to guess the correct answer. The basic rules for writing responses are: • students should be able to select the right response without having to sort out complexities that have nothing to do with knowing the correct answer and • they should not be able to guess the correct answer from the way the responses are written

21. The Exam – Multiple Choice Questions • Write the correct answer immediately after writing the stem and make sure it is unquestionably correct. • Write the incorrect options to match the correct response in length, complexity, phrasing, and style. Avoid using terminology that is completely unfamiliar to students. • Avoid using “all of the above” or “both A & B” as responses, since these options make it possible for students to guess the correct answer with only partial knowledge. • Use the option “none of the above” with extreme caution. • Avoid giving verbal clues that give away the correct answer using absolute terms (e.g.: “always, never, all,”) in the distractors and using two responses that have the same meaning.

22. The Exam – Multiple Choice Questions General Issues • All questions should stand on their own. • Avoid using questions that depend on knowing the answers to other questions on the exam. • Check your exam to see if information given in some questions provide clues to the answers on others. • Randomize the position of the correct responses.

23. The Exam – Multiple Choice Questions Limitations of Multiple-Choice Questions • Difficult and time-consuming to construct. • Depends on student’s reading skills and instructor’s writing ability. • Ease of writing low-level knowledge questions leads instructors to neglect writing questions to examine higher-level thinking. • May encourage guessing (but less so than true-false).

24. The Exam – Multiple Choice Questions • Avoid verbal associations between the stem and the correct option (e.g.: the same reference word should not appear in the stem and an option). • Make sure that the options are grammatically consistent with the stem. • Avoid making the correct answer markedly longer or shorter than the other options. • If there is a logical sequence where the alternatives can be arranged (alphabetical if a single word, in order of magnitude if numerals, in temporal sequence, or by length of response), use that sequence. • Avoid the use of language that your students won’t understand. For example, use “cause” instead of “raison d’ etre” in the question.

25. The Exam – Multiple Choice Questions • Randomly distribute the correct response among the alternative positions throughout the exam. That is, have approximately the same proportion of As, Bs, Cs, Ds, and Es as the correct response. • Watch for specific determiners such as “all”, “always”, “never” that are more likely to be incorrect options. Others like “usually” and “sometimes” are more likely to be the key response. • Multiple-choice items should be independent. That is, an answer to one question should not depend on the answer to another question.

26. The Exam – Multiple Choice Questions • State questions so there can be only one interpretation of their meaning. • Poor: Which one of the following is the best source for applying a packing group? a. Classifications b. Table 3.3.A c. Table 3.6.A d. Table 3.8.A • The answer would depend on how one interprets the question. Are we talking about the source for a Flammable Liquid, Toxic Substance or a Corrosive material? • Better: The source for applying a Flammable Liquid packing group is a. Classifications b. Table 3.3.A c. Table 3.6.A d. Table 3.8.A

27. The Exam – Multiple Choice Questions • Present students with a diagram and ask for application, analysis, or evaluations (e.g.: “What is wrong with this Shipper’s Declaration?”) • Use pictorial materials that require students to apply principles and concepts (e.g.: ULD tags, Hazmat Registration Certificate, Training Certificate, etc.). • Use charts, tables or figures that require interpretation (e.g.: Precedence of Hazards Table, Segregation Table, etc.).

28. True-FalseQuestions

29. The Exam – True–False Questions • The true-false questions typically presents a declarative statement that the student must mark as either true or false. • True-false questions measures the recall of factual knowledge. • This format has the potential to measure higher levels of cognitive ability. Strengths of True-False Questions • They are relatively easy to write and can be answered quickly by students. Students can answer 50 true-false questions in the time it takes to answer 30 multiple-choice questions. • They provide the widest sampling of content per unit of time.

30. The Exam – True–False Questions Limitations of True-False Questions • The problem of guessing is the major weakness. Students have a fifty-percent chance of correctly answering a question without any knowledge of the content. • Questions are often ambiguous because of the difficulty of writing statements that are unequivocally true or false.

31. The Exam – True–False Questions WritingTrue-False Questions • Write significant content and avoid trivial statements. • Write questions that can be classified unequivocally as either true or false. • Avoid taking statements verbatim from the Regulations. T F Poor: Each Unit Load Device containing dangerous goods must clearly display on its exterior an indication that dangerous goods are contained within the Unit Load Device. T F Better: When loading dangerous goods into a Unit Load Device, place a tag on the exterior of the unit to indicate that the ULD contains dangerous goods.

32. The Exam – True–False Questions • Include only a single major point in each question. • Avoid trick questions. T F Poor: Dangerous goods can be carried in the passenger cabin and the flight deck of a passenger aircraft T F Better: Dangerous goods can be carried in the flight deck of a passenger aircraft • The intent of the question is not to confuse the student by asking two questions in one. The first can be answered both true and false.

33. The Exam – True–False Questions • Avoid using words like “always”, “all” or “never” that tend to make the statement false. • Words like “usually”, “often”, “many” usually makes the statement true. • Avoid using negatively worded statements. T F Poor: Biological substance, category B need not appear on the Notification to Captain form T F Better: Dangerous Goods in Excepted Quantities is required on the Notification to Captain form • Put the questions in random order so as to avoid response patterns that could serve as clues (such as T, T, F, T, T, F).

34. The Exam – True–False Questions • Avoid long drawn-out statements or complex sentences with many qualifiers. • Avoid making questions that are true consistently longer than those that are false. • Use slightly more false questions than true questions. False questions tend to discriminate more highly among students than do true questions.

35. MatchingQuestions

36. The Exam – Matching Questions • A matching exercise typically consists of a list of questions or problems to be answered along with a list of responses. The student is required to make an association between each question and a response. • Because matching questions permit one to cover a lot of content in one exercise, they are an efficient way of measurement. It is difficult, however, to write matching questions that require more than simple recall of factual knowledge.

37. The Exam – Matching Questions Guidelines for constructing matching questions • Include directions that clearly state the basis for the matching. Inform students whether or not a response can be used more than once and where answers are to be written. • Put the problems or the stems (typically longer than the response) in a numbered column on the left, and the response choices in a lettered column on the right. Because the student must scan the list of responses for each problem, one should keep the responses brief.

38. The Exam – Matching Questions • Always include more responses than questions. If the lists are the same length, the last choice may be determined by elimination rather than knowledge. • Arrange the list of responses in alphabetical or numerical order if possible in order to save reading time. • All response choices must be credible, but make sure that there is only one correct choice for each stem or numbered question.

39. CompletionQuestions

40. The Exam – Completion Questions • The completion format requires the student to answer a question or to finish an incomplete statement by completing a blank with the correct word or phrase. The advantages of completion questions are: • they provide a wide sampling of content • they minimize guessing compared with multiple-choice and true-false.

41. The Exam – Completion Questions The limitations of completion questions are they: • rarely can be written to measure more than simple recall of information • are more time-consuming to score than other objective types • are difficult to write so there is only one correct answer and no irrelevant clues.

42. The Exam – Completion Questions Guidelines for Writing Completion Questions • Write completion questions that have a single correct answer, if possible. • Use blanks of the same length throughout the exam so that the length is not a clue. • Avoid grammatical clues to the correct response. For example, if the indefinite article is required before a blank, use a(n) so that the student doesn’t know if the correct answer begins with a vowel or a consonant.

43. The Exam – Completion Questions • If possible, put the blank at the end of a statement rather than at the beginning. Asking for a response before the student understands the intent of the statement can be confusing and may require more reading time. • Avoid taking statements directly from the Dangerous Goods Regulations.

44. for joining me in my presentation about The Exam Learning Through Questions Now is the time for your questions Handouts are available Thank you

45. The End By: Anthony Herben