Selected-Response Test Items Multiple-Choice (MC) and Multiple-Response (MR)

1. Selected-Response Test ItemsMultiple-Choice (MC) and Multiple-Response (MR) Dr. Belal Hijji, RN, PhD 22/05/2011

2. Learning Outcomes At the end of this presentation, participants will be able to: Realise the wide gap between universities in the ME and those in the developed countries in relation to studies focusing on educational measurement. Discuss the advantages and disadvantages of multiple-choice items. Identify the parts of a multiple-choice item. Discuss the challenges that face nursing educators when writing multiple-choice items. Identify the rules and principles of writing a MC item’s stem and alternatives.

3. Rationale for the Presentation • It is known that nursing educators from the United States of • America had problems with constructing multiple-choice • items. In a study by Masters et al. (2001), 2,233 minor and • major violations of item-writing guidelines were identified. • Our question would then be: • What is it like when a nursing educator with English as a second language embarks on constructing multiple-choice items without prior training? 3 3

4. Rationale for the Presentation (Continued ….) What is also known is that studies focusing on educational measurement were published from many countries and across various disciplines (Ellsworth et al., 1990; Hansen, 1997; Masters et al., 2001; Jozefowicz et al., 2002; Downing, 2005; Shizuka et al., 2006; Tarrant et al., 2006; Tarrant et al., 2009). However, similar publications from our region are lacking, although the need for this type of studies is more visible. Our question would then be: Should not we investigate the quality of our measurement practice as a critical element of academic performance? 4 4 4

5. Advantages of Multiple-Choice Items Knowledge of facts, specific information, and principles Definitions of terms Understanding of content Application of concepts in clinical situations Analysis of data and clinical situations Comparison and selection of varied interventions Judgments and decisions about actions to take in clinical situations

6. Disadvantages of Multiple-Choice Items These items are difficult to construct at higher cognitive levels. As such, many of them are written at the lower cognitive levels. Teachers often experience difficulty developing plausible distractors. It is often difficult to identify only one correct answer for an item. As a result, these items are time-consuming to construct. What would further compound these disadvantages is the fact that, in our region, multiple-choice items could be written by nursing educators whose level of English language proficiency is inadequate.

7. Parts of Multiple-Choice Item An early and common sign STEM in form of of pregnancy is: incomplete statement Options or Alternatives a. amenorrhea Answer b. morning sickness Distractor c. spotting Distractor d. tenderness of the breasts Distractor

8. What is the Major Challenge for Nursing Educators in ME universities? Nursing educators should demonstrate their ability to adhere to the various principles and rules of writing each of the above-mentioned parts. This is a critical issue for them to infer that the scores a test yields are reliable and meaningful, and is a true reflection of students’ knowledge and abilities. We need to look at the parts as one unit; any violation of the said rules in any part may result in a weakness that increases the chances for students to guess the right answer.

9. What are the Rules and Principles to Consider When Writing MCQ(Stem) The stem should present clearly and explicitly the problem to be solved. Options should be similar. For example: The causes of cataracts include: a. aging.* b. arteriosclerosis. c. hemorrhage. d. iritis. A violation of this rule is provided next. *: Correct answer

10. Cataracts: a. are painful. b. may accompany coronary artery disease. c. occur with aging.* d. result in tunnel vision. The stem of the question does not clearly present the problem associated with cataracts that the alternatives address. In addition, the options are dissimilar.

11. The stem should be brief including ONLY necessary information. An example showing a violation of this rule follows. You are caring for an elderly man who lives alone but has frequent visits from his daughter. He has congestive heart failure and some shortness of breath. Your patient was told recently that he has cataracts. The causes of cataracts include: a. aging.* b. arteriosclerosis. c. hemorrhage. d. iritis. The background information about the patient is irrelevant to the problem addressed.

12. Avoid inserting information in the stem for instructional purposes. The goal of testing is not to teach new Information: Cataracts are an opacity of the lens or capsule of the eye leading to blurred and eventual loss of vision. The causes of cataracts include: a. aging.* b. arteriosclerosis. c. hemorrhage. d. iritis.

13. Avoid using negatives, particularly double negatives, including words such as “no,” “not,” and “except.” In reality, however, not all nursing educators have this knowledge (see below) All of the following are realities about health education except? It does not need special skills It depends on theories of learning It should be planned in participation with the learner It is a continuing education that can happen in many settings (From a second exam) All of them are an example for the Biological dimension except. a. genetic b. sex c. geography (From a final exam)

14. The stem and alternatives that follow should be consistent grammatically. If not, clues may be provided as to the correct or incorrect responses. A consistent verb form should be used with the alternatives. For example: Your patient is undergoing a right carotid endarterectomy. Which pre-operative information would be most important to collect as a baseline for the early recovery period? Her ability to: a. follow movements with her eyes b. move all four extremities* c. rotating her head from side to side d. swallow and gag

15. Avoid ending stems with “a” or “an” because these often provide grammatical clues as to the option to select. It is usually easy to rephrase the stem to eliminate the “a” or “an.” For instance, • Narrowing of the aortic valve in children occurs with an: • a. aortic stenosis.* • b. atrial septal defect. • c. coarctation of the aorta. • d. patent ductus arteriosus.

16. Real-life examples of multiple violations, including the above rule, came from a nurse educator-constructed final exam (See below). ''any physiological or psychological factors necessary for a healthy existence '' is a. a. adaptation b. human needs c. concepts Feelings toward a person, object or idea is an. a. values b. attitudes c. ethics Do other nurse educators in other ME universities have similar problems?

17. Alternatives (Options) The number of words included in each option should be consistent in length. Frequently the correct answer is the longest and this may the testwise student to realize that it is the correct answer. For example, You are assessing an emaciated 14-year-old girl. Her mother describes the following changes: resistance to eating and 20-lb. weight loss over the last 6 weeks. It is most likely that the patient resists eating for which of the following reasons? a. Complains of recurring nausea. b. Describes herself as “fat all over” and fearful of gaining weight.* c. Has other GI problems. d. Seeks her mother’s attention. The correct answer can be shortened to: Is fearful of gaining weight.

18. The options should have the same number of parts. In the following example, including two causes in option “a” provides a clue to the answer. Revising that option to only “aging” avoids this. Causes of cataracts include: a. aging and steroid therapy.* b. arteriosclerosis. c. hemorrhage. d. iritis.

19. The alternatives should be consistent grammatically. The answer and distractors should be similar in structure and terminology. Without this consistency, the test-taker may be clued to the correct response or know to eliminate some of the options without being familiar with the content. See below how the correct answer differs grammatically from the others: You are making a home visit with a new mother who is breastfeeding. She tells you that her nipples are cracked and painful. Which of the following instructions should be given to the mother? a. Put the entire areola in the baby’s mouth during feeding.* b. The baby should be fed less frequently until the nipples are healed. c. There is less chance of cracking if the nipples are washed daily with soap. d. Wiping off the lotion on the nipples before feeding the baby may help.

20. The alternatives should sample the same domain, for instance, all symptoms, all diagnostic tests, all nursing interventions, varying treatments, and so forth. You are examining a patient who is having difficulty breathing. His RR is 40, HR 140, and oxygen saturation 90%. He also complains of headache. Which of the following nursing diagnoses is of greatest priority? a. Activity intolerance b. COPD c. Impaired gas exchange* d. Pain

21. Avoid including opposite responses among the options. This is often a clue to choose between the opposites and not consider the others. A sample item follows: The nurse should determine the correct placement of a nasogastric tube by: a. asking the patient to swallow. b. aspirating gastric fluid from the tube. c. inserting air in the tube and auscultating in the epigastric area.* d. inserting water in the tube and auscultating in the epigastric area. In this example, the correct response (c) is opposite one of the distractors (b), which clues the student to select one of these alternatives. In addition, options “c” and “d” begin with “inserting,” which may provide a visual clue to choose between them.

22. McDonald (2007) suggested that when two sets of opposites are used in the alternatives, there is less opportunity for guessing. Using this principle, the first distractor in the example could be reworded to form a second pair of opposites: The nurse should determine the correct placement of a nasogastric tube by: a. aspirating air from the tube. b. aspirating gastric fluid from the tube. c. inserting air in the tube and auscultating in the epigastric area.* d. inserting water in the tube and auscultating in the epigastric area.

23. Arrange the options in alphabetical, numerical, or chronological order, so that the position of the correct response is randomly distributed throughout the test. Options with numerical values should be listed sequentially, and the values should not overlap. When alternatives overlap, a portion of anoption may be correct, or more than one answer may be possible. An example of this problem is: The normal range for serum potassium level in adults is: a. 2.5 – 4.5 mEq/L. b. 0.5 – 3.5 mEq/L. c. 3.5 – 5.2 mEq/L.* d. 1.5 – 4.5 mEq/L. To correct this problem, see next slide.

24. The normal range for serum potassium level in adults is: a. 0.5 – 1.5 mEq/L. b. 2.0 – 3.2 mEq/L. c. 3.5 – 5.2 mEq/L.* d. 8.5 – 10.3 mEq/L.

25. Multiple-Response Items In this format several alternatives may be correct. Students select all of the options that apply by checking the box that precedes each option, as in the following example: The preliminary diagnosis for your patient, a 20-year-old college student, is meningitis. Which signs and symptoms should you anticipate finding? Select all that apply: ❒ 1. Abdominal tenderness ❒ 2. Fever * ❒ 3. Lack of pain with sudden head movements ❒ 4. Nausea and vomiting * ❒ 5. Nuchal rigidity * ❒ 6. Sensitivity to light * ❒ 7. Sudden bruising in neck area 25

26. The principles for writing multiple-response items are the same as for writing multiple-choice. An additional suggestion for writing these items is: • The responses should be listed in a logical order, for instance, • alphabetically, for ease in reviewing. Alternatives are easier to review if shorter combinations are listed before longer ones. For example, • Causes of cataracts include: • 1. aging. • 2. arteriosclerosis. • 3. hemorrhage. • 4. iritis. • 5. steroid therapy. • a. 1, 2 • b. 1, 5* • c. 2, 4 • d. 1, 3, 4 • e. 2, 3, 5

27. Conclusions • Because testing is a core component of the educational process, many universities offer training to faculty members involved in the process of writing tests. • Nurse educators and others at the UoH are responsible for the quality of the tests they develop. Poorly written test items do not meet the main aim for which they were generated; they are meaningless, have no value, and a waste of time and effort. Added to this the fact that the results such test yield are by no means reliable and truly reflect students’ mastery of a taught content. • Pre-requisites for successful item writing include proper command of English, knowledge of principles and rules of item construction, and sufficient time. • Studies investigating the quality of teacher-constructed test items need to be carried out as soon as possible.

28. A Final Question • Besides adhering to the principles and rules of item construction, how would you know that your test is of good quality?

29. Where to go from here • Access my Homepage at http://faculty.uoh.edu.sa/b.hijah/ and download the seminar. You may wish to find out whether you have successfully applied the said rules and principle to your previous MC tests. • Make use of the information presented in this seminar when you develop your tests in the future.

30. References • Downing, S.M. (2005). The effects of violating standard item writing principles on tests and students: the consequences of using flawed test items on achievement examinations in medical education. Advances in Health Sciences Education 10 (2), 133–143. • Ellsworth, R.A., Dunnell, P., Duell, O.K. (1990). Multiple-choice test items: what are textbook authors telling teachers? Journal of Educational Research 83 (5), 289–293. • Hansen, J.D. (1997). Quality multiple-choice test questions: Item writing guidelines and an analysis of auditing test banks. Journal of Education for Business 73 (2), 94–97. • Jozefowicz, R.F., Koeppen, B.M., Case, S., Galbraith, R., Swanson, D., Glew, R.H. (2002). The quality of in-house medical school examinations. Academic Medicine 77 (2), 156–161. 30 30

31. References (Continued…) • Masters, J.; Hulsmeyer, B.; Pike, M.; Leichty, K.; Miller, M.; & Verst, A. (2001). Assessment of multiple-choice questions in selected test banks accompanying text books used in nursing education. Journal of Nursing Education. 40, 25–32. • McDonald, M. E. (2007). The nurse educator’s guide to assessing learning outcomes. Boston: Jones and Bartlett. • Oermann, M. & Gaberson, K. (2008). Evaluation and Testing in Nursing Education. 3rd edition. New York: Springer. • Shizuka T, Takeuchi O, Yashima T, Yoshizawa K (2006). A comparison of three- and four option English tests for university entrance selection purposes in Japan. Language Testing 2006;23:35–57. • Tarrant, M.; Knierim, A.; Hayes, S.; and Ware, J. (2006). The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments. Nurse Education Today. 26, 662–671. 31 31 31

32. References (Continued…) • Tarrant, M.; Ware, J. and Mohammed, A. (2009). An assessment of functioning and non-functioning distractors in multiple-choice questions: a descriptive analysis. BMC Medical Education. 9:40 doi:10.1186/1472-6920-9-40. 32 32 32

33. Thank You Very Much Any Questions?