Prepare and Use Knowledge Assessments

1. Prepare and Use Knowledge Assessments

2. Introduction • Why do we give knowledge tests? • What problems did you have with tests as a student? • As a faculty member?

3. Quick Review • When assessing students’ knowledge, you will use formative knowledge assessments to help guide students toward meeting the learning objectives. • You will use summative knowledge assessments to ensure that students finish the course with the required knowledge.

4. Objectives • Select methods for assessing students’ knowledge • Prepare knowledge assessments • Develop questions for objective written examinations • Administer and score knowledge assessments • Use assessment results to improve performance

5. Methods #1 • Drills, quizzes, and practice tests • Written exercises • Case studies, clinical scenarios, and patient management problems • Project reports

6. Methods #2 • Essay examinations • Objective written examinations (e.g., true-false, multiple-choice, matching and short-answer questions) • Structured practical examinations • Oral examinations

7. Prepare Knowledge Assessments #1 • Identify the learning objectives or outcomes. • Use simple and clear language. • Include at least one item per objective (too few items – may not be valid, too many items – students become test tired) • Use correct grammar in questions and answers.

8. Prepare Knowledge Assessments #2 • Ensure that questions reflect conditions stated in the objective. • Is the question in any way controversial? • Are the distractors in multiple-choice questions reasonable, and similar in structure and length to the correct answers? • What is the difficulty of the questions?

9. Prepare Knowledge Assessments #3 • Make each test item separate from other items. • Ask the same or similar questions (especially for oral examinations and reports). • Divide long tests into several parts. • Provide clear directions for each type of item. • Decide on the model answers to questions. Use answer keys or checklists for scoring.

10. Arranging Test Items • Subject matter • Type • Level of difficulty

11. Review the Assembled Test #1 • Are there items for all of the objectives? Does the number reflect the time spent on each objective? • Is the number of items included in the test in direct proportion to their importance in learning? • Does each item really measure the students’ attainment of the objective?

12. Review the Assembled Test #2 • Is each set of directions clear? • Is there ample space to write the response? • Are tricky, obvious, or irrelevant questions avoided? • Is each item separate and independent from the rest of the items?

13. Review the Assembled Test #3 • Are similar items grouped together? • Is the test designed so that it is easy to score? • Will students be provided with meaningful feedback about their answers?

14. Preparing a Test Bank • Work in a group to develop questions. • Sort questions based on subject, category, level of knowledge, or type of test item. • Store the questions on a computer or cards. • Ensure that only authorized persons have access to the question bank.

15. Activity • Work in 4 groups. Each group will be assigned 1 type of item (Table 10-1). • In your own words, describe the item and the advantages and disadvantages of that item. • Briefly share with the other groups.

16. Writing True-False Items #1 • The language of true-false test items should be clear, concise, and understandable (avoid words such as more, few, large, and good). • Avoid using negatively stated items (e.g., It is not recommended that. . . .) as these are confusing in a true-false question.

17. Writing True-False Items #2 • Avoid words or expressions that frequently identify a statement as true or false. • Words often found in false statements are only, never, all, every, always, none, and no. • Words often found in true statements are usually, generally, sometimes, customarily, often, may, could, and frequently.

18. Writing True-False Items #3 • Use statements that are either completely true or completely false. Avoid a statement that is partially true and partially false. True-false items should focus on one idea and should challenge, but not trick, the students. • Make the true statements equivalent in length and number to the false statements.

19. Sample Directions Directions: For each of the following statements, print a capital T in the block to the left of the number if the statement is true or a capital F if the statement is false. You will receive one point for each correct answer.

20. Parts of the Multiple Choice Question • Stem • Responses • Distractors

21. Writing Multiple-Choice Questions #1 • Write simple stems that state the problem. • Avoid using negative terms in the stem (such as no, never, none, and not) – if you do, bold-face and/or underline the term to draw attention to it. Example: “Which of the following is not the. .” • Provide a coherent list of possible answers (use a/an when needed).

22. Writing Multiple-Choice Questions #2 • Write reasonable distractors. • All distractors should be of similar length to avoid giving clues to the correct response. • Avoid use of “All” and “None of the above” in responses (when used, use 25% of the time as correct answer when there are 4 choices).

23. Sample Scenario MCQ Mrs. B. is 20 years old and had an IUD inserted a month ago. She came to the health center 2 days ago with vaginal discharge and abdominal and pelvic pain. She reports that she does not have any fevers or chills. • What is your plan? • Gather history, send vaginal cultures, remove the IUD. • Gather history, send vaginal cultures, follow up in 3 days. • Gather history, treat with antibiotics for presumptive PID. • Gather history, send vaginal cultures, treat with antibiotics for PID, and remove the IUD if the woman wishes.

24. Sample Directions Directions: Each of the items on this examination is followed by four possible responses. For each item, select the best response. Indicate your answer by circling the appropriate letter next to your answer. Each correct response is worth one point.

25. Matching Questions Refer to the manual for more information on developing matching questions.

26. Short-Answer Questions Refer to the manual for more information on developing short-answer questions.

27. Test Administration • Time allowed • How to select and record answers • Scoring system to be used • Physical environment • Remain in the room • Lighting and ventilation

28. Score the Test #1 • With written objective examinations, students can mark the answers directly onto the test or onto a separate answer sheet. • When there are many test items, it may be easiest for students to write their answers on a separate answer sheet.

29. Score the Test #2 • Use an answer key (or a computer with scanner) to score the tests. • See the module for more information on scoring knowledge assessments.

30. Using Assessment Results #1 • Knowledge is assessed to determine whether students are meeting the learning objectives and have acquired the required information base to become competent healthcare providers.

31. Using Assessment Results #2 • Formative knowledge assessments help students decide what content areas they need to spend more time studying to prepare for summative assessments. • The results of summative assessments determine if students are meeting the learning objectives.

32. Using Assessment Results #3 • Instruct students to review the material related to the questions they missed. • Give students an opportunity to ask you questions about any test items on which they scored poorly or that they did not understand.

33. Using Assessment Results #4 • If many students had trouble with the same questions, either the teaching methods or materials did not adequately address that learning objective, or the question needs to be rewritten. • Do not be afraid to revise problematic test questions or adapt the teaching methods used to better address the content.

34. Activity • If we have time, let’s try writing some MCQs based on topics in your various curricula and share your questions. • Work in small groups to develop 1 question and then put this on a flipchart to share with the other groups.

35. Summary • What questions do you have about developing, administering and scoring knowledge assessments? • How can we assist faculty members in developing, administering and scoring knowledge assessments to support implementation of curricula?