The 10 Most Common Testing Mistakes – And How You Can Avoid Them

1. The 10 Most Common Testing Mistakes – And How You Can Avoid Them Presented by: Steven B. Just, Ed.D.

2. Lack of testable learning objectives

3. Lack of testable learning objectives

4. Lack of testable learning objectives • The learning objectives drive the creation of the content • Each objective should be concise • Each objective should be “testable” • The verb is crucial: • Avoid “understand,” “know,” “be aware of” • Use “define,” “calculate,” “list”…

5. Poorly-written learning objectives Students should be able to: • Understand the mechanisms of cardiac contraction • Demonstrate a knowledge of basic cardiac pathophysiology • Integrate an understanding of cardiac physiology into selling activities

6. Correct learning objectives Students should be able to: • Describe the electrophysiology of the heart, including the cellular anatomy and electrochemistry of contraction • Read a simple, normal electrocardiogram, and describe how the parts of the recorded wave pattern relate to cardiac events • Identify the gross anatomy of the heart • List the basic categories of cardiac pathologies • Provide examples of the most commonly occurring cardiac pathologies

7. Lack of testable learning objectives • Ensure the questions are directly derived from the learning content • Separate those who have mastered the material from those who haven’t • Ensure real world competencies are assessed • Ensure that all competencies are mastered

8. No clearly identified purpose for exam

9. No clearly identified purpose for exam • Assessment • Quiz • Test • Exam • Evaluation • Pretest • Post-test • Formative Assessment • Summative Assessment • Diagnostic Assessment • Rubric • Performance Assessment • Self-assessment • High Stakes Assessment • Certification

10. No clearly identified purpose for exam Assessment: • A systematic process for returning results in order to describe what students know or can do. • An ongoing process aimed at measuring and improving student learning • Assessments can be in the form of a quiz, test, exam or evaluation.

11. No clearly identified purpose for exam Quiz: • A low-stakes diagnostic assessment in which the results are only to be used for self- or group-diagnosis and prescription

12. No clearly identified purpose for exam Test: • A medium stakes formative assessment designed to inform both the learner and (optionally) the instructor of the learner’s level of knowledge at an intermediary point in the instructional process. There are no long-term consequences for failure. Short-term consequences may include required remediation before proceeding with a learning activity

13. No clearly identified purpose for exam Exam: • A high-stakes summative assessment at the completion of a learning experience for which there are consequences for failure. Results of exams are made available to the learner’s direct supervisor and appropriate training department personnel. Exam results may have career impacting consequences

14. No clearly identified purpose for exam Evaluation: • An assessment that measures, compares, and judges • For example: • Role play evaluations • Smile sheets • Evaluation of a training program • Level 3 and 4 evaluations

15. No clearly identified purpose for exam Assessment: A systematic process for returning results in order to describe what students know or can do. Quiz:A low-stakes diagnostic assessment in which the results are only to be used for self- or group-diagnosis and prescription. Test: A medium stakes formative assessment designed to inform both the learner and (optionally) the instructor of the learner’s level of knowledge at an intermediary point in the instructional process Exam: A high-stakes summative assessment at the completion of a learning experience for which there are consequences for failure. Evaluation:An assessment that measures, compares, and judges

16. Questions not job relevant

17. Questions not job relevant Questions to Ask

18. Questions not job relevant • Does this question test a learning objective? • Is this knowledge required? • Could this be looked up or must it be memorized?

19. Using “double negative” questions

20. Using “double negative” questions Lack of insulin production does not cause: A. a lack of glucagon production * B. depletion of fatty acid stores C. unchanged brain cell uptake of glucose D. diminished fat storage in adipose cells

21. Using “double negative” questions Which of the following is an effect of the LACK OF insulin production: A. increased conversion of glucose into energy B. production of ketones from the metabolism of excess fats * C. increased protein production D. increased formation of glycogen

22. Questions all at Knowledge/Recall level

23. Questions all at Knowledge/Recall level

24. Questions all at Knowledge/Recall level Knowledge Question cues: • List • Identify • Define • Describe

25. Knowledge Question Example List the order in which blood travels through the chambers of the heart from the vena cava to the aorta: • right atrium, right ventricle, left atrium, left ventricle * • right atrium, left atrium, right ventricle, left ventricle • left atrium,right atrium, left ventricle, right ventricle • left atrium, left ventricle, right atrium, right ventricle

26. Questions all at Knowledge/Recall level Comprehension Question cues: • Describe • Predict • Summarize • Estimate • Interpret

27. Comprehension Question: Example An ultrasound image of a patient’s heart indicates definite enlargement of the atria, and normal-appearing ventricles. The overall size of the heart has not yet been determined. Which of the following cardiomyopathies could describe this condition: • restrictive or dilated • hypertrophic or dilated • hypertrophic or restrictive * • dilated or hypertrophic

28. Questions all at Knowledge/Recall level Application Question cues: • Apply • Demonstrate • Calculate • Complete • Classify

29. Application Question: Example The isolated perfused frog heart serves as a good experimental model of human heart contractility. What would you expect to happen to the contractions of the frog heart if a solution with a high calcium ion concentration were applied to it? • increased contractile force * • unchanged contractile force • increased rate of contraction • unchanged rate of contraction

30. Questions all at Knowledge/Recall level Learning Objectives and Questions Must be Aligned Within Bloom’s Taxonomy

31. Questions all at Knowledge/Recall level • Knowledge LO • You will be able to define arthritis. • Knowledge Question • Choose the best definition for the term arthritis

32. Questions all at Knowledge/Recall level • Comprehension LO • You will be able to identify examples of thrombosis. • Comprehension Question • Which of the following is an example of thrombosis?

33. Questions all at Knowledge/Recall level • Application LO • You will be able to apply your knowledge of key pivotal trial results by stating critical supporting data points for your drug. • Application Question • You are in conversation with a physician about the safety data for your product. Identify the critical supporting data points from the pivotal trial results.

34. Questions all at Knowledge/Recall level

35. Questions all at Knowledge/Recall level

36. Failure to analyze exam data

37. Failure to analyze exam data • Do you need to know how well your questions discriminate between weak and strong students? • Point-biserial correlation • Do you need to know what percent of students answered each question correctly? • Difficulty level • Do you need to know where students have misinformation? • Choice distribution • Do you need to know where the group has strengths and weaknesses? • Score by learning objective/topic

38. Failure to retest periodically

39. Ebbinghaus Curve of Forgetting

40. Ebbinghaus Curve of Forgetting

41. Invalid passing score

42. Invalid passing score • The Higher Authority Method: • “Our Vice President said it should be 90” • The Committee Method: • “90 seems about right” • The Received Wisdom Method: • “I don’t know how or when it got set, but it’s always been 90”

43. Invalid passing score The Angoff Method: A Defensible Method For Setting Passing Scores

44. Invalid passing score • Step 1: Identify three to five judges who are familiar with the competency covered by the test.

45. Invalid passing score Step 2: • Does the question test the learning objective? • Are all choices plausible? • Is the identified correct answer correct? Could other choices plausibly be correct? • Is the question properly written according to the rules of valid question writing? • Is the question job relevant?

46. Invalid passing score • Step 3: For each item on the test each judge estimates the probability that a minimally competent person would get it right. • Step 4: Sum the probabilities of each judge • Step 5: Average the judges’ scores

47. Invalid passing score Item 1 2 3 4 5 Total Percent Judge 1 .75 .80 .75 .90 .95 4.15 83% Judge 2 .80 .90 .75 .90 .75 4.10 82% Judge 3 .85 1.00 .90 .80 .85 4.40 88% Averaging the totals for each Judge Cut Score= 84%

48. Using “All of the Above” questions

49. Using “All of the Above” questions Which of the following waves, complexes and intervals can appear on an electrocardiogram: A. P B. QRS C. QT D. All of the above *

50. Using “All of the Above” questions Which of the following waves, complexes and intervals DOES NOT appear on an electrocardiogram: A. P B. QRS C. QT D. U*