Universities Medical Assessment Partnership

1. Universities Medical Assessment Partnership LTSN-01 Workshop Series Quality in Written Assessments

2. Feedback to support student learning Questions validated & reliable The UMAP Project 3500+ Questions for formative and summative assessment Universities Medical Assessment Partnership

3. Question Writing for UMAP • Core knowledge needed by all graduates on qualifying • Link to a core competency necessary for graduation • Competences can be acquired at any stage of a programme (not necessarily at the end) Universities Medical Assessment Partnership

4. Question Writing for UMAP • Questions can be written to test application of knowledge rather than factual recall • Knowledge can be applied in a clinical context – • ‘A person with double vision looking to…. • Or can be related to normal function • ‘When looking to the …..’ Universities Medical Assessment Partnership

5. Bloom's Taxonomy Evaluation Synthesis Analysis Application Comprehension Knowledge Universities Medical Assessment Partnership Bloom, B., Ed. (1956). 'A taxonomy of educational objectives'. in Handbook I: Cognitive domain. New York: McKay

6. MCQs Universities Medical Assessment Partnership

7. MCQs for basic science • Clinical or non-clinical, the question should require some application of knowledge, rather than simple factual recall. • Use a case or context to create a relevant test of applied knowledge • ‘A patient with ECG changes ( either II, III, aVf or inferiorly) – which coronary artery? Universities Medical Assessment Partnership

8. Non-applied MCQ Theme: The leg Focus: Anatomy Lead In Which muscle is responsible for lateral rotation and extension of the hip joint and whose contraction supports the extended knee? Options A Gluteus maximus B Gluteus minimus C Hamstrings D Iliopsoas E Obturator internus Universities Medical Assessment Partnership

9. Applied MCQ Theme: The leg Focus: Anatomy Stem A 65 year old man has difficulty rising from a seated position and straightening his trunk, but he has no difficulty in straightening his leg. Lead in Which muscle is most likely to have been injured? Options A Gluteus maximus B Gluteus minimus C Hamstrings D Iliopsoas E Obturator internus Universities Medical Assessment Partnership

10. When Writing MCQ Options D C A E B Least Most Correct Correct Answer Answer Options should represent a single continuum of choices Universities Medical Assessment Partnership From Susan M. Case & David B. Swanson: Constructing Written Test Questions, 2002 http://www.nbme.org/about/itemwriting.asp.

11. A Bad MCQ Theme: Lifestyle and Health Focus: Contributing Factors Lead In Which of the following is true about the link between lifestyle and health? A. Alcohol is a frequent cause of ill health in as many women as men B. Adherence to medication is seldom associated with better health C. Lifestyle may play a significant role in health of married men D. Lifestyle is hereditary in most cases E. A healthy lifestyle is simply a direct result of advances in medicine Is ‘frequently’ more or less true than ‘seldom’ Options not homogenous and deal with miscellaneous facts Cannot be answered without reading(and re-reading) the options Universities Medical Assessment Partnership

12. A Bad MCQ A. Alcohol B. Adherence C. Marriage D. Heredity E. Medical Advances Associations False True Not homogenous- miscellaneous facts Universities Medical Assessment Partnership From Susan M. Case & David B. Swanson: Constructing Written Test Questions, 2002 http://www.nbme.org/about/itemwriting.asp.

13. A Bad MCQ Theme: Cardiovascular System Focus: Anatomical Features Lead In The way to a man’s heart is… Options A The same as the way to a woman’s B Through his aorta C Through his chest D Through his stomach E Via cardiac surgery SUBJECTIVE Depends on what examiner was thinking! Universities Medical Assessment Partnership From Susan M. Case & David B. Swanson: Constructing Written Test Questions, 2002 http://www.nbme.org/about/itemwriting.asp.

14. A Bad MCQ Theme: Recommended adult water intake Focus: Advice Lead In Adults should drink Options A. 8 glasses of water per day B. Avoid dehydration C. Be given too much salty food • Heart disease when they are over 40yrs old and should take aspirin daily • Take folic acid in early stages of pregnancy Only one feasible grammatical option Universities Medical Assessment Partnership

15. A Bad MCQ Theme: Crime and society Focus: Statements Lead In Crime is A. Always equally distributed among the social classes B. Sometimes overrepresented among the poor C. Frequently overrepresented among the middle class and rich • Primarily an indication of psychosexual maladjustment • Reaching a plateau of tolerability for the nation The ‘testwise’ student has only three options Use of ‘always’, ‘never’, ‘sometimes’ Universities Medical Assessment Partnership From Susan M. Case & David B. Swanson: Constructing Written Test Questions, 2002 http://www.nbme.org/about/itemwriting.asp.

16. The Problem with Frequency Terms Universities Medical Assessment Partnership From Susan M. Case & David B. Swanson: Constructing Written Test Questions, 2002 http://www.nbme.org/about/itemwriting.asp.

17. A Bad MCQ Theme: Infertility in a female Focus: Probability Lead In Following a second episode of infection, what is the likelihood that a woman is infertile? Options A. Less than 20% B. 20 to 30% C. Greater than 50% • 90% • 75% Numeric data not stated constantly Universities Medical Assessment Partnership

18. EMQs Universities Medical Assessment Partnership

19. What is an EMQ? • Greater test of synthesised knowledge than MCQs and True/False • More reliable assessment of knowledge Structure • Theme and a focus • Options • Lead InE.g. ‘select the correct answer from the options above.’ • Stems Usually 5 scenarios deisgned to stimulate synthesisFor each stem, one answer is correct, but all are feasible choices. Universities Medical Assessment Partnership

20. Extended Matching Items: Writing Option Lists Theme: The Heart Focus: Anatomical Features Options A Aorta L Left Ventricle B Aortic Semilunar Valve M Ligmentum Arteriosum C Brachiocephalic Artery N Papillary Muscles D Chordae Tendineae O Pulmonary Semilunar Valve E Interventricular Septum P Pulmonary Trunk F Left Atrioventricular Valve Q Right Atrioventricular Valve G Left atrium R Right Atrium H Left Common Carotid Artery S Right Coronary Artery I Left Pulmonary Artery T Right Ventricle J Left Pulmonary Veins U Superior Vena Cava K Left Subclavian Artery V Trabeculae Carneae All Logical Options Under a Given Theme Universities Medical Assessment Partnership

21. A Bad EMQ: Options Theme: Cell function Focus: Components of a neuron Options A. Axon B. Cell nucleus C. Dendrites D. Myelin sheath E. Nerve Impulse F. Neurotransmitter G. Nodes of ranvier H. Oligodendrocite I. Presynaptic terminals J. Soma K. Synaptic cleft L. None of the above OPTIONS NOT MUTUALLY EXCLUSIVE ‘NONE OF THE ABOVE’ Not a logically equivalent option! Universities Medical Assessment Partnership

22. Extended Matching Items: 5 Options Vs. 15 Options Better discriminator of knowledgeable students A B C D E F G H I J K L M N O 5-option 81 * 4 0 14 * 1 15-option 2 59 2 13 1 2 1 0 1 7 2 2 8 0 0 DISTRIBUTION OF EMQ SCORES FOR 1 QUESTION Universities Medical Assessment Partnership Case SM, Swanson DB. Evaluating diagnostic pattern recognition: a psychometric comparison of items with 15, 5, and 2 options. Paper presentation, Annual Meeting of the AERA. San Francisco, CA: March 1989.

23. When Writing Option Lists F K B J H G A E D I Least Most Correct Correct Answer Answer Options Represent a Single Continuum of Choice Universities Medical Assessment Partnership From Susan M. Case & David B. Swanson: Constructing Written Test Questions, 2002 http://www.nbme.org/about/itemwriting.asp.

24. EMQs: Stem Writing • Stems should be long, options should be short • Stems should stimulate synthesis of information • Not too much or too subtle information that knowledgeable students will be disadvantaged Universities Medical Assessment Partnership

25. Question Reviewing Gold Standard Test • Cover the options – can you answer the question correctly? • If not, check phrasing and ensure there is enough information to point to just one of the answers Points to check for • Are the options homogenous? • Is there too much/too little information in the stem? • Level of knowledge? Universities Medical Assessment Partnership

26. Examinations Supplying questions to feature in consortium exams Bank Piloted Students Recent graduates What Happens Next? Question Review Questions checked Tagging formalised Universities Medical Assessment Partnership

27. Would you like to participate in UMAP? See our website www.umap.man.ac.uk for details of forthcoming question writing workshops across the partner schools. The project is set to expand to include new partner schools at the end of 2005. We have written to all institutions in the UK and are partnership discussions are planned for late 2004. Universities Medical Assessment Partnership