Developing Scenario-based Sequential Patient Management Problem

1. Developing Scenario-based Sequential Patient Management Problem Zubair Amin National University of Singapore paeza@nus.edu.sg

2. Workshop Agenda Introduction: 9.00 to 9.10 Knowledge assessment: 9.10 to 9.30 Features of Sequential MEQ: 9.30 to 10.00 Developing Sequential MEQ: 10.00 to 11.00 Review of questions generated: 11.00 to 11.30 Tips on implementation & Discussion: 11.30 to 12.00

3. Clinical Competence Relevant Skills Relevant Knowledge Attitudes Clinical Problem Solving Clinical Performance Newble et al

4. Behaviour Professional authenticity Cognition Does Shows how Knows how Knows Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

5. Professional authenticity Performance or hands on assessment Does Shows how Written, Oral or Computer based assessment Knows how Knows Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

6. For assessment of clinical competency, we need both knowledge (cognition) and performance measures Knowledge measurement should not replace the performance measurement and performance measurement should not replace knowledge measurement Geoff Norman

7. Clinical Decision Making Reduced Dispersed Elaborated Compiled Empty mind Cluttered mind Deductive thinker Recall/recognition Bordage et al

8. Relevance in Knowledge Assessment To develop competencies in an area of inquiry, students must have a deeper foundation of knowledge However, they should understand facts and ideas in the context of a conceptual framework. Otherwise, they will learn them for the purpose of a test but will not be able to apply knowledge in proper clinical setting. National Academy Press www.nap.edu

9. Options in Written Assessment Closed ended questions: Multiple Choice Questions (MCQ) Extended Matching Questions (EMQ) Semi-structured open ended questions Short Answer Question Modified Essay Questions Long essay questions

10. A 3-year-old child has infantile spasm. She was full-term infant. At 15 month visit, her paediatrician noticed that she is unable to stand without support. She has polysyllabic babbles. Currently, she walks with support. She can speak about 10 words and indicate her want by hand gesture. She has several skin lesions on the back (shown below) and one at the thigh. What is the most likely diagnosis? A) Neurofibromatosis B) Tuberous sclerosis C) Hypomelanosis of Ito D) Chediak Higashi syndrome E) Waardenburg syndrome

11. MCQ with Multimedia A 8-year previously healthy girl, Mariam, is referred to the clinic by the school teacher who noticed the girl appears occasionally to be at lost during the class. Mariam mentioned that she can bring the event by breathing fast and deep. A friend of her took a video of the event. Click here to watch the video.

12. An Electro-encephalogram (EEG) is most likely to show which of the following patterns: Normal EEG Chaotic high voltage EEG with spikes and waves Atypical slow voltage spike and wave complexes Bilateral synchronous three/second spikes and wave Rolandic spikes in centero-temoporal region

13. MCQ with Multimedia: Alternate Example A 8-year previously healthy girl, Mariam, is referred to the clinic by the school teacher who noticed the girl appears occasionally to be at lost during the class. Mariam mentioned that she can bring the event by breathing fast and deep. A friend of her took a video of the event. An EEG shows three/second bilaterally synchronous, symmetrical spikes and slow waves. Click here to watch the video.

14. A 8-year previously healthy girl, Mariam, is referred to the clinic by the school teacher who noticed the girl appears occasionally to be at lost during the class. Mariam mentioned that she can bring the event by breathing fast and deep. A friend of her took a video of the event. An EEG is done which is shown in the next page.

16. Which of the following drug would you consider as the first line of treatment? Phenobarbital Adrenocorticotrophic hormone Diazepam Carbamazepine Velproic acid

17. MCQ Most efficient in testing the breadth of students’ knowledge Mostly tests one or limited aspect of patient problem (e.g., diagnosis, therapy, investigations) Difficult to eliminate cuing effect Developing high quality MCQ needs effort and training

18. Open Ended Questions: Static MEQ or SAQ Higher order thinking: rationalization, clinical reasoning Can be developed as contextual/scenario based Problems: Later questions can give cue to earlier ones Manual marking Limited number of questions for a scenario

19. “Can be designed to assess not just the candidate’s knowledge base, but also the application of that knowledge to interpretation of data, clinical reasoning and problem solving which are important outcomes of medical education.” – M.H. Davies

20. You are the Medical Officer in the Accident & Emergency department. An eighty-one year old man is admitted to your ward with a fall. He complains of pain in his groin. He is unable to move his right leg and you notice that it is slightly shorter and externally rotated. His X –ray is shown here:

21. What is your diagnosis? What are the three most important complications that might arise from this? How would you manage this patient? What advice would you give this patient/care givers on discharge?

22. Advantages Can test the thought process and the reasoning/decision making/ critical analysis abilities of the trainee Students can not go back to earlier questions; no cueing effect Can simulate authentic clinical situations; hence tests the important areas rather than trivia Can combine basic science, clinical, ethical, psychosocial or community aspects relating to the problem Test the depth of knowledge

23. Disadvantages Content experts needed for marking. Optical scanner cannot be used, hence more resource-intensive. Large samples of questions are needed to improve the validity (& reliability). May require double marking. Faculty training is imperative

24. MEQ Writing Steps Develop an authentic practice/clinical scenario as the case Ensure that the questions cannot be answered without the vignette Built in relevant basic sciences, clinical sciences, public health, behavioral / ethics / law /medical sociology questions.

26. Avoid Ambiguity Assessment by ambush developing rare cases, Inclusion of pictures, videos of atypical conditions Extensive ‘window dressings’ Inclusion of ‘red herrings’

27. Approximate Reliability for 4-Hour Testing Time; Wass et al; Lancet 2001

29. Examiner Question

30. Examiner Question

31. Examiner Question

32. Setting Good MEQs: Practical Tips Use MEQ sparingly for selected high impact topics. Reserve MEQ to assess depth of knowledge, as opposed to breadth of knowledge as in MCQ. Include a mixture of short open-ended and restrictive response questions.

33. Provide clue or answer to earlier questions to avoid examinees getting penalized repeatedly for the same mistakes. Do not use excessive videos and high-intensity imageries which may use up all the available bandwidth. Always allow the students to practice sample questions before the actual examination

34. ‘The most productive applications of new technologies to assessment are not necessarily those with the greatest sophistication, speed, or glitz. The greater potential lies in the role of technology could play in realizing the central ideas (..): that assessment should be based on modern knowledge of cognition and its measurement, should be integrated with curriculum and instruction, and should inform as well as improve student achievements.’ Pellegrino J. W., Chudwosky N. & Glaser R. 2001. Knowing what Students Know: The Science and Design of Education Assessment; page 261