Assessment of Clinical Competence of Health Professionals

1. Assessment of Clinical Competence of Health Professionals Professor Hossam Hamdy University of Sharjah

2. Professional Competence The Habitual and Judicious use of: • Communication • Knowledge • Technical Skills • Evidence-based decision-making • Emotions • Values and reflection to improve the health of the individual patient and the community

3. ACGME Competencies • Patient care • Knowledge • Practice-based learning and improvement • Interpersonal and communication skills • Ethics & Professionalism • System-based practice

4. Assessment Five Basic Questions • WHY? • WHAT? • HOW? • WHEN? • WHOM?

5. Purpose of Assessment • Providing feedback • Guiding student learning • Insuring medical competence

6. WHY? • Consider assessment as a diagnostic test for the disease “medical incompetence” • False negative = failing a competent student • False positive = passing a student who is incompetent

7. TRUTH TEST

8. WHAT Should Be Assessed? Blueprinting Test content matches objectives • Assessment matches competencies learnt • Assessment matches format of learning, “PBL”

9. Aim of Blueprinting Reduce: A. Construct under-representation: Biased sampling Few items covering a topic e.g. OSCE B. Construct irrelevant variance: Flawed items format Inadequate sampling of student Behaviour.

10. Blueprinting • Identify clinical problems that examinees should be able to handle. • For each problem, define the clinical tasks in which the examinee is expected to be competent. • Define level of performance / resolution

11. Blueprinting (cont\-) • Prepare a blueprint to guide the selection of problems to be included in the assessment. • Let the characteristics of the clinical problems define the tasks to be included in the assessment.

12. A Simple Blueprint

13. Pre-Clerkship Phase Theme B.Sc. Exam Blueprint 2003-2004

15. HOW?Choice of Method U = W R x W V x W E x W A x W C 1 2 3 4 5 U = Utility R = Reliability V = Validity E = Educational Impact A = Acceptability C = Cost/Available Resources W = Weight

16. Reliability • Reliability = reproducibility • Inter-rater: Multiple examiners across different cases • Inter-case:“multi use format (OSCE)”

17. Reliability Reliability is matter of sampling • Across content • Across Tasks • Across Problems

18. Test Sample Test Sample Domain of Interest  

19. “Reliability”Practical Implications • Sample adequately across content • Clinical competencies are inconsistent across different tasks • Test length is crucial if high-stakes decisions are required • Use as many examiners as possible

20. Validity • Validity = measures what it intends to measure • Select appropriate test formats for the competencies to be tested. This action invariably results in a composite examination.

21. Professional authenticity A Simple Model of Competence Performance or hands on assessment Does Shows how Written, Oral or Computer based assessment Knows how Knows Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

22. Performance assessment in vivo: Masked SPs, Video, Audits….. Does Performance assessment in vitro: OSCE, SP-based test….. Knows how (Clinical) Context based tests: MCQ, essay type, oral….. Knows Factual tests: MCQ, essay type, oral….. Shows how Validity Climbing the Pyramid . . . Does Shows how Knows how Knows

23. Written Assessment Methods Context-free questions Stimulus Format Context-rich questions EMQ, KF, A & R “MCQ” Open ended questions types SAQ, MEQ Response Format MCQ types

24. The MCQ’s Family

25. Written Assessment of Clinical Competence • MCQs 1) Best Single Response 2) Extended Matching • Key Features Questions • Constructed Response Questions

26. Components of an A-type Question A 35-year old man has had a stomach ache all afternoon. He ate the following lunch: two big McDonalds hamburgers, an ice cream shake, large fries. Vignette, Scenario, Or Stem Which is the most likely diagnosis? Lead-in

27. Components of an A-type Question • Abdominal aneurysm • Appendicitis • Bowel obstruction • Cholecystitis • Colon cancer • Pancreatitis • Too much lunch Options A, B, C, D, E & F are distracters Option G is the key

28. Diagram of Rank-Ordered Options for a Good One-Best-Answer Item D C A E B Least Most Correct Correct Answer Answer Note: Options must be homogeneous (eg, all diagnoses, all muscles). You must be able to rank-order the options on the same dimension.

29. Sample Good Question & ‘Cover The Options’ Rule A 32-yo man has a 4-day history of progressive weakness in his extremities. He has been healthy except for an upper respiratory tract infection 10 days ago. His temperature is 100 F, BP 130/80, pulse 94, respirations 42 and shallow. He has symmetric weakness of both sides of the face and the proximal and distal muscles of the extremities. Sensation is intact. No deep tendon reflexes can be elicited; the plantar responses are flexor. Which of the following is the most likely diagnosis? A. Acute disseminated encephalomyelitis B. Guillain-Barré syndrome C. Myasthenia gravis D. Poliomyelitis E. Polymyositis

30. Sample Good Question & ‘Cover The Options’ Rule A 32-yo man has a 4-day history of progressive weakness in his extremities. He has been healthy except for an upper respiratory tract infection 10 days ago. His temperature is 100 F, BP 130/80, pulse 94, respirations 42 and shallow. He has symmetric weakness of both sides of the face and the proximal and distal muscles of the extremities. Sensation is intact. No deep tendon reflexes can be elicited; the plantar responses are flexor. Which of the following is the most likely diagnosis? A. Acute disseminated encephalomyelitis B. Guillain-Barré syndrome C. Myasthenia gravis D. Poliomyelitis E. Polymyositis

31. Extended Matching R-Set

32. Theme: Neuroanatomy: Blood Supply of the Brain Options: A. Left anterior cerebral artery E. Right anterior cerebral artery B. Left posterior cerebral artery F. Right posterior cerebral artery C. Left middle cerebral artery G. Right middle cerebral artery Left lenticulostriate arteries H. Right lenticulostriate arteries Lead-in: For each patient with neurologic abnormalities, select the artery that is most likely to be involved. Vignette: “Items” 1. A 72-year-old right-handed man has weakness and hyperreflexia of the right lower limb, an extensor plantar response on the right, normal strength of the right arm, and normal facial movements. Ans: A 2. A 68-year-old right-handed man has right spastic hemiparesis, an extensor plantar response on the right, and paralysis of the lower two-thirds of his face on the right. His speech is fluent, and he has normal comprehension of verbal and written commands. Ans: G

33. Options and Lead-in: Diagnosis Theme: Abdominal Pain A. Abdominal aneurysm K. Kidney stone B. Appendicitis L. Mesenteric adenitis C. Bowel obstruction M. Mesenteric artery thrombosis D. Cholecystitis N. Ovarian cyst - ruptured E. Colon cancer O. Pancreatitis F. Constipation P. Pelvic inflammatory disease G. Diverticulitis Q. Peptic ulcer disease H. Ectopic pregnancy – ruptured R. Perforated peptic ulcer I. Endometriosis S. Pyelonephritis J. Hernia T. Torsion For each patient with abdominal pain, select the most likely diagnosis.

34. Sample Diagnosis Items: Abdominal Pain • A 25-year-old woman has sudden onset of persistent right lower abdominal pain that is increasing in severity. She has nausea without vomiting. She had a normal bowel movement just before onset of pain. Examination shows exquisite deep tenderness to palpation in right lower abdomen with guarding but no rebound; bowel sounds are present. Pelvic examination shows a 7-cm, exquisitely tender right sided mass. Hematocrit is 32%. WBC is 18,000/mm3. Serum amylase activity is within normal limits. Test of the stool for occult blood is negative. Ans: B • An 84-year-old man in a nursing home has increasing poorly localized lower abdominal pain recurring every 3-4 hours over the past 3 days. He has no nausea or vomiting; the last bowel movement was not recorded. Examination shows a soft abdomen with a palpable, slightly tender, lower left abdominal mass. Hematocrit is 28%. WBC is 10,000/mm3. Serum amylase activity is within normal limits. Test of the stool for occult blood is positive. Ans: E

35. Constructed Response Questions(Short Answer Questions) • A written question • A statement, journal article or report • A problem scenario: clinical, basic science or community • Photographic or other investigative data The response generated by the student may be:

36. A 72-year-old, right-handed man presents with a 6-month history of increasing tremor of his right hand that causes him severe embarrassment, such that he avoids going out. On examination, the tremor is most marked at rest and decreases on maintaining a posture and during movement. There is no intention tremor. Example questions: • The name of this type of tremor • The anatomical site of the pathological lesion • Drugs causing this phenomenon as a side-effect • Additional signs to look for on physical examination • How to treat the condition

37. Assessment of Clinical Performance • Fundamental to “Good Medical Practice” • “What doctors do in controlled representations” of practice e.g. “OSCE” • “What doctors do in real life” Rethams et all, Med Ed 2002.

38. Effective Workplace-Based Assessment • “sampling collection of data concerning specific patient encounters for a number of different domains” • Reflects routine performance

39. Problems of Workplace-Based Assessment • Sampling: variability in the complexity of patients problem • Judges errors: • Time: Requires adequate allocation of time and resources. • Management of Data: not easy • Validity and Reliability: variable

40. Workplace-Based Assessment Methods • Mini-clinical evaluation exercise (mini-CEX) • Direct Observation Clinical Encounter Examination (DOCE) • Clinician encounter cards (CEC) • Blinded patient encounters (BPE) • Direct observation of procedural skills (DOPS) • Case-based discussion (CbD) • Multisource feedback (MSF) 360o

41. Mini-CEX • Real patients 12 - 14 • ~ 6 / years • Only components of the encounter observed • 15-20 minutes “Norcini 2003” • Formative

42. The Traditional Long Case Examinations (LCE) • Unobserved 30 – 45 minute interview and examination on a selected patient • Candidates present only their findings to the examiner

43. (LCE) Validity • It assesses the integrated interaction between the student/doctor and the real patient • Authenticity high • More valid than the task given in an OSCE • Little is known about the construct validity and consequential validity “Educational Impact”

44. (LCE) Reliability • Poor intercase reliability • Content specificity is the most crucial issue in the assessment of clinical competence • Broad sampling across cases is essential “Multiple Biopsies” • Logistics will be difficult

45. Direct Observation Clinical Encounter ExaminationThe DOCEE • Three hour examination of four 45 minutes observed long cases • Real patient selected from a predetermined blue print of common diseases • Check List – clinical encounter domains • Reliability 0.8 (Cronbach’s alpha) (Hamdy et al, Med Educ, 2003)