Conclusions

1. Development of a Comprehensive Set of Assessment Tools for Evaluation of Procedural Skills in Internal Medicine Checklist Survey Round 2 Checklist Survey Round 1 Final Checklist Angoff Ebel Delphi Alison Walzak, MD1; Mary E. Brindle, MD2, Maria Bacchus, MD, MSc 1, Jeffrey Schaefer, MD, MSc1, Irene Ma, MD, MSc1,3 1 Department of Medicine, University of Calgary, Calgary, AB. 2 Department of Surgery, University of Calgary, Calgary, AB. 3 W21C, University of Calgary, Calgary, AB. Results Methods Results Conclusions References • Seven mandated Internal Medicine bedside procedures: Intubation, Paracentesis, Thoracentesis, Lumbar Puncture, Knee Aspiration, Arterial Blood Gas, Central Venous Catheter insertion • Traditional evaluation methods of procedural skills lack standardization • Informal, subject to recall bias, opportunity-dependent • Variable patient, examiner, and trainee factors • Simulation can be used for procedural skill evaluation • Promotes patient safety • Standardized conditions Table 1: Inter-rater and internal reliability of LP checklist and GRS • Video review (n=20) • Trainee performances evaluated independently by 2 trained experts • Assessment tools: LP checklist (Figure 2) and Global Rating Scale (GRS) (Figure 3) • Internal reliability (Cronbach’s alpha) and inter-rater reliability (intraclass correlation coefficient, ICC) of each tool calculated Table 2: Inter-rater reliability of individual GRS items Figure 2: LP Checklist Assessment Tool Objectives Methods Background As part of a systematic approach to developing a comprehensive set of validated assessment tools for the seven bedside procedures mandated by the Royal College of Physicians and Surgeons of Canada for Internal Medicine, this study aims to validate and determine the reliability of two assessment tools for Lumbar Puncture (LP). Table 3: LP Checklist standard setting using Angoff and Ebel methods • Twenty-one item checklist created • Literature review • Each item graded on three-point scale • Expert panel revision (modified Delphi method) • N=6 (neurologist, emergency physician, general internist, adult hematologist, pediatric hematologist, anesthesiologist) • Standard setting: Angoff and Ebel methods (Figure 1) Figure 3: GRS Assessment Tool • High inter-rater reliability for both GRS and LP checklist • Internal consistency high for GRS but low for LP checklist • Lack of agreement on MPL between GRS and LP checklist The Royal College of Physicians and Surgeons of Canada. Objectives of Training in Internal Medicine. 2009. Available at: http://rcpsc.medical.org/information/index.php?specialty=136&submit= Select. Accessed April 18 2011. Clayton, M. Delphi: A technique to harness expert opinion for critical decision making tasks in education. Educ Psychol. 1997(17): 373-86. Angoff, W. Scales, norms, and equivalent scores. In: Thorndike, R, ed. Educational Measurement 2nd ed. Washington, DC: American Council on Education; 1971: 508-600. Ma, IWY et al. Comparing the use of global rating scale with checklists for the assessment of central venous catheterization skills using simulation. Adv in Health Sci Educ. Epub Aug 30 2011. • Importance • (5-point scale) • Inclusion • Additional comments • Difficulty level (easy/med/hard) • Importance (Marginal/important/essential) • Borderline trainees Figure 1: Checklist creation protocol .