1 / 50

Developing an Effective Simulation Lab – “How to Evaluate Residents’ Skills with Simulation”

Developing an Effective Simulation Lab – “How to Evaluate Residents’ Skills with Simulation”. Ann Van Heest MD Director of Education Professor, University of MN Dept of Orthopedic Surgery. Simulation Skills Lab. Skills Acquisition Competency Testing. How do we measure competence?.

thetis
Download Presentation

Developing an Effective Simulation Lab – “How to Evaluate Residents’ Skills with Simulation”

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Developing an Effective Simulation Lab – “How to Evaluate Residents’ Skills with Simulation” Ann Van Heest MD Director of Education Professor, University of MN Dept of Orthopedic Surgery

  2. Simulation Skills Lab • Skills Acquisition • Competency Testing

  3. How do we measure competence? • Idea • Face validity: Does the test is going to measure what it was intended to measure? • Construct validity: Can the test differentiate between experts and novices? • Reliability: Inter-rater and Intra-rater

  4. OUTLINE • Matching the “Right” educational level for resident’s level of learning 1. Upper Extremity Surgical Skills Competency Testing 2. Articular Fracture Reduction Model 3. Basic Arthroscopy Skills Box 4. G1 Basic Skills Mandate

  5. Upper Extremity Surgical Skills Competency Testing

  6. Milestone Project Milestone Topics (n=16) Medical Knowledge and Patient Care Hip and Knee OA Hip fracture Meniscal tear Metastatic bone Rotator Cuff Injury Septic arthritis (peds) Pediatric Supracondylar Humerus fracture Degenerative Spine • ACL injury • Ankle Arthritis • Ankle Fracture • Carpal Tunnel Syndrome • Diabetic foot • Shaft fx. Femur/tibia • Distal Radius fracture • Elbow fracture

  7. Carpal Tunnel Cadaveric Station2 board-certified hand surgeon graders Detailed ChecklistGlobal Rating ScalePass / Fail Biomechanical Testing Load to Failure Post TestEvaluation Distal Radius Fixation with C-armRadiographic Scores Trigger Release Cadaveric Station2 board-certified hand surgeon graders Detailed Checklist Global Rating Scale Pass / Fail Volar Approach Distal RadiusCadaveric Stationboard-certified hand surgeon grader Detailed ChecklistGlobal Rating ScalePass / Fail 6th Lab Session: May 17, 2013 Debrief Pass/Fail Debrief Pass/Fail Debrief

  8. Motor Skills Testing: 3 Bench Stations • Global Rating Scale • Detailed Checklist • Adverse Events • Pass/Fail • Time

  9. Global Rating – Carpal Tunnel n = 2 8 5 7 4 2 Program Year p = 0.040

  10. Carpal Tunnel Release Detailed Checklist (100pts) n = 2 8 5 7 4 2 p = 0.002 Program Year

  11. Pass/Fail Assessment • Did the surgeon achieve the goal of surgery?

  12. Pass / Fail Assessment n = 0 / 21 / 75 / 0 7 / 0 4 / 0 2 / 0 Program Year p < 0.001

  13. Distal Radius Fracture Fixation

  14. Conclusions • This study reports that a surgeon’s ability to release a trigger finger does not correlate specifically to his or her ability to perform a carpal tunnel release or to perform plate fixation of a radius fracture. • The results of this study would indicate that, for 3 different surgical simulations representing procedures of varying complexity, assessments by a single assessment tool is not adequate. • To completely understand a resident’s abilities, assessment by checklist (understanding the steps of the surgery), global rating scales (assessment of basic surgical skills in light of lesser or greater complexity surgeries), and pass/fail assessment (examination of adverse events) are all necessary components.

  15. Competence • High Stakes Exam • G3 level • Pass vs Needs more practice • Competency advancement, not social advancement DEMONSTRATING COMPETENCE IS GRADUATION REQUIREMENT

  16. OUTLINE • Matching the “Right” educational level for resident’s level of learning 1. Upper Extremity Surgical Skills Competency Testing 2. Articular Fracture Reduction Model 3. Basic Arthroscopy Skills Box 4. G1 Basic Skills Mandate

  17. Surgical Simulation Training Program for Articular Fracture Surgery Jenniefer Y. Kho, MD University of Iowa Hospitals and Clinics KaramMD, Ohrt GT, Thomas GW, Yehyawi TM, Lafferty PM, Anderson DD, Marsh JL

  18. Aims • Develop a comprehensive surgical simulation training program utilizing an articular fracture model • Investigate whether simulator training can improve performance in junior residents

  19. Methods

  20. Methods

  21. Sawbones Fracture Model with Barium coated articular surface

  22. Video capture

  23. Methods - Intervention • On-line cognitive module • Video review with traumatologist • Skills module

  24. Methods • Outcome measures • OSATS global rating scale • Articular reduction • Hand motion • Fluoroscopy time, radiation dose • Face Validity (Questionnaire)

  25. Results OSATS GLOBAL RATING SCALE p= 0.68 p=0.06 Δ OSATS global rating score Intervention Control

  26. Results FLUOROSCOPY TIME p= 0.16 p= 0.03 Fluoroscopy time (s) Pretest Posttest Pretest Posttest Control Intervention

  27. Results ARTICULAR STEP--OFF p= 0.69 p= 0.22 Articular step-off (mm) Pretest Posttest Pretest Posttest Intervention Control

  28. Results CUMULATIVE HAND DISTANCE p= 0.8 p= 1.0 Cumulative hand distance (m) Pretest Posttest Pretest Posttest Intervention Control

  29. Results – Face validity Scale ranges from 1 (most) to 5 (least realistic) Overall average score 2.75 Acceptable, but room for improvement

  30. Discussion • Improved OSATS score, decrease in fluoroscopy time and radiation exposure • Articular step-off, hand motion no different • Junior residents with little to no articular fracture experience may benefit from short-term dedicated training

  31. Collaboration with University of Minnesota • Incorporation into surgical skills curriculum at the University of Iowa • Potential for expansion to other programs

  32. OUTLINE • Matching the “Right” educational level for resident’s level of learning 1. Upper Extremity Surgical Skills Competency Testing 2. Articular Fracture Reduction Model 3. Basic Arthroscopy Skills Box 4. G1 Basic Skills Mandate

  33. Low-Fidelity Arthroscopic Simulation Can Differentiate Between Experts and Novices • Jon Braman MD, • Robert Sweet, MD • David Hananel, SimPortal Lab • Paula Ludewig, Ph.D • Ann VanHeest, MD

  34. Panel of Experts • Visualization • Triangulation • Object Manipulation

  35. Triangulation • 10 LED lights • 10 switches • Randomly order lights (training) • Pre-determined order lights (testing) • Time • Errors: Alarm if touches metal • VIDEO

  36. Object Manipulation • 3 bars on left with rings • 3 bars on right with rings • Transfer rings from left to right • (Switch scope) • Time • Errors: Dropped rings • VIDEO

  37. Triangulation p = 0.0013 p = 0.0073

  38. Object Manipulation p = 0.0190 • Errors: Drops • p=1.0

  39. Basic Arthroscopic Skills • Construct Validity • Face Validity • G1 level skills: Readiness for OR • Multi-Center Trial Planned • Competency Testing: • By year in training • Translation to OR Scope Skills

  40. ABOS Surgical Simulation Mandate • “formal instruction in basic surgical skills, provided longitudinally or as a dedicated non-orthopaedic surgery rotation will be required. Skills training will need to be designed to integrate with subsequent post graduate years and should prepare the PGY 1 to participate in orthopedic surgery cases. Skills training will need a dedicated space and a curriculum which must include: • Goals and objectives and assessment metrics. • Skills used in the outpatient management of injured patients including splinting, casting, application of traction devices and other types of immobilization. • Basic operative skills including soft tissue management, suturing, bone management, arthroscopy, flouroscopy and use of basic orthopaedic equipment.

  41. ABOS/AOA-CORD/AAOS • Develop a modular outline of surgical skills curriculum • 3 meetings (Sept, March, May) • Posted on ABOS website with access to all residency programs.

  42. 1. Sterile Technique and Operating Room Set-Up 2. Knot Tying & Suturing 3. Basic Microsurgical Suturing 4. Soft Tissue Handling Techniques 5. Casting and Splinting 6. Traction 7. Compartment Syndrome 8. Bone Handling Techniques - Osteotomy 9. Fluoroscopic Knowledge and Skills 10. K-Wire Techniques 11. Techniques Basic to Internal Fixation of Fractures 12. Principles and Techniques of Fracture Reduction 13. External Fixation 14. Basic Arthroscopy Skills 15. Basic Arthroplasty Skills 16. Joint Sspiration and Injection 17. Patient Safety, Team Training, Obtaining Consent

  43. Example of Module

  44. How to measure Competency? • Upper Extremity Surgical Skills • Articular Fracture Reduction Model 3. Basic Arthroscopy Skills Box 4. G1 Basic Skills Modules • Idea • Face validity: Does the test measure what it was intended to measure? • Construct validity: Can the test teach the desired operative skills? • Reliability: Inter-rater and Intra-rater

More Related