John J. Ferrara MD Kanav Kahol PhD Phoenix Integrated Surgical Residency

1. Evaluating Surgical SkillsAnd Operating Room Performance:Education/Remediation?Certification/Credentialling? John J. Ferrara MD KanavKahol PhD Phoenix Integrated Surgical Residency

2. Evaluating Surgical SkillsChallenges • How to maintain cardinal surgical “art and science” traditions when the sands that support educational paradigms are shifting? • “Publish or perish” to “Produce (RVU’s) or perish” • “Duty” hours • “Public” opinion • Generational chasm • “Linear” educational construct

3. Generation X: The Bridge Boomers (46-64 years) Millennials (18-29 years) Defined by technology Social agenda Secular Parenthood (non-traditional) Time-driven Lead only if asked Blunt Under-consumers/TV? • Defined by work ethic • Independent • Religious • Financial success • Career-driven • Wanna be lead dogs • Kumbaya • Consumer-driven/TV The good news: they respect (boomerang back to) their elders

4. Technical Skills EvaluationLinear Construct

5. Technical Skills Evaluation Parallel Construct Simulation Environment Scalable Adaptable Technology Leveraged Integrative “Real” Environment

6. Goals • Measure Technical Skills in a Simulated Environment • Create a system to measure skill set and to provide immediate feedback to the user • “Battleship down” • Measure Technical Skills in the Operating Room • Develop and validate a system to analyze videos of operations submitted to a panel for assessment

8. Objective Proficiency Measures • Employ neurological and kinesiological features to analyze task (surgical) proficiency • Construct task decomposition based feedback system • Breaks complex motion into simpler units that are: • Easy to analyze • Easy to comprehend • Easy to modify by the user Novice Instrument movements Intermediate Rosen 2002 Expert

9. Hand Motion

10. Motorical ChunkingMeasure of Expertise Expert Novice

11. Dynamic Virtual Reality Systems for Cognitive Training • Train residents for attention, working memory, intermodal transfer • Modify technique simulators to include a cognitive layer • Treat surgery as a combination of psychomotor and cognitive skill Original Task (Laparoscopic Training) Modified to target working memory

12. Marble Mania • Hand motions similar to laparoscopy • High (0.92) correlation with basic surgical gestures • Fine motor skills based game

13. Marble Mania

14. CyberGlove Analysis Non-Dominant Hand Dominant Hand Marble Mania

15. Ambidexterity

16. Technical Proficiency on ProMIS 6.5 6.3 5.9 3.9 2.5 2.9 1.0

17. Skills Evaluation Masters Experts Intermediates Novices

18. Measuring Skills in the “Real” EnvironmentProposed Solution • Computer vision instrument automatically analyzes videos • Develop means/ranges/standard deviations • Set “minimal” performance grade • Benchmarking? • Picks up events the naked eye misses • Detailed movement analysis • Cheap, “portable”, time-efficient • Web based access to rate videos for experts • Web based training tools to train experts to rate videos

19. Video CaptureLaparoscopy • Basic apparatus • Video capture system for laparoscopic system and hand movements • Hand movements captured by external camera • Sites: ceiling/lighting system/tripod • De-identified videos • Our system “syncs” these two streams for presentation and analysis

20. Dual Capture System

21. Skills Evaluation Novice Expert Intermediate Instrument Path Inefficiency Between Groups p<0.05 Expert v Novice P<0.05 Between Groups P<0.05 Tremor

22. Benchmarking?

23. Web-Based Training • Upload/automatically analyze videos on www.ratethesurgeons.com • Experts view videos off-site • Can provide input/feedback • Novice raters • View expert ratings • Receive instruction to become proficient raters • Reward system: pair teaching

24. Correlation of Subjective Measures with Various Objective Measures 1.0 0.4

25. Validation R=0.93 p<0.05 Experts Intermediates Novices

26. Where We are Now • Validation of the technical analysis tool • Evaluation on simulators also being done with videos

27. Future Work • Enhance Database • Develop Benchmarks • Expand Skill Set Instrument Family • Patient Care Applications

28. New Simulation Tasks

29. Motion History Images

30. Virtual World “Acute Care Surgery” Training

31. Challenges“The Uncanny Valley” Avatar Masahiro Mori (1970)

32. ChallengesThe Simulation Perfect Storm • Conventional computing is dead, and with it, the first generation (six figure) simulators • Computing life measured in months • Core processors • Naturalistic computing • Gaming consoles • How to maintain a database when evaluation instruments are constantly changing?

33. Conclusions • We (all) need help • We have no magic bullet • We need genomic variation “The Two Word Definition of Dogma is Brain Dead” Zollinger (sometime during my residency)

35. Challenges Engineers

36. Clinicians

37. Video Capture • Basic apparatus • Video capture system for laparoscopic system and hand movements • Hand movements captured by external camera • Sites: ceiling/lighting system/tripod • De-identified videos • Our system “syncs” these two streams for presentation and analysis Mobile simulator unit

38. We are becoming increasingly challenged with teaching new dogs old tricks AND We are not very good at teaching old dogs new tricks Evaluation Poses a More Daunting Challenge

39. Analysis • Basic movement tracking algorithms from computer vision, an established field with myriad algorithms to track movements and predict efficacy • Proprietary state of the art tools analyze movements