Dynamic Augmented Reality for Haptic Display in Robot-Assisted Surgical Systems

1. ( Above) Two pairs of strain gauges in full bridge configuration to maximize sensitivity and measure bending forces. Dynamic Augmented Reality for Haptic Display in Robot-Assisted Surgical Systems • THE PROBLEM • Lack of haptic feedback in tele-operated systems • This lack of feedback is detrimental to performance of some surgical tasks • TESTING • Validate system using ten surgeons (mixture of experienced and inexperienced) with phantom and animal models to perform the following tasks: • 1) Suturing along a straight line • 2) Knot tying with and without augmented reality system • Validation conducted with and without the augmented reality system. • All experiments performed at the Johns Hopkins Medical Institutions • THE IMPACT • First augmented reality system for robot assisted surgery where forces are displayed to surgeons in an intuitive and accessible way. • Collaboration with Intuitive Surgical Inc, the manufacturer of da Vinci surgical robot for software and hardware. ( Top) Da Vinci System • PREVIOUS WORK • Audio and visual displays to show forces sensed during knot-typing task • Showed sensory substitution of haptic information improves repeatability of applied suture forces and minimizes suture breakage. • OUR APPROACH • Use two DoF force sensing on teleoperated surgical instruments and a dynamic augmented reality system displaying in real time. • 1) Strain gauges added to lower shaft of needle driver tool of da Vinci Surgical System to maximize sensitivity with a linear response and provide measurement of bending forces. • 2) Orientation and position of tool tips and endoscope are tracked using robot kinematics then registered to endoscope data through computer vision • 3) Add a bar graph overlaying each tool, whose relative dimensions and colors change in response to the forces measured at tool tip ( Top) Design for Haptic Display of Bending Forces on Da Vinci Tools • PEOPLE INVOLVED • Graduate Students: Takintope Akinbiyi, Carol Reiley • Engineering Faculty: Allison M. Okamura • REFERENCES • T. Akinbiyi, A. M. Okamura, D. D. Yuh, “Dynamic Augmented Reality for Haptic Display in Robot-Assisted Surgical Systems.” • M. Kitagawa, D. Dokko, A. M. Okamura, B. T. Bethea, and D. D. Yuh, "Effect of Sensory Substitution on Suture Manipulation Forces for Surgical Teleoperation." Engineering Research Center for Computer Integrated Surgical Systems and Technology