Gaze Controlled Robotic Camera System

1. Gaze Controlled Robotic Camera System Anuj Awasthi Anand Sivadasan Veeral Patel

2. Outline • Background • Significance • Problem Statement • Concept • Methodology • Specific aims • Budget • Project Participation • Time Frame

3. Background • Laparoscopic robotic surgery • Eye tracker application • Visual mouse • Human factors • Computer vision based control • Face mouse • Voice control

4. Requirements in Laparoscopic Surgery • Maintain the surgical point of interest in the centre of the image. • Provide the required magnification of the area. • Produce and maintain a horizontal image of the point of interest. • Perform the preceding actions automatically, although they can be modulated by the surgeon.

5. Visual Mouse Application • Obtaining the horizontal and vertical coordinates with the eye tracker • The technique of live streaming of the horizontal and vertical coordinates • Interfacing of eye tracker and computer

6. Eye Tracker System

7. Human Factors Computer vision based control of robotic camera • Camera control based on computer vision tracking of the surgical tools. • Image processing used to differentiate surgical tool of interest from surroundings. • No input required from the surgeon. Disadvantages • Surgeon’s area of interest not taken into consideration • Assumes surgical area to be surgeon’s area of interest always • Surgeon ends up looking at corners of the screen often

8. Human Factors Face Mouse control for robotic camera • Image based system • Tracks facial features of surgeon real time • Controls camera based on pitch, yaw and roll of surgeon’s face Disadvantages • Constant face movements causes strain • Difficult to keep pace with movement of tools

9. Human Factors Voice control of Robotic camera • Uses voice and pedal controls • Uses voice recognition techniques • Set of voice commands the camera Disadvantages • Considerable burden on surgeon • Difficult to perform dual inputs

10. Significance • Reduction in work load on surgeon • Accuracy of surgical tasks • Impact on surgical time • Hands Free Control

11. Problem Statement “To develop a camera control system which reduces the work load on the surgeon without compromising on the quality of surgeon’s video display ”

12. Concept Gaze based robotic camera • Acquire gaze of the surgeon with eye tracker. • Camera manipulation using eye tracker data interfaced with robot controls

13. Robotic Hardware • A small wireless 320 X 240 resolution camera with an inbuilt transmitter • A Receiver Set • Two Servomotors (HS 422) • Links • Usbor Servo Controller • Pivot Post • Gripper • Washer, Set of Clamps, Bolts, Nuts • Eye tracker System

14. Methodology Operation site Surgeon Site

15. Surgical Site • Server System (HOST Computer) • Usbor Servo Controller • Visual C++ 6.0 Coding • Servo Motors • Robot Arm • End Effecter • Inverse Kinematics to be followed • Wireless Camera • AAA Battery supplied • Receiver

16. Surgeon’s Site • Dedicated system (Client ) • Image Acquisition through Internet • Streaming Video • Live Motion JPEG System • Image Processing • Intel’s Open CV Library • Improve Brightness and Contrast • Eye tracker System

17. Fuzzy Based Control Cluster 2 Cluster 3 Pupil Cluster 1 Cluster 4 Cluster 6 Cluster 5

18. Fuzzy C-Means Algorithm • Point of Gaze keeps fluctuating. • Entire Eye tracker screen supposed to be divided into clusters. • Fuzzy C-Means Algorithm used. • Degree of belongingness of the point of gaze to a cluster is supposed to be the Degree of Membership of the fuzzy function • Point of Gaze co-ordinates assumed to be same as co-ordinates of cluster centers.

19. Specific Aims • To cover the surgical area with camera. • To obtain the point of gaze of the surgeon with eye tracker. • To control the robotic camera based on the point of gaze coordinates. • To facilitate Surgeon’s view.

20. Budget

21. Project Participation • Robot Assembling : Veeral, Anuj & Anand • Inverse kinematics : Anuj & Anand • Software for Kinematics Control: Anuj & Veeral • Interfacing Eye tracker and Robot : Anuj, Veeral & Anand • Eye Tracker Output : Anand & Veeral

22. Time Frame

23. References • M. Farid,F. Murtagh,J.L. Starck.” Computer Display Control and Interaction using Eye-Gaze". School of Computer Science ,Belfast,UK. • Atsushi Nishikawa “Face Mouse : A Novel Human-Machine Interface for Controlling the Position of a Laparoscope” IEEE Transactions on Robotics and Automation,Vol. 19,No. 5 ,October 2003. • Murtagh F. ”Eye Gaze Tracking System-Visual Mouse Application Development”,3rd Year Training Report ,E.N.P.S. Engineering Degree, March-August 2001.

24. Reference (Contd..) • M.E. Allaf. “Laparoscopic Visual Field – Voice vs. foot Pedal interfaces for control of AESOP Robot "Surgical Endoscopy.Feb 1998. • A. Casals,J. Amat,E. Laporte. ”Automatic Guidance of an Assistant Robot in Laparoscopic Surgery” International Conference on Robotics and Automation, IEEE 1996. • R. Hurteau,S. DeSantis “Laparoscopic Surgery Assisted by a Robotic Cameraman:Concept and Experimental Results”IEEE 1994.

25. References (Contd….) • George P. Mylonas,Danail Satyanov. ”Gaze Contingent Soft tissue Deformation Tracking for Minimally Invasive Robotic Surgery” MICCAI 2005, LNCS 3749, pp. 843 – 850, 2005. • Shamsi T. Iqbal ,Brian P. Bailey. “Using Eye-Gaze Patterns to Identify User tasks”GHC04,2004

26. THANK YOU!!!!!! QUESTIONS???????