Vision Guided Robotics

1. Vision Guided Robotics and Applications in Industry and Medicine Matthias Rüther

2. Contents • Robotics in General • Industrial Robotics • Medical Robotics • What can Computer Vision do for Robotics? • Vision Sensors • Issues / Problems • Visual Servoing • Application Examples • Summary

3. Robotics • What is a robot? "A reprogrammable, multifunctional manipulator designed to move material, parts, tools, or specialized devices through various programmed motions for the performance of a variety of tasks" Robot Institute of America, 1979 • Industrial • Mostly automatic manipulation of rigid parts with well-known shape in a specially prepared environment. • Medical • Mostly semi-automatic manipulation of deformable objects in a naturally created, space limited environment. • Field Robotics • Autonomous control and navigation of a mobile vehicle in an arbitrary environment.

4. Robot vs Human • Human advantages: • Intelligence • Flexibility • Adaptability • Skill • Can Learn • Can Estimate • Robot Advantages: • Strength • Accuracy • Speed • Does not tire • Does repetitive tasks • Can Measure

5. Industrial Robot • Requirements: • Accuracy • Tool Quality • Robustness • Strength • Speed • Price Production Cost • Maintenance Production Quality

6. Medical (Surgical) Robot • Requirements • Safety • Accuracy • Reliability • Tool Quality • Price • Maintenance • Man-Machine Interface

7. What can Computer Vision do for Robotics? • Accurate Robot-Object Positioning • Keeping Relative Position under Movement • Visualization / Teaching / Telerobotics • Performing measurements • Object Recognition • Registration Visual Servoing

8. Vision Sensors • Single Perspective Camera • Multiple Perspective Cameras (e.g. Stereo Camera Pair) • Laser Scanner • Omnidirectional Camera • Structured Light Sensor

9. Vision Sensors • Single Perspective Camera

10. Vision Sensors • Multiple Perspective Cameras (e.g. Stereo Camera Pair)

11. Vision Sensors • Multiple Perspective Cameras (e.g. Stereo Camera Pair)

12. Vision Sensors • Laser Scanner

13. Vision Sensors • Laser Scanner

14. Vision Sensors • Omnidirectional Camera

15. Vision Sensors • Omnidirectional Camera

16. Vision Sensors • Structured Light Sensor Figures from PRIP, TU Vienna

17. Issues/Problems of Vision Guided Robotics • Measurement Frequency • Measurement Uncertainty • Occlusion, Camera Positioning • Sensor dimensions

18. Visual Servoing • Vision System operates in a closed control loop. • Better Accuracy than „Look and Move“ systems Figures from S.Hutchinson: A Tutorial on Visual Servo Control

19. Visual Servoing • Example: Maintaining relative Object Position Figures from P. Wunsch and G. Hirzinger. Real-Time Visual Tracking of 3-D Objects with Dynamic Handling of Occlusion

20. Visual Servoing • Camera Configurations: End-Effector Mounted Fixed Figures from S.Hutchinson: A Tutorial on Visual Servo Control

21. Visual Servoing • Servoing Architectures Figures from S.Hutchinson: A Tutorial on Visual Servo Control

22. Visual Servoing • Position-based and Image Based control • Position based: • Alignment in target coordinate system • The 3D structure of the target is rconstructed • The end-effector is tracked • Sensitive to calibration errors • Sensitive to reconstruction errors • Image based: • Alignment in image coordinates • No explicit reconstruction necessary • Insensitive to calibration errors • Only special problems solvable • Depends on initial pose • Depends on selected features End-effector target Image of end effector Image of target

23. Visual Servoing • EOL and ECL control • EOL: endpoint open-loop; only the target is observed by the camera • ECL: endpoint closed-loop; target as well as end-effector are observed by the camera EOL ECL

24. Visual Servoing • Position Based Algorithm: • Estimation of relative pose • Computation of error between current pose and target pose • Movement of robot • Example: point alignment p1 p2

25. p1m p2m d Visual Servoing • Position based point alignment • Goal: bring e to 0 by moving p1 e = |p2m – p1m| u = k*(p2m – p1m) • pxm is subject to the following measurement errors: sensor position, sensor calibration, sensor measurement error • pxm is independent of the following errors: end effector position, target position

26. Visual Servoing • Image based point alignment • Goal: bring e to 0 by moving p1 e = |u1m – v1m| + |u2m – v2m| • uxm, vxm is subject only to sensor measurement error • uxm, vxm is independent of the following measurement errors: sensor position, end effector position, sensor calibration, target position p1 p2 u1 v1 v2 u2 d1 d2 c1 c2

27. Visual Servoing • Example Laparoscopy Figures from A.Krupa: Autonomous 3-D Positioning of SurgicalInstruments in Robotized LaparoscopicSurgery Using VisualServoing

28. Visual Servoing • Example Laparoscopy Figures from A.Krupa: Autonomous 3-D Positioning of SurgicalInstruments in Robotized LaparoscopicSurgery Using VisualServoing

29. Registration • Registration of CAD models to scene features: Figures from P.Wunsch: Registration of CAD-Models to Images by Iterative Inverse Perspective Matching

30. Registration • Registration of CAD models to scene features: Figures from P.Wunsch: Registration of CAD-Models to Images by Iterative Inverse Perspective Matching

31. Tracking • Instrument tracking in laparoscopy Figures from Wei: A Real-time Visual Servoing System for Laparoscopic Surgery

32. Summary • Computer Vision provides accurate and versatile measurements for robotic manipulators • With current general purpose hardware, depth and pose measurements can be performed in real time • In industrial robotics, vision systems are deployed in a fully automated way. • In medicine, computer vision can make more intelligent „surgical assistants“ possible.