Lagadic

1. Lagadic Visual Servoing in Robotics, Computer Vision, and Augmented Reality François Chaumette IRISA / INRIA Rennes http://www.irisa.fr/lagadic

2. The Lagadic group Spin-off of the Vista project in January 2004 Created as an Inria project in December 2004 Currently 13 people: • François Chaumette, DR 2 • Éric Marchand, CR 1, HDR 2004 • Alexandre Krupa, CR 2, recruited in Sep. 2004(LSIIT, Strasbourg) • Fabien Spindler, IR 2 • 1 temporary research scientist: C. Collewet from Cemagref • 1 temporary assistant prof.: A. Remazeilles, INSA Rennes • 5 Ph. D. students: Master in Rennes (2), Strasbourg (2) and Grenoble (1) • 1 post-doc: S. Segvic from Croatia • 1 temporary engineer: F. Dionnet from LRP Paris

3. Research field Visual servoing : vision-based control of a dynamic system • Modeling: • Control law: Usually, highly nonlinear and coupled  potential problems Objective: cook so that is as linear as possible

4. Objectives • Modeling visual features • for usual cameras (perspective projection) • for omni-directional cameras • for 2D ultrasound images • Considering high level tasks in complex environments • Robot navigation • Additional constraints (occlusions, joint limits avoidance, etc.) • Visual tracking • real-time • accurate for 6 dof • robust • mono-object • geometrical structure

5. Application fields • Robotics • Manipulating/grasping objects, target tracking • Nuclear/submarine/space/medical, etc. • Eye-in-hand/eye-to-hand systems • Robot arms, mobile robots, UAV • Augmented reality • Insert virtual objects in real images • Virtual reality • Viewpoint generation • Virtual cinematography • Control of virtual humanoid • Cogniscience

6. Experimental platforms • Eye-in-hand, eye-to hand systems, mobile robot, medical robot • Experimental validation, tests before transfer, demonstrations Experimental results very time consuming (same image never acquired, and useless after 40 ms)

7. Recent contributions

8. Modeling image moments • Determination of the analytical form of the interaction matrix for any moment • Determination of combinations of moments (from invariants) for decoupling and linearizing properties usual choice with moments

9. Visual servoing from ultrasound images • Modeling features • No observation outside B-scan corresponding to the current 2D ultrasound image • Automation of spatial calibration procedure • Adaptive visual servoing to position B-scan on a cross-wire phantom • Robotized 3D «free-hand» ultrasound imaging • Conventional 2D ultrasound probe moved by a medical robot • Thanks to calibration step, B-Scans positioned in a 3D reference frame (collaboration with Visages) • Application field: remote examination

10. Navigation from an image database to • Appearance-based representation • Topological description of the environment with key images (no 3D reconstruction) • Image path retrieval from indexing techniques (collaboration with Texmex) • Qualitative visual servoing • Navigation expressed as visual features to be seen (and not successive poses to be reached) • Confident interval for features • Automatic update of features used for navigation (by imposing a progress within the visibility corridor) from

11. Tasks sequencing • Idea : to give as much freedom as possible to take constraints (joint limits, occlusions, obstacles) into account • Scheme more reactive than reactive path planning • Scheme more versatile than classical visual servoing • Redundancy framework revisited: directional redundancy • non linear projection operator to increase the free space where secondary tasks are applied • Visual elementary task managed by a stack • Remove the good task for ensuring the constraints • Put the task back when possible

12. 3D model-based tracking • Virtual visual servoing scheme for pose computation • Virtually moves a camera so that the projection of the 3D model of the object corresponds to the observed image • Statistically robust pose estimation to deal with outliers and occlusions (M-estimation) • Real-time capabilities • Application to visual servoing and augmented reality • Extension to articulated object tracking

13. Texture and contours-based tracking • 2D model-based tracking • Estimation of an homography • Consider both edges and image intensities • 3D model-based tracking • Introducing spatio-temporal constraints in model-based tracking • Joint estimation of pose and displacement

14. Collaborations • Inside Inria : Visages (medical imaging) Icare (Predit Mobivip, Robea Bodega) • In France : 5 Robea projects • Omni-directional vision: Lasmea, Crea, Lirmm • Small helicopters: I3S, CEA • Mobile robot navigation (Lasmea, UTC) • Outside France : • ANU Canberra: modeling, helicopters • ISR Lisbon: jacobian learning • KTH Stockholm, CSIRO Melbourne, Urbana-Champaign

15. Publications • Main journals : IEEE TRA(O): 6, IJRR: 5 • Main conferences: ICRA:18, IROS:14 • Best paper award : IEEE TRA 2002, RFIA’2004 • Finalist papers : IROS’2004, AMDO’2004, ICRA’2004, IROS’2005

16. Transfert • Marker-less: 3D model-based tracker transferred to Total-Immersion for augmented reality (RIAM SORA) • France Télécom R&D: Augmented reality in urban environment • ESA: vision-based manipulation on the ISS with Eurobot

17. Software • ViSP: Open source software environment for visual servoing • Currently available for Linux and Mac OS with QPL license • Written in C++ (~ 100 000 lines of code) • Library of canonical vision-based tasks through many visual features • Suitable for 2D, 2½ D, 3D control laws • Eye-in-hand / eye-to-hand • Redundancy framework • Visual tracking algorithms • Independence wrt. the robotics platform, frame grabber • Simulator included (interface with OpenGL)

18. Positioning wrt. INRIA & French labs • INRIA scientific and technological challenges: • (4): Coupling models and data to simulate and control complex systems • (5): Combining simulation, visualization and interaction (real-time, augmented reality) • (7): Fully integrating ICST into medical technology (medical imaging, medical robotics) • Inside INRIA: • Icare (Num A: Control and complex systems): visual servoing and control • Vista, Movi, Isa: visual tracking • Other French labs: • LASMEA: visual tracking, position-based visual servoing • LSIIT: visual servoing for medical robotics • LRP, I3S

19. Worldwide positioning • Pioneering lab: CMU (1984 – 1994, no more active) • Main labs: • USA: (S. Hutchinson, G. Hager) • Australia (P. Corke), Japan (K. Hashimoto) • Europe: KTH (more recently) • Other labs : almost everywhere (Italy, Spain, Portugal, Germany, Canada, Mexico, Brazil, South Korea, China, etc.) • Visual tracking: Cambridge, EPFL • Lagadic: High visibility in the robotics community • AE IEEE TRA(O) • Look for “visual servoing” ∪ “visual servo” in Google Scholar

20. Evolution wrt. past objectives • From the 2001 Vista evaluation experts report: “Vista is planning to split off its activities in visual servoing and active vision as a separate project. This is an excellent decision” • Evolution wrt. scientific objectives: 80 % well done • Complex objects of unknown shape: image moments • Outliers: M-estimator integrated in the control loop • Applications in robotics: underwater, space, flying robots • Applications outside robotics: virtual reality, augmented reality • Visual servoing directly on image intensity: future objective

21. Objectives: modeling visual features • Spherical projection: • same model for perspective projection and omni-directional cameras • nice geometrical properties • Modeling directly the image intensity (no image processing, many unknown parameters, cooking very challenging) • Enclosing volume for 3D objects (global and sufficient information) • Mobile/flying robots: non holonomic or underactuated systems (modeling and control)

22. site B visual data site A control flow force multi-sensor control 2D trajectory 2D ultrasound images Objectives: medical robotics • Modeling adequate ultrasound features and their interaction • Automatic control of the probe motion to assist medical examination • Automatically follow an organ of interest along the patient skin • Hybrid force/vision control schemes • Remote examination without using haptic device • Robot control combining ultrasound images, force measurement and visual data of the patient provided by a remote camera • Autonomous exploration of a given area (organ, tumor)

23. Objectives: real-time visual tracking • New camera models • Omnidirectional cameras (3D model-based tracking) • Model-based vs model-free approaches • Structure estimation • Joint estimation of pose and structure “a la Debevec” • Model with some degrees of freedom following the work with articulated object • On line structure estimation during visual servoing • Joint estimation of depth and displacement (controlled SLAM) • Initialization • Object detection, recognition and localization • Image-based model of the considered object (collaboration with Vista and EPFL through FP6 Pegase proposal)