Laser-Camera systems and algorithms to sense the environment for robotic applications

1. Laser-Camera systems and algorithms to sense the environment for robotic applications The research project: FP7-People-Marie Curie-COFUND-Trentino-post-doc 2009 - Incoming Presenter: Dr. Ilya Afanasyev Lab. Mechatronics, University of Trento, Italy ilya.afanasyev@gmail.com, ilya.afanasyev@unitn.it PAT mid-term review, Trento, 2011

2. About the grant holder AREAS OF EXPERTISE 1. Robotics. Computer vision. Data Treatment and Analysis. 2. Experimental Physics. Optical sensors (UV, Visible, and IR). QUALIFICATIONS AND EDUCATION 2006 Ph.D. (in Technical Sciences), Vavilov State Optical Institute, St. Petersburg, Russia (www.npkgoi.ru) Specialization: Optical-electronic instruments 1994-2000 full-time student at the Baltic State Technical University, Russia (www.voenmeh.ru) Qualification: Engineer Specialization: Automatic Control Systems for flying apparatus JOB EXPERIENCE 2010 – present Marie Curie-COFUND post-doc, Mechatronics Dep., Faculty of Engineering, University of Trento, Italy 2007 – 2010 Senior engineer, Sensor Department, Development Center Russia, Giesecke & Devrient GmbH (G&D, www.gi-de.com) 2006 – 2007 Senior engineer, Aerospace Optical-Electronic Systems Dep., Vavilov State Optical Institute, Russia (www.npkgoi.ru) 2000 – 2006 Researcher/PhD student, Aerospace Physical Optics Lab., Vavilov State Optical Institute, Russia (www.npkgoi.ru) 18/10/2011

3. Status of the Project The 3-years project in Mechatronics has been doing since May 2010. The current project proposes using a Laser-Camera System (LCS) as an instrument to get 3D information of the world environment. LCS includes a scanning laser rangefinder (LRF) and RGB camera. The rotated LRF captures an object as unstructured “clouds of points” of 3D data, whereas a camera gives a 2D image of the object. The fusion of 2D and 3D sensor’s data gives full and detailed information about the object and its environment. Description of the Project

4. The calibration of laser-camera system • 2D and 3D sensor fusion should be a result of a mutual sensor calibration. The calibration is important for: • a compatibility of sensors measurements, and • having the metrical coordinate system with good accuracy. • We are using the standard camera calibration technique (Jean-Yves Bouget method). http://www.vision.caltech.edu/bouguetj/calib_doc/

5. 3D data processing • Then joint 3D and 2D data processing is used: • to recognize an object, • to estimate its orientation in 3D space, and • to improve LCS calibration by mathematical modeling. • 3D object visualization / reconstruction can be executed by math. modeling with SuperQuadrics (SQ). The explicit equation of superquadrics is The implicit equation of superquadrics is where x,y,z - superquadric system coordinates; η, ω – spherical coordinates, where a1, a2, a3 – parameters of object scaling; a4, a5 – parameters of object shape. http://lrv.fri.uni-lj.si/~franc/SRSbook/SRS.html

6. Object recognition algorithm The object recognition algorithm is based on using RANSAC (RANdomSAmple Consensus) method of SuperQuadrics model fitting to objects presented in 3D “cloud of points”. The solution is verified by evaluating the matching score between the SQ object model and 3D real data by a robust least square fitting. http://www.ing.unitn.it/~afanasye/

7. Mid-term results & achievements - Building a prototype of a robotic vehicle with installed laser-camera system The laser and the camera oriented and calibrated together have been formed the Laser-Camera system

8. Mid-term results & achievements - Testing the sensitivity (laser remission effectiveness) of the Laser-Camera System to registration of the objects from different materials (metals, plastic, wood, paper, polyethylene, etc.) at the different illumination conditions. The polyethylene is transparent for IR rangefinder, but multilayer polyethylene decreases the depth of recognition significantly.

9. Mid-term results & achievements - Capturing 3D data and development of 3D object reconstruction / recognition algorithms (for human body, etc. achieved at www.ing.unitn.it/~afanasye/) 3D Human Body Pose Estimation: photo, “cloud of points”, fitting a math. model to 3D data, final math. model of pose.

10. Future project work • According to the work plan, the future project work should be dealt with: • installation of the developed Laser-Camera System on a robotic vehicle. • research & control of robot vehicle motion. • improving Laser-Camera System calibration technique. • improving object recognition algorithms. • implementation algorithms to a robotic vehicle. • developing algorithms of relative position estimation (robot - object). • verification of the algorithms with hardware facilities.

11. Possible impacts • 3D Laser-Camera System can provide 2D/3D measurements capturing 2D/3D data of an object (with distance and color information) for indoor applications. • Object recognition and pose estimation algorithms can be applied for: • detection and reconstruction by math. models of different real objects (including complex objects, like human body). • detection, classification and selection a definite thing in a heap of different objects for robotic manufacturing (such as a search an object in mechanical details laying in mess). • different scientific projects (with a possibility of a collaboration).

12. Training experiences • During the project I participated: • at 21th Summer School in Jyväskylä University, Finland, 8-19 August 2011 (www.jyu.fi/summerschool/). • in preparation of 3 conference papers: • to 21th Int. Conference: GraphiCon'2011 (Moscow, 2011, http://gc2011.graphicon.ru/); • to RGB-D Workshop on 3D Perception in Robotics (Sweden, http://ias.cs.tum.edu/events/rgbd2011); • to Int. Conference VISAPP-2012 (Rome, 2012, February 2012, http://visapp.visigrapp.org). • Intensive course of Italian as a second language (Corsointensivodi lingua italiana per stranieri), CIAL, University of Trento, test (Livello A2), February 2011. • 3D Modelling in 3ds Max. Training at St. Petersburg State Polytechnical University, 2011. Certificate #1KQYHQDCK1.

13. Training experiences • During the project I have been engaging in: • Leading a Mechatronics group (2 Master students) on measurements of remission from different materials (wood & aluminum foil & polyethylene) by IR rangefinder. • Giving a lecture on “Object Detection with Superquadrics” in “Robotics and Sensor Fusion for Mechatronics Systems” (course of Assoc. Professor Mariolino De Cecco, University of Trento, Italy). Povo, 05 April 2011. • Close collaboration with Italian colleagues for experimental and theoretical work, making seminars with presentation: www.ing.unitn.it/~afanasye/10_UniTN_reports/20_Presentations/. • Creating, discussing and sharing some algorithms in MATLAB: www.ing.unitn.it/~afanasye/.

14. Impacts on career development • The project helps in: • developing the professional skills in computer vision and robotics, • getting an experience in resource, time and money managing, and • strengthening the researcher reputation, • that must be valuable for future work at a research center or an industrial company.

15. Acknowledgements This work of IlyaAfanasyev on laser-camera system creation and algorithms of 3D object recognition, localization and reconstruction has been supported by the grant of EU\FP7-Marie Curie-COFUND - Trentino post-doc program, 2010-2013. He is very grateful to colleagues from Mechatronics Dep. of UniTN, especially supervisor - Professor Mariolino De Cecco. Grazie!!