3D SLAM for Omni-directional Camera

1. 3D SLAM for Omni-directional Camera YuttanaSuttasupa Advisor: Asst.Prof. Dr.AttawithSudsang

2. Outline • Introduction • Localization, Mapping, SLAM, SLAM Application • Related Work • Vision SLAM, SLAM with Omni-directional Camera • Our Problem • Challenge of problem, Propose method, SLAM Algorithm • Etc. • Scopes, Work plan

3. Localization • Robot can estimate its location with respects to landmarks in an environment

4. Mapping • Robot can reconstruct the position of landmarks that its encounter in an environment

5. Simultaneous localization and mapping (SLAM) • Robot build up a map and localize itself simultaneously while traversing in an unknown environment [Paul Newman, 06]

6. Objective • Introduce SLAM method for a hand-held omni-directional Camera moving freely in an unknown environment • Algorithm can reconstruct 3D camera path and 3D environment map

7. SLAM Challenges (Why is SLAM hard?) • “Chicken and Egg” problem • Robot needs map to localize itself • Robot needs to know its location to reconstruct landmark positions • Uncertainty problem • Sensor noise • Data association

8. SLAM • SLAM Problem • How can a robot localize its own position and build up a map in an unknown environment • SLAM Algorithm • A procedure to solve a SLAM Problem • Using a probabilistic approach to solve • Find an appropriate representation for the observation model and motion model • SLAM Application • How to applied the observation model in our interest environment • What is a motion model for our interest robot • It’s our work

9. Simultaneous localization and mapping (SLAM)

10. SLAM Application [Ahn et al.,07; Davison et al,06; ...] • Indoor Environment • Outdoor Environment • At Home • Under water • UAV Navigation [Miro et al.,06; Newman, 07; Han et al., 07; ...] [Choi et al.,06; Ahn et al,06; Motard et al.,07] [Williams et al.,01; Ribas et al.,06] [Jonghyuk Kim et al.,07]

11. Sensor Sonar Laser range finder Video Camera Omni-directional Camera

12. Vision SLAM • Stereo Vision Based SLAM [Thanh et al.,06; Schleicher et al.,06; Schleicher et al.,07; Lemaire et al.,07; Marzorati et al.,07; Han et al.,07, ...]

13. Vision SLAM • Monocular SLAM [Davison et al.,03; Eade et al.,06; Sunderhauf et al.,07; ...]

14. Omni-direction Camera • Advantage • Large field of view (360° field of view) • Real-time information @ 29.97Hz (NTSC) • Vision data with color information • Inexpensive • Disadvantage • low resolution compare to the FOV

15. Omni-direction Camera

16. SLAM with Omni-directional Camera • SLAM with Omni-directional Stereo Vision Sensor [Kim et al., 03] • Visual SLAM by Single-Camera Catadioptric Stereo [Kim et al., 06]

17. SLAM with Omni-directional Camera • Localization of mobile robots with omnidirectional vision using particle filter and iterative SIFT [Tamimi et al., 05] • Localization for Mobile Robots using Panoramic Vision, Local Features and Particle Filter [Andreasson et al., 05]

18. SLAM with Omni-directional Camera • Incremental Topological Mapping Using Omnidirectional Vision [Valgren et al., 06] • Appearance-based SLAM with Map Loop Closing Using an Omnidirectional Camera [Saedan et al., 07]

19. SLAM with Omni-directional Camera • SLAM in Indoor Environments using Omni-directional Vertical and Horizontal Line Features [Kim et al., 08]

20. Our Problem • Propose SLAM method for a hand-held omni-directional Camera • Omni-directional camera can move freely in an unknown indoor environment • Reconstruct 3D camera path and 3D environment map (landmark-based) • No need any initial information or predefine data

21. Challenge • Irregular Sensor • Uncommon sensor’s measurement model • What features should we use to be a measurement • Insufficient information • Bearing-only data • Unpredictable camera trajectory • Don’t have any initial information • A high dimensional state • Need to estimate 3D camera path • Need to estimate 3D environment map

22. Proposed Method

23. Localization Concept Initial state bel(x) x Measurement-update bel(x) observation x Time-update bel(x) x Measurement-update bel(x) observation x

24. SLAM Algorithm • Probabilistic SLAM [Smith and Cheeseman, 86] • The probability distribution of robot state and landmark locations • The observation model • The motion model

25. SLAM Algorithm • SLAM recursive algorithm • Time-update • Measurement Update

26. Using SLAM in our problem • Use EKF SLAM to solve our SLAM problem • Find an observation model • how to measurement landmarks • how to detect features from an omni-image • Find a motion model • determine how a camera move • Find a state representation • how to represent a camera state • how to represent landmarks state [Moutarlier and Chatila, 89]

27. An observation model • Detect features from an omni-image • Point features, Line features • Features association • How features associate with landmarks • Feature measurements • Observation model landmark camera

28. A motion model • Camera can move freely in an environment • Don’t know camera motion model • Cannot predict camera trajectory • Too many state variables to estimate with SLAM only • May need to pre-estimate camera state • Using pre-estimate camera state to predict camera state landmark camera time k time k + m

29. Result evaluation • Localization evaluation • Using ground truth data • Using global localization • Mapping evaluation • Using ground truth data from structural environment

30. Scope of the research • Develop SLAM method for an omni-directional Camera • Develop an algorithm to detect features from omni-directional image • A camera can move freely in 3D environment without knowledge of motion model • Algorithm can reconstruct 3D camera path and 3D environment map • Test a system in an indoor environment • No dynamic objects

31. Work Plan • Study the works in the related fields • Develop algorithms • Test the system • Evaluate a result • Prepare and engage in a thesis defense

32. Thank you

33. SLAM Solution • Solutions to the SLAM Problem • EKF SLAM - using the extended Kalman filter (EKF) to solve the SLAM problem [Moutarlier and Chatila, 89] • Fast SLAM - using the Rao-Blackwellized particle filter to solve the SLAM problem [Montemerlo et al., 02] • Etc.

34. Extended Kalman Filter • Kalman Filter • An efficient algorithm for state estimation problems • Based on linear dynamical systems • A Hidden Markov models with Gaussian distributions • Extended Kalman Filter • Nonlinear version of the Kalman filter • Using Jacobian to linearize nonlinear functions

35. Extended Kalman Filter - Predict Initial state P(x) x Predict bel(x) x Predicted state state transition function previous state control vector Predicted estimate covariance process noise covariance state transition jacobian previous error covariance

36. Extended Kalman Filter - Update Update P(x) x predict state kalman gain Updated state estimate innovation Updated estimate covariance kalman gain predicted estimate covariance innovation covariance Estimate measurement observation function Innovation estimate measurement mesurement observation jacobian Innovation covariance observation noise covariance Optimal Kalman gain

37. A state representation • represent a camera state and landmarks state correspond to a motion model and an observation model

38. Sensor • Sonar • Laser range finder • Video Camera • Omni-directional Camera • Etc. Sonar Video Camera Omni-directional Camera Laser range finder

39. How SLAM will be used • Detect good features from image • Find a good state representation for camera state and landmarks state • Find a good observation model from image features for SLAM • Find a good motion model for SLAM • Don’t know camera motion model • Need camera motion estimation

40. Feature Detection • Point Detector • Harris Detector • Scale & Affine invariant point detectors • Line Detector • Hough (line) transform [Ying and Hu, 04]

41. Feature Detection • Object Detector • Template matching • SIFT [Lowe, 04]

42. Feature Associations • Feature associations from Image • SIFT • Optical flow • Feature associations from measurement • Mahalanobis distance Real Measurement Landmark state Estimate Measurement

43. Camera Motion Estimation • Camera can move freely in an environment • Cannot predict camera trajectory • Too many state variables to estimate with SLAM only • Need pre-estimate camera state • Using relation of two period omni images • Non-linear Least Square • Iterative Extended kalman filter

44. Camera Motion Estimation • Known landmark positions related to a reference frame (from SLAM state) • Known current landmark mesurement (from current image) • Can estimate a camera state related to a reference frame landmark camera time k time k + m

45. System Coordinate • World Frame • Reference Frames • Camera Frame landmark Camera Frame World Frame Camera Camera trajectory Reference Frame

46. SLAM • Prediction • Use a camera estimation state from previous • Measurement Update • Use measurements from current and old features • Landmark Augmentation • Add a new landmark when a camera encounter a new feature • Reference Frame Augmentation • Select new reference frame from old suitable camera state

47. SLAM Application • Indoor Environment • Outdoor Environment [Ahn et al.,07; Davison et al,06; ...] [Bailey,02; Asmar et al., 06; Miro et al.,06; Newman, 07; Han et al., 07; ...]

48. SLAM Application • At Home • Under water [Choi et al.,06; Ahn et al,06; Motard et al.,07] [Williams et al.,01; Ribas et al.,06]

49. SLAM Application • UAV Navigation • Agricultural Robotics [Jonghyuk Kim et al.,07]

50. Type of SLAM • Grid maps or scans • Landmark-based [Lu & Milios, 97; Gutmann, 98: Thrun 98; Burgard, 99; Konolige & Gutmann, 00; Thrun, 00; Arras, 99; Haehnel, 01;...] [Leonard et al., 98; Castelanos et al., 99: Dissanayake et al., 2001; Montemerlo et al., 2002;...]