Introduction to Computer Vision and Robotics: Navigation, Localization and Mapping

1. Introduction toComputer Vision and Robotics:Navigation, Localization and Mapping Tomas Kulvicius Poramate Manoonpong

2. Robot Navigation, Localization, and Mapping “I know where I am & I am able to make a plan to reach destination”

3. A B Problems: Navigation, Localization, Mapping • “How do I get there?”  Navigation from A to B

4. Problems: Navigation, Localization, Mapping 2) “Where am I?”  Localization problem

5. Problems: Navigation, Localization, Mapping 3) “Where have I been?”  Mapping

6. Biological Inspiration

7. Biological Inspiration • MacFarland (1999) has identified three important strategies in navigation: • Pilotage: Navigation using familiar landmarks or features of some sort (visual, olfactory , etc.). For example, some insects navigate towards objects that are associated with their nest. • Compass orientation: Navigation using a particular compass direction without using landmarks. For example, the small white butterfly moves in the same direction regardless of wind direction day after day. • True navigation: The ability to navigate to a goal point without the use of landmarks and regardless of the direction. For example, Pigeons seem to use true navigation and can return to their loft no matter where they are released.

8. Biological Inspiration Compass orientation True navigation

9. Biological Inspiration • Visual sensing Landmarks (objects, mountains, lakes) • Olfactory cues  Indentify home, track food source or home • Sun compass  Orientation with respect to the sun (requires internal clock) • Magnetic compass Earth’s magnetic field (polarity & gradients) • Star compass Orientation with respect to the stars at night (e.g., North Star or Polaris) • Dead reckoning  Record the distances and directions travelled from the home point • Memory Memorize specific gradients or landmarks near their home base Some Important components for navigation

10. Biological Inspiration • Visual sensing Landmarks (objects, mountains, lakes) • Olfactory cues  Indentify home, track food source or home • Sun compass  Orientation with respect to the sun (requires internal clock) • Magnetic compass Earth’s magnetic field (polarity & gradients) • Star compass Orientation with respect to the stars at night (e.g., North Star or Polaris) • Dead reckoning  Record the distances and directions travelled from the home point The use of landmark by the digger wasp (Tinbergen’s study, 1951)

11. Biological Inspiration • Visual sensing Landmarks (objects, mountains, lakes) • Olfactory cues  Indentify home, track food source or home • Sun compass  Orientation with respect to the sun (requires internal clock) • Magnetic compass Earth’s magnetic field (polarity & gradients) • Star compass Orientation with respect to the stars at night (e.g., North Star or Polaris) • Dead reckoning  Record the distances and directions travelled from the home point

12. Biological Inspiration • Visual sensing Landmarks (objects, mountains, lakes) • Olfactory cues  Indentify home, track food source or home • Sun compass  Orientation with respect to the sun (requires internal clock) • Magnetic compass Earth’s magnetic field (polarity & gradients) • Star compass Orientation with respect to the stars at night (e.g., North Star or Polaris) • Dead reckoning  Record the distances and directions travelled from the home point

14. Biological Inspiration • Visual sensing Landmarks (objects, mountains, lakes) • Olfactory cues  Indentify home, track food source or home • Sun compass  Orientation with respect to the sun (requires internal clock) • Magnetic compass Earth’s magnetic field (polarity & gradients) • Star compass Orientation with respect to the stars at night (e.g., North Star or Polaris) • Dead reckoning  Record the distances and directions travelled from the home point Odor based navigation in rats

15. Biological Inspiration • Visual sensing Landmarks (objects, mountains, lakes) • Olfactory cues  Indentify home, track food source or home • Sun compass  Orientation with respect to the sun (requires internal clock) • Magnetic compass Earth’s magnetic field (polarity & gradients) • Star compass Orientation with respect to the stars at night (e.g., North Star or Polaris) • Dead reckoning  Record the distances and directions travelled from the home point • Memory Memorize specific gradients or landmarks near their home base Odor based navigation in ants (pheromone trails)

17. Biological Inspiration • Visual sensing Landmarks (objects, mountains, lakes) • Olfactory cues  Indentify home, track food source or home • Sun compass  Orientation with respect to the sun (requires internal clock) • Magnetic compass Earth’s magnetic field (polarity & gradients) • Star compass Orientation with respect to the stars at night (e.g., North Star or Polaris) • Dead reckoning  Record the distances and directions travelled from the home point Sahara desert ant (Cataglyphis bicolor)

19. From Biological Inspiration to Robot Implementation

20. Mobile Robots NASA’s 'curious' rover E-puck robot ARMAR III AMOS robot

21. E-puck Robot Setup

22. Sensors for Robot Navigation, Mapping, Localization

23. Sensors for Robot Navigation, Localization, Mapping • Computer vision enable a robot to see and recognize landmarks, orient using sun sensors, star sensors. Star sensor Star sensor Volpe, IROS 1999 Sun sensor

24. Sensors for Robot Navigation, Localization, Mapping • Computer vision enable a robot to see and recognize landmarks, orient using sun sensors, star sensors. • Olfactory sensors (gas or alcohol sensing) enable a robot to detect chemical substance or trail and a specific odor source. Sensor response to alcohol (70%)

25. Sensors for Robot Navigation, Localization, Mapping • Computer vision enable a robot to see and recognize landmarks, orient using sun sensors, star sensors. • Olfactory sensors (gas sensing) enable a robot to detect chemical substance or trail and a specific odor source. • Compass provide an indication of magnetic north. But unreliable when interfering with magnet or metal. Devantech Magnetic Compass Analog Dinsmore compass

27. Sensors for Robot Navigation, Localization, Mapping • Computer vision enable a robot to see and recognize landmarks, orient using sun sensors, star sensors. • Olfactory sensors (gas sensing) enable a robot to detect chemical substance or trail and a specific odor source. • Compass provide an indication of magnetic north. But unreliable when interfering by magnetic or metal. • Clock (on board)  essential in connection with a sun sensor.

28. Sensors for Robot Navigation, Localization, Mapping • Wheel encoders or joint angle sensors  measure distance traveled and change in orientation and are use for path integration (dead reckoning) . • Range finders  enable a robot to estimate its distance from objects in the environment. (IR, Ultrasonic sensors at short distances, laser scanner in longer distance). • Gyroscopes provide heading directions and to improve odometric readings. • GPS enable outdoor robots to determine (within centimeters) their latitude and longitude. Unfortunately, it does not work indoors as well as on Mars, the moon, and other planets. • Etc. Wind, pressure, sound, heat sensors!!!

29. Navigation “Moving from a starting point to a goal”

30. I. Outdoor Navigation • Robot’s initial position at “START” • Move to “GOAL” • Straight-line path is “NOT possible” due to obstacles • GOAL & Two landmarks (L1, L2) are visible on a clear day!

31. I. Outdoor Navigation • Three possible scenarios: • Clear day, visible goal, unknown distance to goal: “What sensors should we use?”

32. I. Outdoor Navigation • Three possible scenarios: • Clear day, visible goal, unknown distance to goal: • Sensors: Vision, Compass, and Sonar or IR sensors. • Strategy: “Wandering Standpoint” (Puttkamer 2000)

33. I. Outdoor Navigation • Three possible scenarios: • Clear day, visible goal, unknown distance to goal: • Sensors: Vision, Compass, and Sonar or IR sensors. • Strategy: “Wandering Standpoint” (Puttkamer 2000) • Try to reach the goal from start in direct line, when encountering an obstacle  randomly turn left/right OR check small turning angle to avoid an obstacle and then turn to that direction Move around the obstacle (boundary-following) until the goal is clear or visible then heading to the goal. q

34. I. Outdoor Navigation • Three possible scenarios: • Clear day, visible goal, unknown distance to goal: • Sensors: Vision, Compass, and Sonar or IR sensors. • Strategy: “Wandering Standpoint” • Vision to recognize the goal • Keeping the goal image in the center of a view finder • Moving in the same compass direction toward the goal • If an obstacle is detected (the goal is not visible), then move to left or right (randomly) and travel around the obstacle • When the original compass direction is detected and/or the goal is visible, the robot can turn and change its heading then cont. toward the goal

35. I. Outdoor Navigation • Three possible scenarios: • Clear day, visible goal, unknown distance to goal: • Sensors: Vision, Compass, and Sonar or IR sensors. • Strategy: “Wandering Standpoint” • Vision to recognize the goal • Keeping the goal image in the center of a view finder • Moving in the same compass direction toward the goal • If an obstacle is detected (the goal is not visible), then move to left or right (randomly) and travel around the obstacle • When the original compass direction is detected and/or the goal is visible, the robot can turn and change its heading then cont. toward the goal This results in “Path 1” “However, this is a simple strategy but not optimal one! & might lead to an endless loop in case of extreme obstacle placements” Finding the best path from start to goal is called “path planning”, using e.g., wavefront, machine learning algorithm! (RL)

36. I. Outdoor Navigation • Three possible scenarios: • 2) Goal not visible from start location, landmarks visible, goal visible from landmarks: • Sensors: Vision • Strategy: “Navigation by landmarks” • Move to visible landmarks • From landmarks to the goal •  This results in “Path 2” q

37. I. Outdoor Navigation • Three possible scenarios: • 3) Goal and landmarks not visible, direction to the goal known: • Sensors: GPS or Compass & Wheel Encoders (obtain current position!) • Strategy: “Dead reckoning” q

38. Dead Reckoning • Basically this is “keeping track of your distance and your turns”, then adding them all together (“path integration”) to figure out your “total displacement”

39. Dead Reckoning Goal Don’t see the goal from starting point!!! Start Recording distance and heading direction for “path integration“

40. Dead Reckoning Goal Start X=0, Y = 0, X=5, Y = 0 ……. , q = N q = E X=10, Y = 0 Recording distance and heading direction for “path integration“

41. Dead Reckoning Goal … X=10, Y = 2.5 Start X=0, Y = 0, X=5, Y = 0 ……. , q = N q = E X=10, Y = 0 Recording distance and heading direction for “path integration“

42. Dead Reckoning Goal X_final=10, Y_final = 5, q_final = N … X=10, Y = 2.5 Start X=0, Y = 0, X=5, Y = 0 ……. , q = N q = E X=10, Y = 0 Recording distance and heading direction for “path integration“

43. Dead Reckoning Goal X_final=10, Y_final = 5, q_final = N … “Return to start” X=10, Y = 2.5 Start X=0, Y = 0, X=5, Y = 0 ……. , q = N q = E X=10, Y = 0 Recording distance and heading direction for “path integration“

44. Dead Reckoning Goal X_final=10, Y_final = 5, q_final = N Q … Back in a shortest path home (D) X=10, Y = 2.5 Start Q X=0, Y = 0, X=5, Y = 0 ……. , q = N q = E X=10, Y = 0 Recording distance and heading direction for “path integration“

45. Dead Reckoning • D = sqrt (X_final ^2 + Y_final ^2) • = arcsin(Y_final/D) • Go home: • Distance = D • Heading = 90+ Goal X_final=10, Y_final = 5, q_final = N Q Q … Back in a shortest path home (D) X=10, Y = 2.5 Start Q X=0, Y = 0, X=5, Y = 0 ……. , q = N q = E X=10, Y = 0 Recording distance and heading direction for “path integration“

46. Dead Reckoning • Control inputs to the robot: linear velocity V(t), and rotational velocity (w (t)) • Starting position (x0, y0) and orientation (q0) • The current robot pose (x, y, q) can be computed as: q (x0, y0, q0)

47. Dead reckoning & path integration: navigation of desert ants (Cataglyphis) Don’t know where is the food!!  Do random exploration to search for the food!

48. Problems: Dead reckoning & path integration for navigation

49. Problems: Dead reckoning & path integration for navigation Leg lengths were normal during the outbound journey but manipulated during the homebound run Leg lengths were manipulated during the outbound journey & the homebound run Wittlinger et al. 2006, Science,312 (5782): 1965-1967

50. Problems: Dead reckoning & path integration for navigation