1 / 105

Perception and Perspective in Robotics

Perception and Perspective in Robotics. – Paul Fitzpatrick –. MIT CSAIL USA. experimentation helps perception. Rachel: We have got to find out if [ugly naked guy]'s alive. Monica: How are we going to do that? There's no way.

bethan
Download Presentation

Perception and Perspective in Robotics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Perception and Perspective in Robotics – Paul Fitzpatrick – MIT CSAIL USA

  2. experimentation helps perception Rachel: We have got to find out if [ugly naked guy]'s alive. Monica: How are we going to do that? There's no way. Joey: Well there is one way. His window's open – I say, we poke him. (brandishes the Giant Poking Device)

  3. robots can experiment Robot: We have got to find out where this object’s boundary is. Camera: How are we going to do that? There's no way. Robot: Well there is one way. Looks reachable – I say, let’s poke it. (brandishes the Giant Poking Limb)

  4. object segmentation affordance exploitation (rolling) manipulator detection (robot, human) edge catalog object detection (recognition, localization, contact-free segmentation) the root of all vision poking

  5. use constraint of familiar activity to discover unfamiliar entity used within it reveal the structure of unfamiliar activities by tracking familiar entities into and through them theoretical goal: a virtuous circle familiar activities familiar entities (objects, actors, properties, …)

  6. practical goal: adaptive robots • Motivated by fallibility • Complex action and perception will fail • Need simpler fall-back methods that resolve ambiguity, learn from errors • Motivated by transience • Task for robot may change from day to day • Ambient conditions change • Best to build in adaptivity from very beginning • Motivated by infants • Perceptual development outpaces motor • Able to explore despite sloppy control

  7. Head (7 DOFs) Right arm (6 DOFs) Left arm (6 DOFs) Torso (3 DOFs) Stand (0 DOFs) giant poking device: Cog

  8. giant poking device: Cog

  9. talk overview • Learning from an activity • Poking: to learn to recognize objects, manipulators, etc. • Chatting: to learn the names of objects • Learning a new activity • Searching for an object • Then back to learning from the activity…

  10. virtuous circle poking objects, …

  11. ball! virtuous circle poking, chatting objects, words, names, …

  12. poking, chatting, search search objects, words, names, … virtuous circle

  13. poking, chatting, search search objects, words, names, … virtuous circle

  14. talk overview • Learning from an activity • Poking: to learn to recognize objects, manipulators, etc. • Chatting: to learn the names of objects • Learning a new activity • Searching for an object • Then back to learning from the activity…

  15. talk overview • Learning from an activity • Poking: to learn to recognize objects, manipulators, etc. • Chatting: to learn the names of objects • Learning a new activity • Searching for an object • Then back to learning from the activity…

  16. object segmentation affordance exploitation (rolling) manipulator detection (robot, human) edge catalog object detection (recognition, localization, contact-free segmentation) poking

  17. object segmentation poking

  18. “Active Segmentation” • segmenting objects • through action

  19. “Active Segmentation” • segmenting objects • by coming into contact with them

  20. Edges of table and cube overlap Color of cube and table are poorly separated Cube has misleading surface pattern Maybe some cruel grad-student faked the cube with paper, or glued it to the table a simple scene?

  21. active segmentation

  22. active segmentation

  23. Sandini et al, 1993

  24. where to poke? • Visual attention system • Robot selects a region to fixate based on salience (bright colors, motion, etc.) • Region won’t generally correspond to extent of object • Poking activation • Region is stationary • Region reachable (right distance, not too high up) • Distance measured through binocular disparity

  25. visual attention system person approaches shakes object moves object hides object stands up attracted to skin color attracted to bright color, movement smooth pursuit attracted to skin color smooth pursuit (Collaboration with Brian Scassellati, Giorgio Metta, Cynthia Breazeal)

  26. tracking

  27. poking activation

  28. evidence for segmentation • Areas where motion is observed upon contact • classify as ‘foreground’ • Areas where motion is observed immediately before contact • classify as ‘background’ • Textured areas where no motion was observed • classify as ‘background’ • Textureless areas where no motion was observed • no information

  29. minimum cut foreground node allegiance to foreground • “allegiance” = cost of assigning two nodes to different layers (foreground versus background) pixel-to-pixel allegiance pixel nodes allegiance to background background node

  30. minimum cut foreground node allegiance to foreground • “allegiance” = cost of assigning two nodes to different layers (foreground versus background) pixel-to-pixel allegiance pixel nodes allegiance to background background node

  31. grouping (on synthetic data) proposed segmentation “figure” points (known motion) “ground” points (stationary, or gripper)

  32. point of contact 1 2 3 4 5 6 7 8 9 10 Motion spreads suddenly, faster than the arm itself  contact Motion spreads continuously (arm or its shadow)

  33. point of contact

  34. segmentation examples Side tap Back slap Impact event Motion caused (red = novel, Purple/blue = discounted) Segmentation (green/yellow)

  35. segmentation examples

  36. car table segmentation examples

  37. 0.2 cube car bottle ball 0.16 0.12 0.08 0.04 0 0 0.05 0.1 0.15 0.2 0.25 0.3 boundary fidelity measure of anisiotropy (square root of) second Hu moment measure of size (area at poking distance)

  38. signal to noise

  39. object segmentation edge catalog poking

  40. “Appearance Catalog” • exhaustively characterizing • the appearance of a low-level feature

  41. sampling oriented regions

  42. sample samples

  43. most frequent samples 1st 21st 41st 61st 81st 101st 121st 141st 161st 181st 201st 221st

  44. selected samples

  45. some tests Red = horizontalGreen = vertical

  46. natural images 00 22.50 900 22.50 450 22.50 450 22.50

  47. object segmentation edge catalog object detection (recognition, localization, contact-free segmentation) poking

  48. “Open Object Recognition” • detecting and recognizing • familiar objects, • enrolling unfamiliar objects

  49. object recognition • Geometry-based • Objects and images modeled as set of point/surface/volume elements • Example real-time method: store geometric relationships in hash table • Appearance-based • Objects and images modeled as set of features closer to raw image • Example real-time method: use histograms of simple features (e.g. color)

  50. geometry+appearance • Advantages: more selective; fast • Disadvantages: edges can be occluded; 2D method • Property: no need for offline training angles + colors + relative sizes invariant to scale, translation, In-plane rotation

More Related