Opportunistic Perception

1. Opportunistic Perception Paul Fitzpatrick

2. machine perception training data learning perception  

3. machine perception experience example detection training data   training data learning perception  

4. machine perception environment robot behavior experience   experience example detection training data   training data learning perception  

5. Active Segmentation: an active vision approach to object segmentation

6. active segmentation • Object boundaries are not always easy to detect visually • Solution: Robotsweeps arm through ambiguous area • Any resulting object motion helps segmentation • Robot can learn to recognize and segment object without further contact

7. car table segmentation example

8. what is it good for? • Not always practical! • No good for objects the robot can view but not touch • No good for very big or very small objects • But fine for objects the robot is expected to manipulate Head segmentation the hard way!

9. learning about and exploiting affordances a toy car it rolls forward a bottle it rolls along its side with Giorgio Metta a toy cube it doesn’t roll easily a ball it rolls in any direction

10. Feel the Beat: using amodal cues for object perception with Artur Arsenio

11. amodal versus modal cues amodal mode-specific nested amodal timing color synchronicity location pitch duration intensity temperature rate shape … rhythm texture

12. One object (the car) making noise Another object (the ball) in view Problem: which object goes with the sound? Solution: Match using amodal cues (period) and intermodal cues (relative phase) 8 7 6 5 frequency (kHz) 4 3 50 2 1 0 car position 0 0 500 1000 1500 2000 -50 0 500 1000 1500 2000 2500 60 ball position 50 40 0 500 1000 1500 2000 2500 time (ms) matching sound and vision

13. Cross-modal object recognition Causes sound when changing direction after striking object; quiet when changing direction to strike again Causes sound while moving rapidly with wheels spinning; quiet when changing direction Causes sound when changing direction, often quiet during remainder of trajectory (although bells vary)

14. Car Hammer Cube rattle Snake rattle 2 0 Log(visual peak energy/acoustic peak energy) -2 -4 -6 -8 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 Log(acoustic period/visual period) Cross-modal object recognition 4 Cross-modal recognition rate: 82:1%

15. robot is looking towards its arm as human moves it robot is looking away from its arm as human moves it appearance, sound, and action of the arm all bound together sound detected and bound to the motion of the arm recognizing the body

16. Shadowy Contacts: Time to contact from shadows with Eduardo Torres-Jara

17. visually-guided touching using shadows Robot sees target, arm, Robot moves to reduce Robot moves to reduce and arm’s shadow visual error between visual error between arm and target arm’s shadow and target

18. shadow cast by weak ambient light

19. shadow cast by strong directional light

20. time to contact estimation

21. reflection of arm in mirror

22. reflection of arm in water

23. reflection of arm on acrylic

24. detecting object shadows

25. Platform Shoe: shoes as a platform for vision with Charlie Kemp

26. 1 2 3 4 5 6 7 8 9 10 11 12 view from a shoe

27. detecting when the foot is planted • darker image • motion blur • large time derivative • lighter image • motion blur • large time derivative • average image • no motion blur • small time derivative

28. the features Image brightness Temporal derivative Spatial derivative Combined & Filtered

29. gait analysis spatial derivative temporal derivative image brightness combined & filtered swing/planted detection orientation

31. ground segmentation

32. extract stable views for recognition