1 / 52

Human – Robot Communication

Human – Robot Communication. Paul Fitzpatrick. Human-readable actions Reading human actions Conclusions. Human – Robot Communication. Motivation for communication. Motivation. What is communication for? Transferring information Coordinating behavior What is it built from? Commonality

Download Presentation

Human – Robot Communication

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Human – Robot Communication Paul Fitzpatrick

  2. Human-readable actions • Reading human actions • Conclusions Human – Robot Communication • Motivation for communication

  3. Motivation • What is communication for? • Transferring information • Coordinating behavior • What is it built from? • Commonality • Perception of action • Protocols

  4. Communication protocols Computer – computer protocols TCP/IP, HTTP, FTP, SMTP, …

  5. Communication protocols Human – human protocols Human – computer protocols Initiating conversation, turn-taking, interrupting, directing attention, … Shell interaction, drag-and-drop, dialog boxes, …

  6. Communication protocols Human – human protocols Human – robot protocols Human – computer protocols Initiating conversation, turn-taking, interrupting, directing attention, … Shell interaction, drag-and-drop, dialog boxes, …

  7. ENGAGED ACQUIRED Pointing (53,92,12) Fixating (47,98,37) Saying “/o’ver[200] \/there[325]” Requirements on robot • Human-oriented perception • Person detection, tracking • Pose estimation • Identity recognition • Expression classification • Speech/prosody recognition • Objects of human interest • Human-readable action • Clear locus of attention • Express engagement • Express confusion, surprise • Speech/prosody generation

  8. Example: attention protocol • Expressing attention • Influencing other’s attention • Reading other’s attention

  9. Foveate gaze • Motivation for communication • Human-readable actions • Reading human actions • Conclusions

  10. Human gaze reflects attention (Taken from C. Graham, “Vision and Visual Perception”)

  11. Types of eye movement Ballistic saccade to new target Right eye Vergence angle Smooth pursuit and vergence co-operate to track object Left eye (Based on Kandel & Schwartz, “Principles of Neural Science”)

  12. Engineering gaze Kismet

  13. Collaborative effort • Cynthia Breazeal • Brian Scassellati • And others • Will describe components I’m responsible for

  14. Engineering gaze

  15. Engineering gaze “Cyclopean” camera Stereo pair

  16. Tip-toeing around 3D New field of view Field of view Object of interest Narrow view camera Rotate camera Wide View camera

  17. Example

  18. Influences on attention

  19. Influences on attention • Built in biases

  20. Influences on attention • Built in biases • Behavioral state

  21. slipped… …recovered Influences on attention • Built in biases • Behavioral state • Persistence

  22. Directing attention

  23. Head pose estimation • Motivation for communication • Human-readable actions • Reading human actions • Conclusions

  24. Head pose estimation (rigid) Yaw* Pitch* Roll* Translation in X, Y, Z * Nomenclature varies

  25. Head pose literature Horprasert, Yacoob, Davis ’97 McKenna, Gong ’98 Wang, Brandstein ’98 Basu, Essa, Pentland ’96 Harville, Darrell, et al ’99

  26. Head pose: Anthropometrics Horprasert, Yacoob, Davis McKenna, Gong Wang, Brandstein Basu, Essa, Pentland Harville, Darrell, et al

  27. Head pose: Eigenpose Horprasert, Yacoob, Davis McKenna, Gong Wang, Brandstein Basu, Essa, Pentland Harville, Darrell, et al

  28. Head pose: Contours Horprasert, Yacoob, Davis McKenna, Gong Wang, Brandstein Basu, Essa, Pentland Harville, Darrell, et al

  29. Head pose: mesh model Horprasert, Yacoob, Davis McKenna, Gong Wang, Brandstein Basu, Essa, Pentland Harville, Darrell, et al

  30. Head pose: Integration Horprasert, Yacoob, Davis McKenna, Gong Wang, Brandstein Basu, Essa, Pentland Harville, Darrell, et al

  31. My approach • Integrate changes in pose (after Harville et al) • Use mesh model (after Basu et al) • Need automatic initialization • Head detection, tracking, segmentation • Reference orientation • Head shape parameters • Initialization drives design

  32. Head tracking, segmentation • Segment by color histogram, grouped motion • Match against ellipse model (M. Pilu et al)

  33. Mutual gaze as reference point

  34. Mutual gaze as reference point

  35. Tracking pose changes • Choose coordinates to suit tracking • 4 of 6 degrees of freedom measurable from monocular image • Independent of shape parameters In-plane rotation Translation in depth X translation Y translation

  36. Remaining coordinates • 2 degrees of freedom remaining • Choose as surface coordinate on head • Specify where image plane is tangent to head • Isolates effect of errors in parameters Tangent region shifts when head rotates in depth

  37. Surface coordinates • Establish surface coordinate system with mesh

  38. Initializing a surface mesh

  39. Example

  40. Typical results Ground truth due to Sclaroff et al.

  41. Merits • No need for any manual initialization • Capable of running for long periods • Tracking accuracy is insensitive to model • User independent • Real-time

  42. Problems • Greater accuracy possible with manual initialization • Deals poorly with certain classes of head movement (e.g. 360° rotation) • Can’t initialize without occasional mutual regard

  43. Motivation for communication • Human-readable actions • Reading human actions • Conclusions

  44. Other protocols

  45. Other protocols • Protocol for negotiating interpersonal distance Person backs off Person draws closer Beyond sensor range Too far – calling behavior Too close – withdrawal response Comfortable interaction distance

  46. Other protocols • Protocol for negotiating interpersonal distance • Protocol for controlling the presentation of objects Comfortable interaction speed Too fast – irritation response Too fast, Too close – threat response

  47. Other protocols • Protocol for negotiating interpersonal distance • Protocol for controlling the presentation of objects • Protocol for conversational turn-taking • Protocol for introducing vocabulary • Protocol for communicating processes Protocols make good modules

  48. Cameras Eye, neck, jaw motors Ear, eyebrow, eyelid, lip motors QNX Motor ctrl Attent. system Eye finder L dual-port RAM sockets, CORBA Tracker Dist. to target Motion filter Face Control Percept & Motor NT speech synthesis affect recognition Emotion Drives & Behavior Skin filter Color filter audio speech comms Speakers Recog. pose Track pose Track head CORBA Linux Speech recognition Linux Speech recognition CORBA Microphone

  49. Wide Camera Left Foveal Camera Right Foveal Camera Wide Camera 2 Wide Frame Grabber Left Frame Grabber Left Frame Grabber Right Frame Grabber Skin Detector Color Detector Motion Detector Face Detector Distance to target Eye finder Foveal Disparity Tracked target W W W W Wide Tracker Attention Salient target Ballistic movement Disparity Locus of attention Behaviors Motivations Fixed Action Pattern Saccade w/ neck comp. VOR Smooth Pursuit & Vergence w/ neck comp. Affective Postural Shifts w/ gaze comp. Arbitor Eye-Head-Neck Control . .. Motion Control Daemon 2 2 2 2 Q, Q, Q q , dq ,dq q , dq ,dq q , dq ,dq q , dq ,dq v s f p f p v s p v s f Eye-Neck Motors

  50. Other protocols • What about robot – robot protocol? • Basically computer – computer • But physical states may be hard to model • Borrow human – robot protocol for these

More Related