eNTERFACE 08 Project 1 “ MultiParty Communication with a Tour Guide ECA” Mid-term presentation

1. eNTERFACE 08Project 1 “MultiParty Communication with a Tour Guide ECA” Mid-term presentation August 19th, 2008

2. Project Summary (Repeating the lesson ;) ) • System configuration • Work in progress • Speech Regonition • Nonverbal Input Handling • Input Understanding • Dialogue Management • Character Animation Player • Conclusion Outline

3. We aim to create an ECA based system which will interract with two users by: • Capturing their behaviors (speech, nonverbal behaviors) • Combining and handling input data • Generating and displaying response to these behaviors in the real time Summary

5. System Configuration • Input • Speech recognition • Nonverbal behavior detection (Face detection, Facial orientiation, Hand raising) • Central Part • Input Understanding, Dialogue Management • Output • ECA animation player • Platform to provide communication between the components

6. Two SR components will be running on different machines • Keywords will be dynamically changed according the conversational state • No significant overlapping between microphones (distance ~ 1.5 m) • Implementation in progress: • Keywords detection with confidence score and speech duration by using Loquendo API Speech Recognition

7. Implemented: • System which uses OpenCV to calculate difference between the frames to detect the appearance ofusers • To do: • Hand raising to ask the question by using OpenCV (If time permits) Nonverbal Inputs: Appearance of the User and Hand Raising

8. Nonverbal Inputs: Face Orientation • Implemented: • Facial Orientation Detection • Work in progress : • Face tracking: Users are starting mutual conversation or leaving the system?

9. Combines two verbal channels and image processing data to recognize users’ utterances • Examples: who is speaking, who is adressee, are users’ starting mutual conversation...? • Component is still in early stage since input components are not being implemented Input Understanding Component

10. Make decisions “when and what to do to whom”: • Handle multi-modal input events • Handle user interruptions while the agent is doing something • Keep a model of each user • Keep the domain knowledge, discourse model, and context memory • Generate multi-modal outputs Dialogue Management Component- Functionalities

11. Dialogue management component based on information state theory [Larsson’00] in still being implemented • The progress of dialogue is represented by a set of variables • These variables are updated or queried by a basic unit called dialogue move like ask, answer, repeat, inform • Conversation strategies (plans) can be composed to respond to different circumstance • Most appropriate plan are selected and scheduled by simple inference Dialogue Management Component- Progress

12. What to do when the agent is interrupted by the user? • What to do when speech recognition fails? • What to do when the users are talking to each other? • What to do when the users starts to talk to the system at the same time? • How ECA’s gaze direction can be controlled? • How grounding can be implemented using verbal and non-verbal information? • What to do when the response of the user is not what the system expected? Dialogue Management Component- Issues

13. Character animation (ECA) player generates speech and synchronized animation by using XML based GSML language • Multiple threads in animation player obtain speech and gesture synchrony • Set of animations implemented • Gazing, beats, facial expressions (joy, sad...), symbolic gestures • To do: construct ECA behaviors according the system scenario (canned behaviors, general states) Character Animation Player

14. Character Animation Player: Example

15. We are in (slow :S) progress • To integrate a baseline system we need to implement: • SR component • Component to detect facial orientation • Input Understanding • DM component • Behavior specification script for character Player • Expected Outcome: there are many... Conclusion