Feedback

1. Feedback Elisabetta Bevacqua, Dirk Heylen, , Catherine Pelachaud, Isabella Poggi, Marc Schröder

2. Roddy + Ruth

3. Modeling

4. Listener system • Listener’s modules: generator of listener’s behaviours • Input: video and audio data from real world • Player: 3D agent Greta • Backchannel library: lexicon of backchannel signals • Whiteboard Psyclone: communication protocol system

5. Reactive Listener module • The reactive module generates listener’s responses according to speaker’s head movement • To detect head movements we integrated Watson (Gratch et al.) • At the moment Greta reacts with a head nod every time the speaker performs a nod or a shake • In the future Greta will be able to react with different backchannel signals and/or copy the speaker’s head movement

6. Analysis • Head movements • tracking • classification

7. Television

8. Data Annotation (SAL)

9. Cognitive Listener module • We use the SAL Wizard of Oz to trigger deliberative listener behaviours for Greta • Pre-calculated FAP files are selected according to the wizard’s decision • The Player displays the selected FAP files

10. Backchannel lexicon • We aim at building a listener ECA able to display backchannel signals according to its style of behaviour: assertive/not assertive, believing/not believing, interested/not interested and so on • We need to define a set of backchannel signals that users are able to interpret and understand • To define such a library of recognizable signals we performed a perceptive test

11. Perception/Feedback • Samples* for subjects to judge • questionnaires (semantic scales) • ask to label things • Facial Expressions • Affect Bursts

12. Perceptive Test • Perceptive test: find a mapping between signals and meanings • Questions: • it is possible to identify a signal (or a combination of signals) for each meaning, • a combination of signals can alter the meaning attached to each backchannel single signal.

13. Subjects and material • Sixty French subjects (age mean 20.1) • Tasks: select meanings for facial expressions and head signals displayed by Greta • 21 different signals • 12 meanings: • agree, disagree, accept, refuse, interested, not interested, believe, disbelieve, understand, don't understand, like, dislike • As the list of meanings was too long, subjects were divided into two groups: group1 and group2

14. Signals • Signals can be simple (containing just a single action) or complex (containing several actions): 1tension of the lips

15. Result

16. 3

17. 11

18. First question • Q1 it is possible to identify a signal (or a combination of signals) for each meaning • agree and accept: the signal nod proved to be very significant. All signals containing nods were interpreted as signals of agreement and acceptance • like: the signal smile conveys this meaning • understand: this meaning can be conveyed through the combination of smile and raise eyevrows

19. Second question • Hyp2: a combination of signals can alter the meaning of backchannel single signals. • Tension alone and frown alone do not mean dislike, but their combination does • To convey the meaning disbelievetilt and frown must be displayed together • The signal frown means don’t understand but when a shake is added their combination loses this meaning • Tilt alone and gaze right down alone do not mean not interested, but their combination does

20. Experiment: Affect bursts as listener feedbackResearch questions 1. Affect bursts used as listener feedback => same emotion? 2. How acceptable is such feedback? (3. Difference between German and Dutch listeners?)

21. Method • Stimuli • select German affect bursts from Schröder (2003) • embed into neutral German / Dutch speaker utterance “Yeah, then I told myself, why don’t you try it <pause> and then I did it!” • 10 emotions, 2 affect bursts each=> 20 stimuli per language • e.g. Dutch + admiration-wow • e.g. Dutch + anger-growl • … • e.g. German + worry-ohoh • e.g. German + startle-ah • …

23. Results

24. Results • maybe: social appropriateness + admiration, elation, relief, worry - threat, startle, anger

25. Discussion • “Acceptability” is very ambiguous • general appropriateness in the context (intended) • strange as reaction to speaker utterance • technical aspects • mismatch between sound quality • timing of feedback • social appropriateness: display rules • social norms prescribed by one’s culture as to “who can show what emotion to whom, when” (Ekman, 1977)

26. Discussion (2) • Tentative set of display rules for affect bursts • display gratifying emotions (admiration); • display empathy emotions (elation, worry, relief); • do not display negative evaluation (disgust, contempt) • do not display aggression (anger, threat) • Can explain most observations • but not high acceptability of boredom

27. Summary and Questions • For some emotions, highly recognisable affect bursts were judged to fit well with the context • Perception of emotional feedback may depend on: • social acceptability (display rules); • semantic/pragmatic interaction between speaker utterance and affect burst; • timing of feedback; • relation between speaker/listener; • formality of the situation; • …

28. Future work • In the future we aim at: • integrating perception of audio data • defining a set of rules to decide when a reactive backchannel signal must be triggered and which signal Greta should display • defining different styles to create variety of agents (assertive/not assertive, interested/not interested, believing/not believing, and so on) and evaluating their impression on users

29. CONTEXT first phase has ended succesfully move to the next… and put the findings in context