On the interaction of audio and visual cues to friendliness in interrogative prosody

1. On the interaction of audio and visual cues to friendliness in interrogative prosody David House

2. Perceiving question intonation • Question intonation in Swedish • Raised topline (Gårding 1979) • Also delayed peak (e.g. Neapolitan Italian)?

3. Perception experiment • Test sentence: Hon vill bara flyga.?(She only wants to fly.?) • Final focal accent on flyga (accent 2) • Variables: peak timing and peak height • Task: question or statement

6. Interrogative face? Slow nod + Eyebrow lowering Smile + Fast nod Statement Question

7. Percent question response INTERROGATIVE FACE DECLARATIVE FACE 100 100 80 80 60 60 Percent Percent high pitch high pitch low pitch low pitch 40 40 20 20 0 0 1 2 3 4 5 6 1 2 3 4 5 6 Stimulus number Stimulus number

8. Interrogative face • Only minor influence of face • Face supporting or confusing • Similar to results from Srinivasan & Massaro (2003) • Face gestures not good enough? • What are face gestures for? • Another possibility • Expression of speaker orientation towards addressee (attitude)

9. Recordings of users of public information kiosk Animated talking-head agent, August Sept. 1998 – Feb. 1999 Recording environment

10. Final rise Final low … vad heter du? …what’s your name?

11. Results: final rise

12. Functions of final rises • August database analyzed according to presumed intentions of the speakers • (Bell and Gustafson 1999) • Socializing or information seeking • Men/Women • 40% socializing • 60% information seeking • Children • 50% socializing • 50% information seeking

13. Perception of a friendly question

14. Subjects and procedure • Pair-wise test • Within pitch level: all stimuli 1-2, 1-3, 1-4, 2-3, 2-4, 3-4 • Across pitch level: same peak position 1-1, 2-2, 3-3, 4-4 • Total of 32 stimuli (reversed order 16 + 16) • Random order, forced choice, one repetition of each stimulus • Task • Choose the friendlier sounding question of the pair • Subjects • 14 native speakers of Stockholm Swedish

15. Results

16. Results

17. Audiovisual stimuli I

18. Info face Early low (F0 peak) Late high

19. Friendly face Early low (F0 peak) Late high

20. Results: face from parameters Scores from 27 subjects

21. Stimulus 1L Rule-based prosody: (Carlson & Granström, 1973) (Bruce & Granström, 1989) Stimulus 4H … vad heter du? …what’s your name?

22. Results (audio only)

23. Comparing pairs

24. Data-driven visual synthesis • Jonas Beskow, Mikael Nordenberg & Magnus Nordstrand (Thanks!) • Corpus of acted emotional speech • Optical motion capture • (Beskow et al. 2004; Nordstrand et al. 2004) • Evaluation • (Beskow & Cerrato 2005)

25. Audiovisual stimuli II

26. Angry face database Early low (F0 peak) Late high

27. Happy face database Early low (F0 peak) Late high

28. Neutral face database Early low (F0 peak) Late high

29. Results: face from data Scores from 27 subjects

30. Results Parametric face Data-driven face

31. Biological codes: intonation • Universal meanings of intonation (Gussenhoven 2002) • Frequency code (Ohala 1984) • high F0 (pitch) = question (submissive and non-assertive) • Effort code • increased effort = focal accent (dominant and assertive) • Production code • high F0 (pitch) = new topic, low F0 (pitch) = phrase ending