Каналы мультимодальной коммуникации: относительный вклад в понимание дискурса

1. Каналы мультимодальной коммуникации: относительный вклад в понимание дискурса «Мультимодальная коммуникация» 15 ноября 2013 А.А. Кибрик (ИЯз РАН и МГУ) Н.Б. Молчанова (BearingPoint) aakibrik@gmail.com

2. What is the contribution of different communication channels? • Traditional approach of mainstream linguistics: the verbal channel is so central that prosody and the visual channel are at best downgraded as “paralinguistics” • Applied psychology It is often stated that (figures go back to Mehrabian 1971): • body language conveys 55% of information • prosody conveys 38% of information • the verbal component conveys 7% of information • Who is right?

3. Relative contribution of three communication channels? DISCOURSE Vocal channelsVisual channel Verbal channelProsodic channel

4. Experimental design • Isolate the three communication channels • Present a sample discourse in all possible variants (23=8) • Present each of the eight variants to a group of subjects • Assess the degree of understanding in each case • Such assessment may lead to estimates of the contributions of communication channels

5. Studies in this line of research • Èl’bert 2006, year paper • Èl’bert 2007, diploma thesis • Reinterpreted and refined in Kibrik and Èl’bert 2008 • Molchanova 2008, year paper • Molchanova 2009, year paper • Molchanova 2010, diploma thesis • Reinterpreted and refined in Kibrik and Molchanova 2013

6. Èl’bert 2007, Kibrik and Èl’bert 2008 • Russian TV serial “Tajny sledstvija” – “Mysteries of the investigation” • Context excerpt: 8 minutes • Experimental excerpt: 3 min. 20 sec. • consisting of conversation alone, to ensure that we are testing the understanding of discourse rather than of the film in general • Two vocal channels have been separated: • Verbal: running subtitles • Prosodic: superimposed filter creating the “behind a wall” effect • Participants: • Native speakers of Russian • Eight groups of 10 to 17 participants

7. Eight experimental groups • Group 0: only the context excerpt • Groups 1 (one communication channel) • Verbal: subtitles, temporally aligned • Prosodic: filtered sound • Visual: video • Groups 2 (two communication channels): • Verbal + prosodic = original sound • Verbal + visual: subtitles and video • Prosodic + visual: filtered sound and video • Group 3: original material

8. Verbal + visual

9. Procedure • The context and the experimental excerpts were shown to a group of subjects on a large screen • Each subject answered 23 multiple-choice questions concerned with the experimental excerpt alone • What Tamara Stepanovna offers Masha before the beginning of the conversation: • a. to take off her coat • b. to have a cup of tea •  c. to have a seat • d. to have a drink • Percentage of correct answers is used as an assessment of a subject’s degree of understanding

10. Results • All three channels are substantially informative • Verbal > visual > prosodic • Integration of visual and prosodic channels is difficult

11. Molchanova 2010Kibrik and Molchanova 2013 • Methodological issues • The following aspects of the prior study have been changed (improved) • Stimulus material • Methods of isolating the channels • Questionnaire • Participants and interviewing procedure

12. Stimulus material: discourse type • Shortcomings of movies • Plot facilitates guessing • Possible familiarity with the movie • Quasi-natural behavior of actors • Solution: natural dialogue • Guessing game original.avi, 0:19 – 0:57

13. Stimulus material: speakers • Shortcomings of the prior studies • Same-sex speakers  indistinguishable in the prosody-only version • Solution: • Different sexes: F0 range is different

14. Methods of isolating the channels: Verbal channel • Shortcomings of subtitles • Subtitles belong to the visual mode • Hard to read without punctuation • Especially at the rate of speech • And especially in the “verbal + visual” condition • Solution: spoken prosody-free signal • Each word in transcript is recorded individually from the corresponding person • All thus elicited words are glued together in the right order

15. Visual + verbal (the robot condition)

16. Verbal channel • Remaining problem • Unnatural input • No reduction • No intonation • etc.

17. Methods of isolating the channels: Prosodic channel • Shortcomings of the prosodic material as used in previous studies • Excessive noise • Solution: • Loudness is decreased radically at all frequencies except for the speaker’s average F0 frequency • This has led to a more satisfactory “behind the wall” (or “behind the glass”) effect

18. Visual + prosodic (the mermaid condition)

19. Questionnaire • Shortcomings of prior studies • Èl’bert 2007: gap between Group 0 (38.3%) and Group 3 (87.4%) is insufficient • Solution • Testing stage • Identify trivial questions (high Group 0) –5 • Identify unfortunate questions (low Group 3) –2 • 30  23 • Group 0: 34.5% correct answers • Group 3: 88.0% correct answers

20. Participants and interviewing procedure • Shortcomings of prior studies • Uncontrolled social status and geographical origin of participants • Multiple participants in one room may affect each other’s performance • Need for a big screen • Solutions • Control for social status and geographical origin; homogeneous group • Comparable, independent, and comfortable conditions • Detailed guidelines • Remote implementation • Stimulus materials at Youtube.com • Questionnaire at Googledocs

21. Kibrik and Molchanova 2013: Results • Each individual channel is substantially informative and prevails over the null condition (34.5%) F-test: verbal and visual: p<0.05, prosodic: p=0.127 • Verbal (58.8%) > visual (52.2%) > prosodic (40.2%) F-test: verbal > prosodic, visual > prosodic: p<0.05, verbal > visual: p=0.071

22. Kibrik and Molchanova 2013: Results • Two-channel conditions prevail over the one-channel conditions much more clearly than in the previous experiment (verbal+prosodic – 73.5%, verbal+visual – 88.2%) F-test: all pairwise comparisons but “visual+prosodic > visual”: p<0.05; all two-channel conditions > all one- channel conditions: p<0.0001 • A dramatic dip in the visual+prosodic condition is even clearer F-test: significant difference from the two other two-channel conditions, p<0.0001

23. Kibrik and Èl’bert 2008 vs. Kibrik and Molchanova 2013 • General picture is remarkably similar • In the new study all effects are clearer

24. Normalized contribution of three channels • Suppose the three channels are independent • Sum up all percentages of individual channel contributions and normalize to 100% • Identify normalized contribution

25. Normalized contribution of three channels

26. Gender differences • Molchanova 2010: gender advantages • Percentages of correct answers

27. Conclusions • All communicatioin channels are highly significant  the traditional linguistic viewpoint is incorrect • The verbal channel is the leading one  the viewpoint popular in applied psychology is incorrect • Information from the prosodic and the visual channels is primarily used through integration with the verbal channel • Very similar results have been attained in different studies, in spite of very different methodological details

28. Further questions • Auditory or graphic presentation of the “verbal alone” channel? • Explore different discourse types, such as monologic discourse • …and: Other suggestions on this approach?

29. Acknowledgements • Olga Fedorova • Anna Laurinavičiute • Andriy Myachykov • RGNF #11-04-00153

30. Thanks for your attention visual channel language verbal channel prosodic channel