Chapter 6 Audiovisual Speech Perception

1. Chapter 6Audiovisual Speech Perception Perry C. Hanavan, Au.D.

2. Questions Lipreading and speechreading difined the same in the text book? True False

3. Seeing at the Speed of Sound • Watch: What It’s Like to Read Lips

4. Definitions • Lipreading • the person relies ONLY on the visual speech signal and other visual cues provided by the talker • Speechreading • the person attends to both the visual AND auditory cues provided by the talker

5. General Tendencies • Normal hearing persons rely on speechreading • Persons listening in background noise tend to use visual cues for speech recognition • Persons with hearing loss tend to rely more on the visual signal for speech recognition • The greater the hearing loss, the more the tendency for reliance on visual cues

6. Speechreading • Lip cues • Facial expression cues • Gesture cues • Body language cues • Linguistic • Situational cues • Auditory cues*

7. Speechreading for Communication • Normal hearing adults • Infants

8. Afternoon Clinic Appointments • You arrive at work following lunch and find two patients waiting for clinical services. • Dr. White is a 50 year old physician • Mr. Black is a 20 year old assembly line worker in a noisy factory • Who do you predict will have the better speechreading skills?

9. Characteristics of Good Speechreaders • Visual word decoding • Working memory (ability to store and manipulate items in memory simultaneously) • Lexical identification speed (determining whether letters are a word) • Phonological processing speed (whether two words rhyme) • Verbal inference (how well one can complete a sentence with missing words) • Age (Honnel et al 1991) (younger better)

10. Characteristics of Good Speechreaders • Ability to capitalize on contextual cues • Willingness to guess • Mental agility • Willingness to revise interpretations of partially recognized messages • Linguistic skills • World knowledge • Modal differences (Erber, 1974) • Neurophysiological (Summerfield, 1992) • Cognitive (Ronnberg et al, 1999)

11. Factors that Influence Speechreading Difficulty

12. Predictive Powers

13. Lipreading: Follow the Eyes • Lip shapes vary based on phoneme(s) produced • Eyebrows rise with questions • Tend to gaze at eyes, nose and mouth • Occasional looks at forehead, cheeks and chin • Prosodic information—lipreader monitored upper face such as forehead wrinkling, eyebrow rising, and eye widening

14. Gaze Saccades/Patterns

15. Gaze-direction-based MEG averaging during audiovisual speech perception

16. Lipreading: Follow the Eyes • Prosodic judgments: • lipreader monitored upper face such as forehead wrinkling, eyebrow rising, and eye widening

17. Lipreading: Follow the Eyes • Phonetic judgments: • monitored lower face such as lip and jaw movement

18. Factors Influencing Speechreading

19. Difficulty Lipreading • One third speech sounds visible • mid and back consonants invisible • Vowels not highly visible • Rapidity of speech – 150 to 250 word/min • Coarticulation • Stress can change appearance of word • Talker variability • Visemes and homophenes

20. Question Define viseme: • Lip gloss • Seeing eye dog • Groups of speech sounds that APPEAR identical on the lips • Words that LOOK identical on the mouth E. C & D

21. Visemes and Homophenes • Visemes • Groups of speech sounds that APPEAR identical on the lips • p b m – f v – s z – n t d s z – k g • Homophenes 40-60 % of words • Words that LOOK identical on the mouth • pan ban man

22. Consonants Grouped as Visemes

23. Homophenous Word Pairs

24. Vowels and Lipreading • Vowels not considered highly visible • Vowels tend to be audible • Intense, long duration • Front vowels • Lips flat or spread • Back vowels • Lips rounded

25. Production of Vowels • Cardinal Vowel Chart (YouTube)

26. Consonants • Place of production • Individuals with high frequency hearing loss may have difficulty hearing place cues • F2 transition (1000-2400 Hz range) • Place cues tend to be visible • Bilabial, labiodental, linguadental sounds visible • Manner of production • Cues not visible, must be heard • F1 transition (250-1000 Hz range) • Voicing • Cues not visible • Low frequency range

27. Vowel Formants

28. Consonants and AV

29. Mumble Looks away when speaking Chews gum Unusual accent Speech impediment Smiles too much Moves around while talking No facial expression Shouts High pitch voice Talks to rapidly Uses long complicated sentences Wears a beard/mustache Wears dark glasses Difficult Speaking Behaviors

30. Talkers • Easier to lipread someone familiar • Family members, teachers, etc. • Females easier to lipread than males • However, auditory plus vision may be more difficult as females are less audible to person with hearing loss

31. Question Clear speech…which is not a principle of? • Talk slightly faster • Talk slightly louder • Talk slightly slower • Talk with some pauses • None of the above

32. Talker Clear Speech Principles • Cessations or pauses after phrases and sentences • Louder slightly • Enunciate all phonemes precisely and accurately (use clear speech versus conversational speech) • Accentuation (full range of voice intonation and stress on key word) • Rate of speech slowed

33. Conversation Vs. Clear Speech Shum, http://www.audiologyonline.com/articles/making-speech-more-distinct-12469

34. Conversation Vs. Clear Speech Shum, http://www.audiologyonline.com/articles/making-speech-more-distinct-12469

35. Conversation Vs. Clear Speech Shum, http://www.audiologyonline.com/articles/making-speech-more-distinct-12469

36. Message • Structure – complexity of message, frequency of use, linguistic context • Frequency of usage – how often a word occurs in everyday conversations • Neighborhoods – fewer lexical neighbors can be beneficial • Context – words specified by context are easier

37. Lexical Neighborhoods

38. Environment • Viewing angle – face to face • Distance – favorable seating • Room conditions – lighting, lighting angle, shining light, interfering objects, room noise

39. Speechreader • Audibility • Use of appropriate amplification system, ALDS, cochlear implant • Use of eyeglasses if necessary • Emotional and physical state

40. Tadoma Method: Speechreading • Deafblind Users of Tadoma Speechreading Method

42. Heather Whitestone: 1994 I find lip reading very stressful and frustrating because I am often confused. For example, if you look at person’s lips saying dog and saw, they look the same. With my hearing aid alone, I do not hear "s" or "d" sounds. So usually I have to use my common sense. For example, if someone said, "The dog is running across the street." Then I knew it was not the saw who ran across the street – it was the dog. Most hearing people do not understand that people in my position have to think incredibly fast in order to keep up with conversations. One-on-one conversations are not that stressful, but group conversations when coupled with background noise are nearly impossible. Lip reading is a grueling and exhausting mental exercise and lip readers are constantly thinking and trying to discern what is actually being said. I get real mad at those who think that I am stupid simply because I cannot hear. The truth is I get exhausted after a while and simply cannot keep up. At that point, I begin to guess at what is being said and eventually give up and choose to be quiet.

43. Baldi • Meet Baldi (iPhone app) • Baldi (2 iPhone app)

44. Question An oral interpreter? • Repeats message in view of person with hearing loss • Reads famous speeches • Speaks for person who uses ASL • All the above • None of the above

45. Oral Interpreters • A trained professional who sits in clear view of a person with a hearing loss and silently repeats a talker’s message as it is spoken

46. Bisensory Perceptions • What we see may influence what we hear • What we hear may influence what we see • McGurk Effect • McGurk Effect

47. Lip Movements • “…it is important to realize that we move our articulators to produce acoustically distinct sounds and NOT visually contrastive movements.” • Mark Ross

48. Speechreading Factoid • Campbell et al, (1981) surveyed the literature and found 38-58% of individuals with HL have accompanying visual deficiencies • Johnson et al (1981) found 65% entering NTID demonstrated defective vision • Vision loss may be greater among individuals with HL • Individuals with HL need vision evaluation***

50. Auditory plus Vision • When BOTH auditory and visual information is available, individuals with hearing loss tend to do better on communication tasks • Example (Auditory plus Vision): • Speech Recognition Score = 50% • Speechreading Score = 20% • Combined Visual/Auditory Score = 90%