1 / 197

Speech perception as a window into language processing:

Speech perception as a window into language processing: Real-time spoken word recognition, specific language impairment, and CIs. Bob McMurray Dept. of Psychology Dept. of Communication Sciences and Disorders. Thanks to. Richard N. Aslin Michael K. Tanenhaus Meghan Clayards. Jennifer

dolan
Download Presentation

Speech perception as a window into language processing:

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Speech perception as a window into language processing: Real-time spoken word recognition, specific language impairment, and CIs Bob McMurray Dept. of Psychology Dept. of Communication Sciences and Disorders

  2. Thanks to Richard N. Aslin Michael K. Tanenhaus Meghan Clayards Jennifer Cole Allard Jongman Funding The National Institute of Deafness and other Communication Disorders Intellectual Community Larissa Samuelson John Spencer Mark Blumberg Ed Wasserman Prahlad Gupta Eliot Hazeltine Karla McGregor Amanda Owen Jean Gordon Karen Iler Kirk Chris TurnerColleen Mitchell Bruce Gantz Matt Howard Carolyn Brown & Jerry Zimmerman Steve Luck J. Bruce Tomblin Marlea O’Brien Vicki Samelson Joe Toscano Keith Apfelbaum Gwyn Rost Ashley Farris-Trimble Marcus Galle Cheyenne Munson Dan McEchron Joel Dennhardt Jessica Walker Lea Greiner

  3. Introductions “I do ba’s and pa’s”

  4. Neuroscience Computer Science Psychology Cognitive Science Diverse fields are united by their commitment to understand the basic mechanisms or processes that underlie perception, cognition, and language… wherever they occur. Philosophy Linguistics

  5. Language disorders  Useful way to justify basic research to NIH...

  6. Introductions Culture Culture Culture Culture Brain/ Body Brain/ Body Brain/ Body Brain/ Body Brain/ Body Cells Cells Cells Cells Cells Genes Genes Genes Genes Phys./Social Enviroment Phys./Social Enviroment Phys./Social Enviroment Phys./Social Enviroment Phys./Social Enviroment • Developmental Science • Development is • Multiply determined. • Product of interactions between levels of analysis. • Characterized by non-obvious causation. • Has no single end-state Behavior Behavior Behavior Behavior Genes

  7. Language disorders  Useful way to justify basic research to NIH...  Useful reminder of the multi-potential nature of language development… … but not a rigorous way to approach basic theoretical questions.

  8. “Clinical Cognitive Psychology” Child Welfare Research Station Experimental Developmental Psych. Independent Speech Pathology Iowa Test of Basic Skills ??? Carl Seashore Cora Busey Hillis & Beth Wellman Boyd McCandless & Charlie Spiker Wendell Johnson E.F. Lindquist Bruce Tomblin

  9. Language disorders  Useful way to justify basic research to NIH...  Useful reminder of the multi-potential nature of language development… … but not a rigorous way to approach basic theoretical questions.

  10. Individual differences (including disorders) in language development and outcomes:  Reveal range of variation that our theories must account for.  Allows examination of the consequences of variation in the internal structure of the system.

  11. Braitenberg (1984) “Vehicles” Light Sensor Wire Wheel/Motor Light Avoiding Light Seeking Simple mechanisms give rise to complex behavior. But many such mechanisms are possible. - Easier to understand mechanism by building outward, rather than observing inward. Disordered language users allow us to observe consequences of a change in mechanism.

  12. Individual differences (including disorders) in language development and outcomes:  Reveal range of variation that our theories must account for.  Allow us to examine the consequences of variation in the internal structure of the system.

  13. The Future of Cognitive Science, UC Merced, May 2008 Neuroscience Computer Science Psychology Philosophy Linguistics

  14. Neuroscience Movement Sci. Clinical Psych Robotics Biology Computer Science Psychology Mathematics Anthropology Philosophy Linguistics Education Psychiatry Speech Pathology

  15. Individual differences (including disorders) in language development and outcomes:  Reveal range of variation that our theories must account for.  Allow us to examine the consequences of variation in the internal structure of the system. But simultaneously  Detailed understanding of the process of language use and development may enable us to better understand disorders.

  16. A process-oriented approach to individual differences. beach • Define the process: • What steps does the brain/mind/language system/child take to get from some clearly defined input to some clearly defined output? • 2) How can we measure this process as it happens? • 3) Identify a population: • What will we relate variation in process to? • 4) What dimensions can vary within that process? • Which covary with outcome variables?

  17. Goal today: Show how understanding the real-time (and developmental) processes that underlie language in normal listeners can offer an important (complementary) viewpoint on individual differences. But first: I have to show you what those processes look like (and dispel a few misconceptions about speech perception). • Disclaimer • Much of this work is examines adults or older kids. • Easier to measure real-time process using more complex tasks • Easier to conceptualize process without having to worrying about development (as much). • Consequently, we take an individual differences approach, rather developmental (but ask me about development).

  18. Overview • Speech perception as a language process • Problems of Speech and word recognition • Fine-grained detail and word recognition. • Revisiting categorical perception • Using acoustic detail over time. • The beginnings of a comprehensive approach. • 2) Individual differences • A process view of individual differences. • Case study 1: SLI • Eye-movement methods for individual differences. • Case study 2: Cochlear Implants.

  19. Overview • Speech perception as a language process • Problems of Speech and word recognition • Fine-grained detail and word recognition. • Revisiting categorical perception • Using acoustic detail over time. • The beginnings of a comprehensive approach. • 2) Individual differences • A process view of individual differences. • Case study 1: SLI • Eye-movement methods for individual differences. • Case study 2: Cochlear Implants.

  20. The Domain: Speech & Words Speech perception, word recognition and their development are an ideal domain for these questions. • Excellent understanding of input • Acoustics of a single word. • Statistical properties of a language beach

  21. Aspiration/VOT Frequency F2 First Formant Time Voicing

  22. Aspiration/VOT Frequency Time Voicing

  23. 50 45 40 35 30 25 # of tokens 20 15 10 5 0 -5 10 25 40 55 70 85 100 115 130 145 VOT (ms)

  24. Major theoretical issue: lack of invariance. • Acoustic cues do not directly distinguish categories due to • Talker variation (Allen & Miller, 2003; Jongman, Wayland & Wang, 2000; Peterson & Barney, 1955). • Influence of neighboring phonemes (coarticulation) (Fowler & Smith, 1986; Delattre, Liberman & Cooper, 1955) • Speaking rate variation (Miller, Green & Reeves, 1986; Summerfield, 1981) • Dialect variation (Clopper, Pisoni & De Jong, 2006)

  25. 35 30 25 20 # of tokens 15 10 5 0 -5 10 25 40 55 70 85 100 115 130 145 VOT (ms) 50 45 40 35 30 25 # of tokens 20 15 10 5 0 -5 10 25 40 55 70 85 100 115 130 145 VOT (ms) Allen & Miller, 1999

  26. 400 500 600 F1 (Hz) 700 800 ɛ ʌ 900 1000 2200 2000 1800 1600 1400 1200 1000 F2 (Hz) Cole, Linebaugh, Munson & McMurray, 2010, J. Phon

  27. 400 500 600 F1 (Hz) 700 800 ɛ ʌ 900 1000 2200 2000 1800 1600 1400 1200 1000 F2 (Hz) Cole, Linebaugh, Munson & McMurray, 2010, J. Phon

  28. The Domain: Speech & Words Speech perception, word recognition and their development are an ideal domain for these questions. • Excellent understanding of input • Acoustics of a single word. • Statistical properties of a language • But difficult problem to solve • Tractable output units. beach

  29. The Domain: Speech & Words Activate words • But phonemes: • Have no meaning in isolation. • Theoretically controversial (Port, 2007; Pisoni, 1997) • Hard to measure directly… (e.g., Norris, McQueen & Cutler, 2000; Pisoni & Tash, 1974; Schouten, Gerrits & Van Hessen, 2003) • … particularly in populations with poor phoneme awareness, metalinguistic ability. • … particularly in a way that gives online (moment- • by-moment) measurement. Identify phonemes Extract acoustic cues

  30. The Domain: Speech & Words Activate words Identify phonemes Extract acoustic cues

  31. The Domain: Speech & Words Identify phonemes Extract acoustic cues Meaning (semantics) Sentence Processing (syntax) Reference (pragmatics) • Words • Functionally relevant: Crucial for semantics, sentence processing, reference. • Most everyone agrees on them (but see Elman, 2008, SRCLD) Activate words

  32. Online Word Recognition X basic bakery bakery X ba… kery bacon X X bait barricade X baby Major theoretical issue in word recognition: time • Information arrives sequentially • At early points in time, signal is temporarily ambiguous. • Later arriving information disambiguates the word.

  33. Online Word Recognition sack candle dog • If input is phonemic, word recognition is characterized by: • Immediacy • Activation Based • Parallel Processing • Competition Input: s... æ… n… d… ə… l time soup sandal

  34. Measuring Temporal Dynamics How do we measure unfolding activation?Eye-movements in the Visual World Paradigm Subjects hear spoken language and manipulate objects in a visual world. Visual world includes set of objects with interesting linguistic properties. asandal, asandwich, a candle and an unrelated items. Eye-movements to each object monitored throughout task. Tanenhaus, Spivey-Knowlton, Eberhart & Sedivy, 1995 Allopenna, Magnuson & Tanenhaus, 1998

  35. Task A moment to view the items

  36. Task

  37. Task Sandal

  38. Task Repeat 200-1000 times…

  39. Task Bear Repeat 200-1000 times… (new words, locations, etc)

  40. Why use eye-movements and visual world paradigm? • Relatively naturaltask. • Easy to use with clinical populations: • - Children with dyslexia (Desroches, Joanisse, & Robertson, 2006), - Autistic children (Brock, Norbury, Einav, & Nation, 2008; Campana, Silverman, Tanenhaus, Bennetto, & Packard, 2005) • - People with aphasia (Yee, Blumstein, & Sedivy, 2004, 2008). - Children with SLI (Nation, Marshall, & Altmann, 2003)

  41. Why use eye-movements and visual world paradigm? • Relatively naturaltask. • Easy to use with clinical populations: • Eye-movements generated veryfast(within 200ms of first bit of information). • Eye movementstime-lockedto speech. • Subjectsaren’t awareof eye-movements. • Fixation probability maps ontolexical activation.. • Measures a functional language ability.

  42. Eye movement analysis % fixations Time 200 ms Trials 1 2 3 4 5 Target = Sandal Cohort = Sandwich Rhyme = Candle Unrelated = Necklace

  43. 0.9 0.8 0.7 0.6 Target 0.5 Fixation Proportion Cohort 0.4 Rhyme 0.3 Unrelated 0.2 0.1 0 0 500 1000 1500 2000 Time (ms) s æ nd ə l Allopenna, Magnuson & Tanenhaus, 1998 McMurray, Samelson, Lee & Tomblin, 2010

  44. 0.9 0.8 0.7 0.6 0.5 Fixation Proportion 0.4 0.3 0.2 0.1 0 0 500 1000 1500 2000 Time (ms)

  45. Identify phonemes Extract acoustic cues Meaning (semantics) Sentence Processing (syntax) Reference (pragmatics) • Words • Functionally relevant: Crucial for semantics, sentence processing, reference. • Most everyone agrees on them (but see Elman, 2009, SRCLD) • Easy to measure directly… (e.g., Tanenhaus, Spivey-Knowlton, Sedivy & Eberhart, 1995; Allopenna, Magnuson & Tanenhaus, 1998; … even in populations with poor phoneme awareness, metalinguistic ability. • … online (moment-by-moment) data. Activate words • Measuring speech perception through the lens of spoken word recognition… • Ensures that whatever differences we find matter for the next level up. • Theoretically more grounded. • Multi-dimensional online measure

  46. The Domain: Speech & Words Speech perception, word recognition and their development are an ideal domain for these questions. • Excellent understanding of input • Acoustics of a single word. • Statistical properties of a language • But difficult problem to solve • Tractable output units. • Spoken word recognition • But problem of time beach

  47. The Domain: Speech & Words Speech perception, word recognition and their development are an ideal domain for these questions. • Excellent understanding of input • Acoustics of a single word. • Statistical properties of a language • But difficult problem to solve • Tractable output units. • Spoken word recognition • But problem of time • Associated with many impairments. beach

  48. Task • Auditory, Speech or Lexical Deficits have been reported in a variety of clinical populations • Specific / Non-specific Language Impairment • Dyslexia / Struggling Readers • Autism • Cerebellar Damage • Broca’s Aphasia • Downs Syndrome • Hard of Hearing • Cochlear Implant Users • Cognitive Decline • Schizophrenia

  49. So what’s the process? ? How do listeners map a highly variable acoustic input onto lexical candidates as the input unfolds over time?

More Related