From Sound to Sense and back again: The integration of lexical and speech processes

1. From Sound to Sense and back again: The integration of lexical and speech processes David Gow Massachusetts General Hospital Bob McMurray Dept. of Brain and Cognitive Sciences University of Rochester

2. Sense Sound The Speech Chain Complex computations from sound to sense must be broken up for study. Assume intermediate representations: Phonemes… Words… Syntactic Phrases…

3. The Standard Paradigm Sense The Standard Paradigm Words Phonology Phonemes Sound

4. Phonemes*essential * or other sublexical category The Standard Paradigm Sense The Standard Paradigm Delimited fields of study. • Speech Perception Words • Spoken Word Recognition Phonology Phonemes • Phonology Sound

5. 100 100 Discrimination % /p/ • Sharp identification of tokens on a continuum. Discrimination ID (%/pa/) 0 0 B VOT P • Discrimination poor within a phonetic category. Why? Categorical Perception (CP) Continuous Acoustic Detail => Discrete Categories Does CAD affect speech categorization?

6. Sense Categorical Perception (CP) • Defined fundamental computational problems. • CP is output of • Speech perception • Input to • Phonology • Word recognition. Words Phonology Phonemes Sound

7. CP • But… • Not all speech contrasts are categorical. • Lots of tasks show non-categorical perception. Fry, Abramson, Eimas & Liberman (1962) Pisoni & Tash (1974) Pisoni & Lazarus (1974) Carney, Widden & Viemeister (1977) Hary & Massaro (1982) Pisoni, Aslin, Perey & Hennessy (1982) Healy & Repp (1982) Massaro & Cohen (1983) Miller (1997) Samuel (1997)…

8. Sense ? CP Words CP tasks don’t necessarily tap a stage of this problem. Sound Lexical activation… seems a good bet. Why has the Standard Paradigm persisted? Categorical Perception is about phonetic classification. The minimal computational problem: compute meaning from sound.

9. Why has the Standard Paradigm persisted? Even when continuous acoustic detail affects word recognition, it is seen as outside of core word recognition.

10. Segmentation Cue extra-segmental process. Why has the Standard Paradigm persisted? Even when continuous acoustic detail affects word recognition, it is seen as outside of core word recognition. • Example: Word Segmentation • Vowel Length • Stress/Meter • Coarticulation Words Phonemes Word Recognition CAD

11. Does continuous acoustic detail affect interpretation via core word-recognition processes? • No. Standard Paradigm is fine… • Yes. Hmm… Sublexical Filter (phonemes) • Need to use stimuli with: • Precise control over CAD • Need to use tasks that: • reflect only minimal computational problem: meaning. • are sensitive to acoustic detail.

12. Visual World Paradigm Visual World Paradigm • Subjects hear spoken language and manipulate objects in a visual world. • Visual world includes set of objects with interesting linguistic properties (names) • Eye-movements to each object are monitored throughout the task. Tanenhaus, Spivey-Knowlton, Eberhart & Sedivy (1995) Allopenna, Magnuson & Tanenhaus (1998)

13. Meaning based, natural task: Subjects must interpret speech to perform task. • Fixation probability maps onto dynamics of lexical activation. • Context is controlled: • meaning  lexical activation. • Eye-movements fast and time-locked to speech.

14. ? Does continuous acoustic detail affect interpretation? Is lexical activation sensitive to continuous acoustic detail?

15. McMurray, Tanenhaus & Aslin (2003) • Combine tools of • speech perception: • 9-step VOT continuum. • spoken word recognition: • visual world paradigm

16. Methods A moment to view the items

17. 500 ms later

18. Bear Repeat 1080 times…

19. 200 ms Trials 1 2 3 4 5 Time Target =Bear Competitor =Pear Unrelated =Lamp, Ship

20. 0.9 0.8 0.7 0.6 0.5 0.4 Fixation proportion 0.3 0.2 0.1 0 0 400 800 1200 1600 VOT=0 Response= Time (ms)

21. target Fixation proportion Fixation proportion time time Predictions What would lexical sensitivity to CAD look like? Systematic effect on competitor dynamics. Fixations to the competitor. Categorical Results Gradient Effect target target competitor competitor competitor competitor

22. 20 ms 25 ms 30 ms 10 ms 15 ms 35 ms 40 ms 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 0 400 800 1200 1600 0 400 800 1200 1600 2000 Results Response= Response= VOT VOT 0 ms 5 ms Competitor Fixations Time since word onset (ms)

23. P B Sh L Task? Phoneme ID Not part of minimal computational problem. Same stimuli in metalinguistic task… …more categorical pattern of fixations Continuous acoustic detail is not helpful in metalinguistic tasks…

24. Summary Word recognition shows gradient sensitivity to continuous acoustic detail. Not extra-segmental: VOT CAD affects higher-level processes. • Consistent with other studies: • Andruski, Blumstein & Burton (1994) • Marslen-Wilson & Warren (1994) • Utman, Blumstein & Burton (2000) • Dahan, Magnuson, Tanenhaus & Hogan (2001) • McMurray, Clayards, Aslin & Tanenhaus (2004) • McMurray, Aslin, Tanenhaus, Spivey & Subik (in prep)

25. CAD affects higher-level processes. From other work: Lexical activation influences sublexical representations. The Standard Paradigm? Sense Words Phonology Phonemes Samuel & Pitt (2003) Magnuson, McMurray, Tanehaus & Aslin (2003) Samuel (1997) Elman & McClelland (1988) Continuous Acoustic Detail

26. CAD affects higher-level processes. The Standard Paradigm? Sense From other work: Words Lexical activation influences sublexical representations. Phonology Phonemes Phonological regularity affects signal interpretation. Continuous Acoustic Detail Massaro & Cohen (1983) Halle, Segui, Frauenfelder & Meunier (1998) Pitt (1998) Dupoux,Kakehi, Hirose, Pallier & Mehler, (1999)

27. Sense Perhaps interaction and integration make sense. Do they help solve sticky problems? ? Words Phonology Phonemes YES Continuous Acoustic Detail

28. The Emerging Paradigm • Integration of work in: • spoken word recognition • speech perception • phonology • New computations simplify old problems and solve new ones. • Cognitive processes: Lexical activation & competition. • Perceptual processes: sensitivity to CAD & perceptual grouping.

29. CAD is helpful in language comprehension. • Word segmentation • Coping with lawful variability due to assimilation • Combination of approaches helps solve both problems.

30. Lexical Segmentation Some lexical processes can’t work in the Standard Paradigm

31. The SWR Solution [  k t I v d I p A  t m I n t]

32. [  k t I v d I p A  t m I n t] active

33. [  k t I vd I p A  t m I n t] activedepartment

34. [  k t I v d I p A  t m I n t] activedepartment actof dip artmint apart departin are par Standard Paradigm: Template matching overgenerates

35. succeed suck activation seed ‘ k s I d - Cycle Frauenfelder & Peeters (1990) • Overgeneration resolved through competition in • TRACE (McClelland & Elman 1986) Problem: What if the speaker is trying to say “suck seeds”?

36. Words Implied processing model requires separate segmentation process Segmentation Phonemes Recognition CAD The Speech Solution • Cues shown to affect segmentation: • Initial strong syllable • Initial lengthening • Increased aspiration • Increased glottalization Lehiste, 1960; Garding,1967; Lehiste, 1972; Umeda, 1975; Nakatani & Dukes, 1977; Nakatani & Schaffer,1978; Cutler & Norris, 1988…..

37. Words Segmentation Phonemes Recognition CAD Problem: cues are subtle and varied, extra-segmental processes are inelegant ? Is there a better mechanism?

38. Syntax Syntax GRAMMAR primed GRAMMAR primed  Tax INCOME inhibited Tax INCOME primed Gow & Gordon (1995) The proposal had a strange syntax that nobody liked. ^ The proposal had a strange sin tax that nobody liked. ^ • CAD affects interpretation. • does not trigger segmentation.

39. Good Start Model • Observation: All segmentation cues happen to enhance • word-initial features • Strengthened cues facilitate activation, making • intended words stronger competitors • Incorporating CAD: • Solves overgeneration problem. • No extra-segmental segmentation process. Gow & Gordon (1995)

40. Summary When continuous acoustic detail affects lexical activation, speech and SWR models can be integrated and simplified

41. Assimilation The emerging paradigm reframes computational problems

42. ripe berries? [  a I p ]# berries right berries? [ G  I m]# berries nonword? Redefining Computational Problems • English coronal place assimilation • /coronal # labial/ [labial # labial] • /coronal #velar/ [velar # velar] • Standard Paradigm: Change is • discrete • phonemically neutralizing

43.  ripe Standard Paradigm solution: Phonological inference (Gaskell & Marslen-Wilson, 1996; 1998; 2001) Knowledge driven inference: If [labial # labial] infer /coronal # labial/ • greem beans  green (Gaskell & Marslen-Wilson, 1996; Gow, 2001) ripe berries  right (Gaskell & Marslen-Wilson, 2001; Gow, 2002) Moreover: Assimilation effects dissociated from linguistic knowledge (Gow & Im, in press)

44. F3 Transitions in /æC/ Contexts 2800 2750 coronal 2700 assimilated Frequency (Hz) 2650 labial 2600 2550 Pitch Period Assimilation Produces CAD Assimilatory modification is acoustically continuous F2 Transitions in /æC/ Contexts 1850 1800 1750 coronal Frequency (Hz) 1700 assimilated labial 1650 1600 1550 Pitch Period This is not discrete feature change!

45. Regressive Context Effects Sma Select the catp box

46. Subject Hears: Assim_Non-Coronal (cat/p box) 0.6 0.5 0.4 Fixation Proportion 0.3 0.2 Coronal (cat) 0.1 Non-Coronal (cap) 0 0 400 800 1200 1600 Time (ms)

47. Subject Hears: Assim Non-Coronal (cat/p drawing) 0.6 0.5 0.4 Fixation Proportion 0.3 0.2 Coronal (cat) Non-Coronal (cap) 0.1 0 0 400 800 1200 1600 Time (ms)

48. Progressive Context Effects Progressive effect in the same experiment

49. Assimilation: Use of CAD Assimilation is resolved through phonological context. Partially-assimilated items show regressive context effects (Gow, 2002; 2003) progressive context effects (Gow, 2001; 2003) Fully assimilated items show neither* (Gaskell & Marslen-Wilson, 2001; Gow, 2002;2003)

50. assimilation # context Infinite regress (eternal ambiguity)…. or something more interesting?