Tracking the timecourse of multiple context effects in assimilated speech

1. Tracking the timecourse of multiple context effects in assimilated speech Bob McMurray Dept. of Brain and Cognitive Sciences University of Rochester David Gow Massachusetts General Hospital With thanks to Dana Subik, Joe Toscano & John Costalis

2. Laboratory Phonology Spoken Word Recognition Overview 1) Bridging fields yields: New solutions to old problems. New questions. 2) Coping with Coronal-Place Assimilation during online recognition. 3) Implications for language processing & phonology.

3. Bridging Fields: Laboratory Phonology Laboratory Phonology: How perceptual and articulatory constraints drive sound change and shape phonological systems Rich information source in the signal: Constraints inferred through acoustic and articulatory measures. Do phonological constraints inform word recognition? Can details of word recognition inform phonological constraints?

4. Phoneme perception models that relate acoustic properties to categorical perception Spoken word recognition models that assume phonemic inputs as input to the lexicon and meaning. Ignore systematic acoustic variation. Bridging Fields: Spoken Word Recognition Perceptual models tend to come in two varieties:

5. Limits of categorical perception Categorical perception (CP) is task-dependent, and does not appear to take place in tasks that involve spontaneous, naturalistic speech understanding. McMurray, Aslin, Tanenhaus, Spivey & Subik (in prep) Within category variation that should be lost in CP affects lexical processes Andruski, Blumstein & Burton (1994), Gow & Gordon, 1995; Utman, Blumstein & Burton (2000), Dahan, Magnuson, Tanenhaus & Hogan (2001), Gow (2001; 2002; 2003) McMurray, Tanenhaus & Aslin (2003)

6. Perception Phonology Properties of the signal must be related to meaning—lexical activation. Perception Phonology Meaning Systematic acoustic variation and SWR Speech perception and phonology relate signal properties to perception.

7. Case study: English Place Assimilation

8. cap box? [ k  p ]# box cat box? [ g  I m]# berries nonword? Assimilation • English coronal place assimilation • /coronal # labial/ [labial # labial] • /coronal #velar/ [velar # velar] • Prior work has treated this change as • discrete • phonemically neutralizing

9.  cap How are words recognized despite neutralization? Phonological inference (Gaskell & Marslen-Wilson, 1996; 1998; 2001) If [labial # labial] infer /coronal # labial/ • greem beans  green (Gaskell & Marslen-Wilson, 1996; Gow, 2001) cap box  cat (Gaskell & Marslen-Wilson, 2001; Gow, 2002)

10. 1850 2800 1800 2750 1750 coronal 2700 assimilated Frequency (Hz) Frequency (Hz) 1700 labial 2650 1650 2600 1600 1550 2550 Assimilation as Continuous Detail Assimilatory modification is acoustically continuous. F2 Transitions in /æC/ F3 Transitions in /æC/ Contexts Contexts Pitch Period Pitch Period Assimilation blends cues to two places of articulation

11. An Alternative View Coronality of assimilated item Labiality of assimilating item Assimilation redistributes and blends information cat box [kQtp # bAks]

12. Blend might facilitate recognition of context Assimilating context might disambiguate blend In theory: assimilation creates correlated cues… [ kQtp # bAks ] How can we determine if listeners use this information during recognition?

13. These questions require a method that: • Measures lexical activation. • Sensitive to continuous acoustic detail. • Sensitive to temporal uptake of information. • Measures consideration of multiple items in parallel.

14. Visual World Paradigm Visual World Paradigm • Subjects hear spoken language and manipulate objects in a visual world. • Visual world includes set of objects whose names represent competing hypotheses for the input. • Eye-movements to each object are monitored throughout the task. Tanenhaus, Spivey-Knowlton, Eberhart & Sedivy (1995) Allopenna, Magnuson & Tanenhaus (1998)

15. Fixation probability ~ lexical activation. • Sensitive to within-category acoustic variability (McMurray, Tanenhaus & Aslin, 2003; Dahan, Magnuson, Tanenhaus & Hogan, 2001) • Eye-movements fast and time-locked to speech—temporal dynamics. • Multiple competitors in same trial. • Meaning based, natural task: Subjects must interpret speech to perform task.

16. Experiment 1 Assimilation facilitates recognition. Present subjects with assimilated or non assimilated speech. Measure activation for items that follow assimilation.

17. Prediction: More fixations to goose after assimilated consonants. Methods Subject hears “select the maroongoose” “select the maroonggoose”

18. Spliced from “maroon duck” “select the maroonduck” “select the maroongoose” “select the maroongduck” *** “select the maroonggoose” Spliced from “maroon goose” 34 Subjects. 16 sets of items. Subjects exposed to pictures/names before each block. Stimuli cross-spliced from natural tokens—assimilation is not complete… continuous acoustic information. Eye-movements temporally aligned to onset of second word (goose or duck).

19. 200 ms Trials 1 2 3 4 5 … many more trials Time Target =Maroon Goose Competitor =Maroon Duck Unrelated =Patriotic Duck and Goose

20. 0.9 0.8 Assimilated 0.7 Non Assimilated 0.6 0.6 0.5 Fixation Proportion Assimilated 0.5 0.4 Non Assimilated 0.3 0.4 0.2 0.3 0.1 Fixation Proportion 0 0.2 0 200 400 600 800 1000 Time (ms) 0.1 0 0 100 200 300 400 500 Time (ms) Results Looks to the target (goose)

21. Looks to the competitor (duck) 0.25 0.2 0.15 Fixation Proportion Assimilated Non Assimilated 0.1 0.05 p = .03* 0 0 200 400 600 800 1000 Time (ms)

22. Experiment 1: Summary • Continuous variation due to assimilation • not variability to be conquered… • signal to be used. • Assimilated coronals allow progressive operations. • facilitate consistent targets • exclude inconsistent competitors earlier Consistent with prior work using priming (Gow, 2001; 2003; Gow & Im, in press)

23. ? cat cap Does subsequent context regressively modify the interpretation of assimilated segments? catpbox catp drawing Lexical Ambiguity? Even incomplete modification can create lexical ambiguity. cat box  catp box

24. Prediction: Fixations to cat or cap is a function of context. Experiment 2 Subject hears “select the catpbox” “select the catpdrawing”

25. 0.6 0.5 0.4 Fixation Proportion 0.3 0.2 Coronal (cat) 0.1 Non-Coronal (cap) 0 0 400 800 1200 1600 Time (ms) catp box Assim Non-Coronal

26. 0.6 0.5 0.4 Fixation Proportion 0.3 0.2 Coronal (cat) Non-Coronal (cap) 0.1 0 0 400 800 1200 1600 Time (ms) catp drawing Assim Coronal Regressive effect is more biasing for non felicitous assimilation… perceptual locus?

27. Progressive effects too? ? Regressive effects: Context biases interpretation of ambiguous token. Will we see a progressive effect at the same time?

28. 0.9 0.8 0.7 0.6 0.5 Non-assimilated Fixation Proportion 0.4 Assimilated Non-assimilated 0.3 Assimilated 0.2 0.1 0 0 200 400 600 800 1000 Time (ms) Weaker effects Possibly due to item variability Target Competitor 0.4 0.35 0.3 0.25 0.2 Fixation Proportion 0.15 0.1 0.05 0 0 200 400 600 800 1000 Time (ms)

29. Experiment 2: Summary • Assimilated coronals allow regressive operations: • Context useful in resolving ambiguity. Similar Progressive operations to experiment 1. What kind of computation is responsible? … relationship to continuous detail in signal important

30. Continuous Signal  Continuous Response Progressive & Regressive effects vary continuously across items. Experiment 1: Progressive effect on target

31. Experiment 2: Regressive effect What can the acoustics properties of these items tell us about perceptual variability?

32. Measured F1, F2, F3, Closure Duration of original items. Regression: F1, F2, F3, Closure Interaction with Labiality.

33. Not enough power (items) to reach significance but model accounted for: • Experiment 1 • Progressive effect: 75% of variance • Experiment 2 • Progressive effect: 57% of variance. • Regressive effect: 37% of variance Continuous acoustic variation is related to perceptual processes… how?

34. A Perceptual Account Feature cue parsing (Gow, 2003) [ k  t p b l E d ]

35. Feature cue parsing (Gow, 2003) Any feature is encoded by multiple cues that are integrated

36. Feature cue parsing (Gow, 2003)

37. Feature cue parsing (Gow, 2003) Assimilation creates cues consistent with multiple places

38. Feature cue parsing (Gow, 2003) Extract feature cues

39. Feature cue parsing (Gow, 2003) Group feature cues by similarity and resolve ambiguity

40. Integration is by the same process within a segment.

41. Standard component of information integration in perception.

42. Feature cue parsing (Gow, 2003) example: cat…. catp# box catp# drawing catp#  | | | | [cor] [cor] [COR] [cor] [lab] [LAB] [lab] [lab]

43. Feature cue parsing (Gow, 2003) catp# Box catp# Drawing catp#  | | [cor] [cor] [COR] [cor] [lab] [LAB] [lab] [lab] example: cat…. Progressive and regressive effects fall out of grouping

44. Implications for Phonology Feature cue parsing based on basic perceptual grouping principles: • Not specific to assimilation. • Parsing errors may lead to sound change: • Pressure on languages to avoid errors • Maximize contrast between adjacent segments. • Minimize juxtaposition of similar segments

45. Pre-Shona -b w a [LAB] [labio-velar glide] Shona -b a [LAB] (labio) [velar glide] Feature parsing errors may lead to sound change e.g. Shona Dissimilation (Ohala, 1981) Gow & Zoll (2002)

46. Conclusions English coronal place assimilation neutralizes phonemic distinctiveness. Perceptual recovery cannot be based on symbolic processes. • Continuous perceptual mechanisms sensitive to systematic acoustic variation yield • Progressive activation of upcoming material • Regressive ambiguity resolution. Such mechanisms may play a role in sound change.

47. Perceptual Processes Sound Change Perceptual Processes … Conclusions Bridging spoken word recognition and laboratory phonology helps both fields. • In phonology: perceptual mechanisms for handling variation may constrain languages’ sound structures. • In SWR: assimilation is not noise to be conquered, but signal to be used. Systematic Phonological Variation