Intonational prominence: Production in dialogue, comprehension during visual search

1. Intonational prominence:Production in dialogue, comprehension during visual search Shari R. Speer (with much input from Kiwako Ito, Co-I) Ohio State University Linguistics

2. Intonation & Pragmatic Status English Pitch Accent Distinctive tonal movement associated with stressed syllable.  perceivable prosodic prominence Local targets L and H (Pierrehumbert 1980) H*, (H+!H*), L*, L+H*, L*+H (a/c current ToBI guideline)

3. Pitch Accent Types & Pragmatics Pierrehumbert & Hirschburg, 1990 Compositional tune meaning PA typemeaning H* New items introduced to a discourse, added to mutual belief space. L* Already part of mutual beliefs. Salience-without-predication L*+H Uncertainty, lack of speaker commitment L+H* Contrast with some alternative related item

4. Pitch Accent Types & Pragmatics Pierrehumbert & Hirschberg, 1990 Compositional tune meaning PA typesmeaning H* New items introduced to a discourse, added to mutual belief space. L* Already part of mutual beliefs. Salience-without-predication L*+H Uncertainty, lack of speaker commitment L+H* Contrast with some alternative related item

5. Categorical distinction between H* and L+H* F0 minima preceding the accentual peak should be stable for L+H*, but not for H* Perception A clear categorical boundary should be observed in both identification and discrimination task along with H* height continuum. Tonal shape H*L+H* flat or subtle rise clear rise

6. Categorical distinction between H* and L+H* • F0 minima between two successive H* are consistently aligned to the syllable onset (Ladd & Schepman, 2003). • Listeners use the F0 minima alignment cue to distinguish ambiguous names, e.g. Norma Nelson vs. Norman Elson (Ladd & Schepman, 2003). • Although context identification tasks show categorical distinction between non-emphatic vs. emphatic contexts, listeners perform poorly in discrimination tasks along the accentual peak continuum. (Ladd & Morton, 1997)

7. Categorical distinction between H* and L+H* Testing tonal shape (Arvaniti & Garding, to appear) • The height of peak and its alignment vary according to the degree of emphasis. • Dialectal differences: Contrast between H* & L+H* preserves in Southern Californian but not in Minnesotan. H* <--> L+H* “gradient variation between two ends of a continuum” (Ladd & Schepman, 2003)

8. Analyzing Unscripted Speech • Task: Give instructions for tree decoration • Monitor display with a picture of ornament with its tag (e.g. blue ball) and a picture of a tree, indicating the location of the ornament. • Newness, Givenness & Contrastiveness manipulated by the order of mention.

9. * * * * * * * * * * * * * orange candy 16 17 * 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

10. * * * * * * * * * * * * * blue candy 17 * * 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

11. * * * * * * * * * * * * * green candy * * * 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

12. * * * * * * * * * * * * * orange candy 16 17 * 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

13. * * * * * * * * * * * * * blue candy 17 * * 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

14. * * * * * * * * * * * * * green candy * * * 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

15. Example pitch traces

16. Accentuation Proportion

17. Accentuation Proportion

18. L+H* distribution

19. Is intonation informative? Effect of Intonation on visual search Listeners use pitch accent to single out the object referred to (Dahan et al. 2002) 1. “Put the candle/candy below the triangle.” 2. “NOW, put the CANDLE above the square.” “NOW, put the candle ABOVE THE SQUARE.” fixation to candy candle  “CANDLE” candle  “candy” fixation to candle

20. Research Questions How does intonational prominence (L+H*) on a modifier adjective affect visual search? L+H*: Evokes a contrast set that contains alternative related item(s) (Pierrehumbert & Hirschberg, 1990) BLUE ball

21. Research Question How does intonational prominence (L+H*) on a modifier adjective affect visual search? L+H*: Evokes a contrast set that contains alternative related item(s) (Pierrehumbert & Hirschberg, 1990) green ball ball BLUE

22. Why track the eyes? • Eyemovements can be traced: • during natural conversation • for both speaker & listener • to investigate the online language processing (Tanenhaus and Trueswell,2005) • Accentual prominence evokes reference to an alternative discourse entity (Dahan et al., 2002). • Listeners fixate target objects 200ms from the onset of auditory stimuli during simple visual search (Allopenna, et al. 1998). As much as 150ms of this time is used to program and execute a ballistic saccade (Matin, Shao, & Boff, 1993).

23. Experiment 1:Procedure • Subjects followed recorded audio instruction to decorate Christmas trees. • (ASL e5000) at 60Hz.

24. Experiment 1: Visual Stimuli • Ornaments displayed on a grid with 11 cells (8 target + 3 filler)

25. Experiment 1: Sound Stimuli • Accentual patterns of stimuli were selected based on the analysis of spontaneous speech (Ito, Speer & Beckman, 2003).A trained female phonetician produced stimuli with the intended Pitch accent patterns. • Recorded 44.1KHz, 16bits • F0 extracted with 10ms window • All stimuli were ToBI transcribed by two native speakers of American English.

26. Experiment 1: Stimuli Adjective NounAccent Pattern New New H* !H* New Given H*!H* Given Given H*!H* Given New H* H* NewC Given L+H* no accent Given NewC H*L+H*

27. Experiment 1: Stimuli • Felicitous Patterns green onion ORANGE onion L+H* no accent brown ball brown ANGEL H* L+H* • Infelicitous Patterns gray stocking brown STOCKING H* L+H* orange candy ORANGE onion L+H* no accent

28. Experiment 1: Stimuli All stimuli were ToBI-transcribed by two native speakers of American English. Ave. 300 Hz Ave. 299 Hz

29. Experiment 1: Results

30. Experiment 1: Results

31. Experiment 1: Given/New • Objects named by given nouns were fixated sooner and more often than objects named by new nouns. • No additional advantage came from having both a given adjective and a given noun (compared to difference for given noun alone). • Visual search facilitated by previous mention of object nouns, and previous successful search for the referent in object-organized array.

32. Experiment 1: Results green onion  ORANGE onion [L+H* noacc]  orange ONION [H* L+H*]

35. Experiment 1: Results brown drum  brown ANGEL  BROWN angel

36. Experiment 1: Felicitous vs. Infelicitous • Felicitous L+H* on the color term in contrastive environments (blue egg, SILver egg) facilitated visual search compared to infelicitous L+H* on non-contrastive noun (grey candy, gold CANdy). • Felicitous L+H* on the object noun in contrastive environments (blue egg, blue CANdy) showed no advantage over infelictous L+H* on the adjective (blue egg, BLUE candy). • Listeners ‘tune’ their sensitivity to contrastive accent on the basis of the visual task?

37. Experiment 2: Questions • Does L+H* on the contrastive adjective lead to anticipatory eye-movement compared to H*? e.g. brown drum --> RED/red drum • Does infelicitous L+H* on the color adjective lead to an expectation strong enough to lead a garden-path in eye-movement? e.g. red onion --> GREEN drum

38. Experiment 2 Procedure Same as Exp 1 • Subjects followed audio instruction to decorate Christmas trees. • Ornaments displayed on a grid with 8 target + 3 filler cells • Eye-movements monitoring at 60Hz. Stimuli • Prepared in the same way as Exp1 • Same speaker and ToBI transcribers

39. Experiment 2: Results brown drum  RED drum [L+H* no acc]  red drum [H* !H*]

40. Experiment 2: Results red onion  green drum  GREEN drum

41. Experiment 2: Results red onion  GREEN drum [L+H* no acc] drum onion

42. The effect of infelicitous L+H*

43. Experiment 2: Results red onion  GREEN drum [L+H* no acc] drum onion

44. Experiment 2: Results red onion  green drum drum onion

45. Experiment 2: Summary • For cases with repetition of the immediately preceding noun, L+H* showed a processing advantage as compared to H* on the adjective (brown drum --> RED/red drum) • For cases without repetition of the noun, infelicitous L+H* led to incorrect anticipatory fixations to the most-recently mentioned ornament type. significant delay in fixations to the real target

46. Experiment 2: Summary • NO incorrect anticipatory fixations were observed when non-repeated targets were felicitously presented with [H* !H*]

47. Discussion • Dahan et al results showed that L+H* accent led to the expectation of contrast. Contrastive accent led listeners to fixate the object type that had *not* just been mentioned. • Results here show contrast effects from L+H* accent on the modifier that preceded the noun. L+H* led listeners to restrict the set of expected referents to those of the object type that *had* been most recently mentioned. This evidence suggests that L+H* evokes generation of a contrast set in the discourse representation of the listener. The set is based on the accented word interpreted within the structure of the utterance. Here, contrastive accent on the adjective restricted expectations about the coming head noun to the set of objects specified by the immediately preceding noun.

48. Conclusions so far • Naïve speakers predictably used L+H* accents to convey contrast when directing a visual search during spontaneous dialogue. • Eyemovement latencies and fixation probabilities showed listeners’ immediate use of pitch accent information during discourse comprehension. • Listeners were sensitive to the relevance of intonational cues for determining the intended referent during visual search. • L+H* accents on adjectives evoked an immediate interpretation of contrast, which led to anticipatory fixations to the most-recently mentioned object type. • These effects were not due to simple salience of the L+H* accented item, nor to word duration effects.

49. Additional studies • “Christmas Tree” production and Eyemovement studies with Japanese speakers • Further investigation of discourse structure and the distribution of pitch accent type in English • Additional studies examining other aspects of intonational effects on discourse structure, e.g. • Deaccentuation (reduction, deletion, pronouns) in production and comprehension

50. THANK YOU! Acknowledgements: Ping Bai Allison Blodgett Laurie Maynell Mary Beckman NSF BCS-018464 NIDCD R01-DC007090