Discourse-driven awareness of contrast and the effect of pitch accent on referential resolution

1. Discourse-driven awareness of contrast and the effect of pitch accent on referential resolution Kiwako Ito Ohio State University

2. Referential updates during running discourse Speakers/Listeners need to: • Correctly represent the currently discussed discourse entities. • While maintaining the recently discussed discourse representations, inhibit past entities upon processing incoming referential information. • The relations among discourse entities need to be updated.

3. Pitch prominence and referential resolution • Prominent pitch accent evokes non-anaphoric interpretation of accented entity, whereas the lack of accent leads to anaphoric interpretation “Put the candle below the triangle.”  “Now, put the CAN…(fixations to candy) “Now, put the can…(fixations to candle) (Dahan et al. 2002; Arnold, 2008)

4. Pitch prominence and referential resolution • Pitch prominence on the prenominal adjective leads to anticipatory fixations to contrastive referent. “Hang a blue ball.”  “Hang a GREEN drum” (Initial fixations to ball: accent-driven garden-path) (Ito & Speer, 2008; to appear; Ito et al. ms) • Similar effect of pitch prominence has been reported for German (Weber et al., 2006) and Japanese (Ito et al., under review).

5. Awareness of visual contrast Visual information guides the comprehension of speech input. Kamide et al. (2003) Spivey et al. (2002)

6. Eberhard et al. (1995) • Visual context is sufficient to lead contrastive interpretation of prominent accent. “Touch the LARGE/large blue square.”

7. Discourse-driven vs. visually-available contrast How is an accentual prominence processed when visual context and discourse context license two different contrast sets? Do listeners interpret accentual prominence according to visual context or discourse context? How does the awareness of visual contrast affect the use of pitch prominence for referential resolution?

8. Dual contrast displays

9. Experiments • Experiment 1: Passive naming of a visual object standing in contrast with another referent within a display. • Experiment 2: naming and search Naming of a visual object is followed by a search and click task.

10. Expt1: fixation during simple naming • Participants: 36 OSU undergraduates • 48 trials (16 target): 8 animals (cat, lion, rabbit, turtle, monkey, squirrel, frog, fish) • 2 sizes: Large / Small • 4 colors: Green / Pink / Orange / Purple • Task: name the animal in the red square “What is this?” • Eye movements were monitored using Tobii1750 (50Hz).

11. Example display: Experiment 1

12. Contrast counterpart Example display: Experiment 1 Minimal pair

13. Fixation likelihood during naming(n=31)

14. Naming patterns (n=31*48=1488)

15. Epxt 1:Group comparison Bear Noun Users (22) Modifier Users (9)

16. Expt2: detection after naming • Participants: 40 OSU undergraduates • Display: identical to Expt 1 • Task: name  search and click “What is this?”  “Now, where is the… • Prosodic manipulation: L+H* vs. H* Emphatic LARGEgreen monkey Non Emphatic largegreen monkey

17. Expt 2: naming  search

18. Now, where is the …

19. ToBI analysis Emphatic: L+H* on Size Non-Emphatic: !H* on Size SizeL+H*Colorno-accNounno-acc Size!H*ColorH*Noun!H* Mean Point F0: 450 Hz Mean Point F0: 252Hz

20. Auditory stimuli: where is the ***p< .001, **p<.01, *p<.05, ^p<.1

21. Auditory stimuli: critical NP ***p< .001, **p<.01, *p<.05, ^p<.1

22. Expt 2: fixation patterns during naming(n=38)

23. Expt 1 vs. Expt 2 Experiment 1: N=31 Experiment 2: N=38

24. Naming patterns (n=38*48=1824)

25. Expt 2: Group comparison Bear Noun Users (27) Modifier Users (11)

26. Predicted garden-path effect of L+H*on large

27. Predicted garden-path effect of L+H*on large

28. Expt2: Discourse-driven garden-path effect (n=38)

29. Expt2: Discourse-driven garden-path effect (n-=38)

30. Awareness of contrast during“Now, where is the”

31. Aware vs. Unaware group“Now, where is the”

32. Two possible scenarios… • Weaker garden-path effect in ‘aware’ listeners. • L+H* on size may be interpreted to distinguish the minimal pair (e.g., large vs. small green monkeys.) • Fixation to the target should be facilitated. • Equal or stronger garden-path effect in ‘aware’ listeners. • L+H* on size may be interpreted according to local discourse context. • Awareness to visual contrast may reflect more attuned executive control  Listeners may show higher sensitivity to accentual information.

33. Unaware participants: minimal pair < distractor

34. Unaware participants: minimal pair < distractor

35. Aware participants:minimal pair > distractor

36. Aware participants:minimal pair > distractor

37. Group comparison Unaware (32) Aware (6)

38. Summary • Non-interactive passive naming did not lead to immediate gist of visual contrast. Passive naming task did not elicit informativeexpressions.  modification is avoided unless it is necessitated by communication • The presence of discourse led to an increase in the fixations to the immediate contrast counterpart.  linguistic context affects visual processing

39. Summary • As the majority remained unaware of the minimal pair, accentual prominence (L+H*) was interpreted according to the local discourse context. • Even those who paid attention to the visual context interpreted L+H* according to the discourse context. Possibility: Listeners’ awareness of visual contrast may reflect larger cognitive capacity, which may allow more efficient use of pitch accent for online referential resolution.

40. Remaining work • Individual differences or trial-based differences? • Experiment 3: • Interpretation of L+H* in the absence of discourse context • Relationship between the exposure time and the browsing pattern • Interactive referential expression

41. Acknowledgement • Ping Bai • Elizabeth McCullough • Ross Metusalem • Shari Speer • Speerlab members • NIH R01 DC007090-01A2

42. Distracting memory taskWhat is this? • 16 trials • basket ball • pins • dollar bill • vending machine • Etc… • Inserted before each critical trial.