Comparison of Nonverbal Markers in Hungarian Dialogues

1. Comparison of the nonverbal ways of marking the discourse relation of concession and the functions of lexical search and approximation in Hungarian dialogues – Relations and correspondances between discourse-pragmatic functions (concession versus lexical search) and nonverbal features of the Hungarian DM mondjuk (’let’s say’; ’although’) Ágnes Abuczki Hungarian Academy of Sciences February 2015

2. Goals, outline Empirical study on a multifunctional (polysemous) Hungarian lexical item (mondjuk, meaning „let’s say” or „although, it must added that”) that operates at discourse level and marks relationshop between discourse unit as well as the attitude of the speaker --> discourse marker (henceforth: DM) 2 salient functions of the item will be described by multimodal features in order to enhance its meaning disambiguation Questions: Is there a significant relation/correspondancebetween the discourse-pragmatic function of a DM and Manual gesticulation Facial expression Gaze direction of the speaker Duration of the DM Pause preceding the DM?

3. Material • Multimodal HuComTech corpus (Hungarian only) • 22 eaf files (involving audio, video, multimodal pragmatic and automatic prosodic annotation): • 22 informal conversations • 22 interviewees Number of tokens of DMs segmented: • Mondjuk (~let’s say): 208

4. Most salient functions of the selected DM • 2 most salient functions of mondjuk('let’s say'): • LXS APPR= lexical search + approximation (46 tokens); • CON= concession (41 tokens).

5. Most salient functions of mondjuk('let’s say') - markers of lexical search + approximation (abbreviated as LXS; can be glossed as about, like): „gyorsan megy a motorom mondjuk 120–140-nel” (‘my bike is really fast, it can do DMmondjuk120–140 kmphs’) (HuComTech, 017_I) - markers of concession (abbreviated as CON; can be glossed as although, but): „szeretek a belvárosban élni mondjuk elég nagy a szmog” (‘I like living in the city centre DMmondjuk the air is polluted’) (HuComTech, 019_I)

6. Methods • Segmentation of the selected word • Tagging functions • Low-level prosodic features and temporal features (durations and preceding pauses) were extracted from the segmented sound files (.wav) using Praat and Prosogram scripts. • The nonverbal-visual features (gaze direction, facial expression, hand gestures) of the speaker’s behaviour were extracted from the manually-performed video annotations of the recordings and can be automatically queried using the ELAN software. • The queries on the relation of each function and each nonverbal feature were run separately and were ultimately joined in contingency tables for statistical analysis.

7. Methods: Segmentation and annotation in ELAN

8. The relation of function and manual gesticulation • Prior to queries, I expected to find correspondences between discourse functions and hand movements. • I considered manual gesticulation: • any handshape type annotated other than the default handshape of the actual speaker (most common default type: half-open-flat hands) • any handshape change during uttering a DM • I queried the relation of hand gesticulation and each of the salient functions of DM one by one in separate queries (with the ‘Find overlapping labels’ command), and then joined them in contingency tables for statistical analysis in SPSS 19. 0.

9. The relation of pragmatic functions and hand gestures (mondjuk – let’s say) initial observation and hypothesis: lexical search and gesticulation contrary to expectations  Significant (2(1)=12,442, p<0,01)

10. The relation of pragmatic functions and gaze direction (mondjuk – let’s say) typical marker of concession: averted gaze direction type during; not significant (p>0,05) typical marker of lexical search and approximation: upwards gaze direction type; not significant (p>0,05)

11. The relation of pragmatic functions and facial expressions (mondjuk – let’s say) Recalling affect display during mondjuk_LXS_APPR

12. Comparison of the durations of a DM expressing different functions • Why was it analyzed? I expected that different functions are realized in different durations. • Method: Queries were run by a Praat script in order to measure the duration of the individual DM tokens performing the two most salient functions, and save them in a spreadsheet file. • Representation of results: box-and-whiskers plots - it shows the median and variation of duration.

13. Duration and pragmatic function • My hypothesis about the duration of the various functions of this DM: • Mondjuk (let’s say) expressing lexical search and approximation is expected to be realized longer than mondjuk expressing concession

14. Distribution of the duration of DMS with different functions iindependent samples t-test on mondjuk (say): significant iindependent samples t-test on ugye (is that so?): not significant

15. Silence annotation in Praat • Silence annotation was performed following the segmentation of DMs with the aim to test the hypothesis if DMs are predominantly separated by pauses (as they are often described in the literature). The phonetic parameters set for automatic silence annotation were as follows: • minimum pitch: 100 Hz (subtract mean) • time step: automatic (0,01 s) • silence threshold: - 45 dB • minimum silent interval duration: 0,15 s • minimum sounding interval duration: 0,05 s • As a result, annotation segmented the recordings into sounding and silent segments:

16. Silence annotation in Praat The difference between the two categories has not been found significant by Pearson’s Chi-Square test (p>0,05).

17. Multiple layer searches in ELAN

18. Conclusions: prototypical sets of features of the canonical uses of mondjuk (say) performing its two different functions

19. References • L. Hunyadi, Szekrényes I., Borbély A., Kiss H., Annotation of spoken syntax in relation to prosody and multimodal pragmatics. In: Proceedings of 3rd Cognitive Infocommunications Conference. Kosice: IEEE Conference Publications, 2012, 537–541. • B. Fraser, “Topic orientation markers,” Journal of Pragmatics, 41, 2009, pp. 892–898. • W. Chafe, “Consciousness and language,” Cognition and Pragmatics (Handbook of Pragmatics Highlights), D. Sandra, J. Östman, J. Verschueren, Eds. Amsterdam/Philadelphia: John Benjamins, 2009, pp. 135–145. • Boersma P., Weenink, D, 2007. Praat: doing phonetics by computer 5.0.02. University of Amsterdam: Institute of Phonetic Sciences. http://www.praat.org • http://www.physics.csbsju.edu/stats/exact_NROW_NCOLUMN_form.html This research was supported by the European Union and the State of Hungary, co-financed by the European Social Fund in the framework of TÁMOP 4.2.4. A/2-11-1-2012-0001 ‘National Excellence Program’.

20. Thank you for your attention.