1 / 49

Incremental references in dialogue: data from corpora, questions for psychologists

Incremental references in dialogue: data from corpora, questions for psychologists Massimo Poesio (Uni Essex) Leverhulme Network opening workshop Edinburgh, 14 th August 2005 Network subprojects involving Essex

andrew
Download Presentation

Incremental references in dialogue: data from corpora, questions for psychologists

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Incremental references in dialogue: data from corpora, questions for psychologists Massimo Poesio (Uni Essex)Leverhulme Network opening workshop Edinburgh, 14th August 2005

  2. Network subprojects involving Essex • Incremental interpretation in dialogue, focusing especially on interpretation of anaphoric expressions (Question (1)) • Alignment and related phenomena: misunderstandings, completions (Question (2)) • Visual attention and reference (Question (3))

  3. The original TRAINS data

  4. The TRAINS data, IIncremental interpretation of references M: so we should move the engine at Avon,engine E, to …S: engine E1M: E1S: okayM: engine E1, to Bath, to / or, we could actually move it to Dansville to pick up the boxcar there

  5. The TRAINS data, IIShifting the focus of (visual) attention M: while this is happening, take engine E1 to Dansville, pick up the boxcar, and come back to AvonS: okayM: okay then load the boxcar with bananas

  6. The original TRAINS data

  7. More recently:the Bielefeld Toy Plane Corpus

  8. Completions and incremental reference in the Bielefeld ToyPlane corpus Inst: So, jetzt nimmst DuWell, now you grasp Cnst: eine Schraubea screw. Inst: eine <-> orangene mit einem Schlitz.an <-> orange one with a slit Cnst: Ja. Inst: Und steckst sie dadurch, also And you put it through, let’s see ….Cnst: von oben

  9. The theoretical modelresulting from TRAINS : PTT • Key characteristics: • Building on (Compositional) DRT (Muskens, 1996) • Common ground as a record of the discourse situation (Barwise and Perry, 1983) • An account of incremental semantic interpretation • (An account of GROUNDING and alignment: last year’s Bielefeld / CATALOG presentations) • Other people involved in the development: • David Traum, Hannes Rieser, Colin Matheson, Reinhard Muskens

  10. From DRT to PTT • a. A: There is an engine at Avon. • B: It is hooked to a boxcar • DRT: [ x,w,y,u,s,s’| engine(x), Avon(w), s: at(x,w), boxcar(y), s’:hooked-to(u,y), u=x]

  11. Common ground and discourse situation in PTT [ce1,ce2,K1,K2| K1=[x,w,s| engine(x), Avon(w), s: at(x,w)], ce1: assert(A,B,K1) K2=[y,z,s’| boxcar(y), s’:hooked-to(z,y), z=x], ce2: assert(B,A,K2)]

  12. Utterances in the discourse situation [u1,u2, ce1,ce2,K1,K2| u1: utter(A,”there is an engine at Avon”),K1=[x,w,s| engine(x), Avon(w), s: at(x,w)],ce1: assert(A,B,K1),generate(u1,ce1),u2: utter(B,”it is hooked to a boxcar”), K2=[y,z,s’| boxcar(y), s’:hooked-to(z,y), z=x], ce2: assert(B,A,K2),generate(u2,ce2) ]

  13. Incremental interpretation in PTT • (Poesio, 1991, 1992, 1994, 1995, 1999): each perceptual event leads to an update of the discourse situation • Sub-sentential (and sub-word) utterances • Perception of non-verbal events (e.g., gestures), attention shift • Following each update, the discourse situation is further updated via non-monotonic inference • (formulated in terms of Prioritized Default Logic, Brewka 1994)

  14. Micro conversational events (Poesio, 1995) boxcar  [u|u:utter(A,”boxcar”), Noun(u),sem(u)=x [|boxcar(x)], + SYN INFO (NEXT)] umm [u,ce| u: utter(A,”umm”), ce: keep-turn(A), generate(u,ce)]

  15. MCEs in the example from the Bielefeld corpus Inst: So, jetzt nimmst DuWell, now you grasp Cnst: eine Schraubea screw. Inst: eine <-> orangene mit einem Schlitz.an <-> orange one with a slit Cnst: Ja. Inst: Und steckst sie dadurch, also And you put it through, let’s see ….Cnst: von oben

  16. MCEs in the example from the Bielefeld corpus [mce1,ce1| mce1:utter(Inst,“so"), Adv(mce1), ce1:take-turn(Inst), generate(mce1,ce1)]; [mce2,ce2| mce2:utter(Inst,“jetzt"), Adv(mce2), ce2:keep-turn(Inst), generate(mce2,ce2)]; [mce3| mce3:utter(Inst,"nimmst"), Verb(mce3), sem(mce3)= Qx(Q(x’[e| e: grasp(x, x’)]))]; [mce4| mce4:utter(Inst,"Du"), Pro(mce4), sem(mce4)= P.P (you)]; [mce5| mce5:utter(Cnst,"eine"), Det(mce5), sem(mce5)= P’P([y| ]; P’(y); P(y))] [mce6| mce6:utter(Cnst,"Schraube"), Noun(mce6), sem(mce6)= v([ |screw(v)]];

  17. MCE1 CE1 MCE2 CE2 MCE3 MCE4 mce1:utter(Inst,“so"),ce1:take-turn(Inst), generate(mce1,ce1)]; U1:S U4:NP U2:NP MCE3:”nimmst”:V U3:NP MCE4:”Du”:Pro TAG-based syntactic interpretation with MCEs

  18. Incremental interpretation of references M: so we should move the engine at Avon,engine E, to …S: engine E1M: E1S: okayM: engine E1, to Bath, to / or, we could actually move it to Dansville to pick up the boxcar there

  19. Two approaches to the semantics of anaphora and accessibility (Poesio and Muskens, 1997) • Accessibility via discourse structure (cfr. Grosz and Sidner / SDRT) • Can be extended to MCEs (Poesio and Rieser, in progress) • Compatible with view of anaphoric underspecification as homonymy (Poesio, 2001) • Problematic for visual situation & dynamics of questions / imperatives • Accessibility via resource situations (Poesio, 1993, 1994)

  20. Anaphoric expressions and resource situations the  [u|u:utter(A,”the”), Det(u),sem(u)= K  P  P’ [ x | ]; [ | x = y. K; P(x)]; P’(x) sem(the engine at Avon)  K  P’ [ x | ]; [ | x = y. K; [|engine (y), at(y,Avon))]]; P’(x)

  21. Pronouns and resource situations sem(Sie)  K P P’ [ x | ]; [ | x = y. K; [|P(y)]]; P’(x)

  22. Resource situations: MapS

  23. Resource situations in the discourse situation [MapS, u1| MapS=[a,b,c,d,e, ….. w,s, ……| Avon(a), engine(x),s: at(x,a), Bath(b), ….. ],…..u1: utter(M,”the engine at Avon”),sem(u1) =  K  P’ [ x | ]; [ | x = y. K; [|engine (y),at (y, a))] ]; P’(x)]

  24. Incremental reference interpretation • res-sit(u1)=MapS • ( K  P’ [ x | ]; [ | x = y. K; [|engine (y),at (y, a))] ]; P’(x)) (MapS)  • sem(u1) =  P’ [ x | ]; [ | x = y. MapS; [|engine (y),at (y,a))] ]; P’(x)

  25. Shifting the focus of (visual) attention M: while this is happening, take engine E1 to Dansville, pick up the boxcar, and come back to AvonS: okayM: okay then load the boxcar with bananas

  26. The theoretical model: mutual (visual) focus of attention • Grosz 1977: conversational participants may share a MUTUAL FOCUS OF ATTENTION, which may change in time • Participants keep track of where this is • Poesio (1992, 1993, 1994a, 1994b): • Separate linguistic and visual mutual foci • The visual focus is a situation, the MUTUAL SITUATION OF ATTENTION (MSOA) • This situation may serve as a resource situation

  27. MSOAs as Resource situations: DansvilleS

  28. MSOAs in the discourse situation [MapS, DansvilleS| MapS=[a,b,c,d,e, ….. w,s, ……| Avon(a), engine(x), s: at(x,a), Bath(b), …..],DansvilleS=[d,x,s | Dansville(d), boxcar(x), s: at(x,d)], …..….MSOA = DansvilleS]

  29. Follow-the-movement • Grosz 1977: FOCUS SHIFT RULES • FOLLOW THE MOVEMENT (Poesio, 1994) • Part of the intended effect of an utterance instructing an agent to move an object from one location to another is to make the terminal location of the movement the new mutual situation of attention.

  30. Pick up the boxcar And come back to Avon Follow the movement Take engine E1 to Dansville

  31. Using MSOAs as resource situations [DansvilleS, u1| DansvilleS=[d,x,s | Dansville(d), boxcar(x), s: at(x,d)],MSOA = DansvilleS….u1: utter(M,”the boxcar”),sem(u1) =  K  P’ [ x | ]; [ | x = y. K; [|boxcar (y)] ]; P’(x)]

  32. Some questions • Theoretical questions • Two models of anaphoric accessibility • Formal aspects of the incremental interpretation mechanism • Dynamic behaviour of questions and instructions • Empirical questions • Incremental interpretation • Initial interpretation of anaphoric expressions: homonymy or polysemy? • Initial resolution: surface only? • Follow the movement • Effect of motion verbs? Or bridging? (Cfr. Asher and Lascarides, 1998)

  33. Empirical developments since 1994 • Psychological side: the Rochester visual world work • Corpus side: improved methods for annotating anaphora, now also in dialogue

  34. Preliminary answers from psychology • Rapid incrementality in reference: Spivey et al, 1995 / Tanenhaus et al, 1995 • Task constraints are taken into account in selecting the antecedent: Chambers et al 2002 • Pick up the cube. Now put the cube in a / the can. • Also: Altmann and Kamide 1999 • Brown-Schmidt et al 2004: Referential domains circumscribed to small spatial regions • Other evidence for ‘follow the movement’ effects: • Stevenson et al, 1993: Bill sent a letter to Sue • Matsui 2000: John moved from New Zealand to England. He loves the sheep. • But also: challenges to the ‘mutual situation’ account (e.g., (Horton and Keysar, 1995))

  35. Further progress on corpus annotation • GNOME: detailed coding scheme for anaphora (for texts) • VENEX: coding scheme for both text and dialogue (MapTask-style) (in Italian) • ARRAU: • Further agreement studies • Annotation of the TRAINS dialogues

  36. Hopes / plans for network • Discussions with developers of the other theoretical frameworks represented (Cooper / Ginzburg, Dynamic Tree theory) • Other theoretical / computational frameworks for references to the visual situation • POTLOG Symposium ? • Develop methodology for annotating multimodal data • E.g., Donna Byron’s data • If possible, design psychological experiments • Work also on the ‘other’ side of PTT (grounding and alignment)

  37. The Bielefeld Toy Plane Corpus • 22 video-filmed, speech recorded and transcribed dialogues • two agents, Instructor and Constructor • constructing a “Baufix” airplane • different sight conditions: total screen, half-screen, face to face • 3675 contributions • 160 sentence cooperations (4.34 %) • in most of them cooperation other-initiated (95%)

  38. Common ground: beyond assertion A They have at their disposal enormous assets // and their policy B //look can I just come in on that// last year A //YES IN A MINUTE IF YOU MAY AND WHEN I’M FINISHED // then you’ll know B // yes I’M SO SORRY (Coulthard 1977)

  39. Common ground: beyond assertion B: Go to Elmhurst, pass the courthouse and go to Elmhurst and then to Elmhurst, uh north.A: mm hum.B: Towards Riverton, till you come to that Avila HallA: Oh yesB: Dju know where that//is?A: //uh huhA: Oh surelyB: Avilla Hall on the corner of Bor//donA: //uh huhB: Well there, on Bordon you turn back to town, left. (George Psathas, "Direction-giving in Interaction," in Boden and Zimmerman, ed.)

  40. Locutionary acts in the common ground “The fact that a speaker is speaking, saying the words he is saying in the way he is saying them, is a fact that is usually accessible to everyone present. Such observed facts can be expected to change the presumed common background knowledge of the speaker and his audience in the same way that any obviously observable change in the physical surroundings of the conversation will change the presumed common knowledge.” (Stalnaker, Assertion, p. 323)

  41. The time-order of sentence processing • GARDEN-PATH phenomena shows that parsing is INCREMENTAL (Bever, 1974; Frazier, 1987) • Marslen-Wilson 1973, 1975: semantic information ALSO accessed immediately • Swinney, 1979: lexical access incremental • Just and Carpenter,1980: IMMEDIACY HYPOTHESIS (“Every word encountered should be processed to the deepest level possible before the eye moves on to the next word”) • Eye-tracking work (Tanenhaus et al, 1995, tomorrow): really fine-grained incrementality

  42. Alignment at all levels Pickering & Garrod

  43. Clarification questions (Ginzburg and Cooper, Purver and Ginzburg) A: Did Bo leave?B: BO? A: Bo Smith.B: Yes, half an hour ago. Matthew: It wasn’t all that bad. At least the pool was clean.Lara: MR POOL?Matthew: The pool.Lara: Oh. <laugh> (“What is the intended content of your utterance ‘Bo’?”) (“Did you utter the words ‘Mr. Pool’?”)

  44. U3:() U1: U2:  Semantic interpretation and compositionality BINARY SEMANTIC COMPOSITION

  45. Intentions and obligations OBLIGATIONS: [o | o:OblCnst ([|address(Cnst, ce1)])] INTENTIONS: [i | i:IntInst&Cnst ([|join(Cnst, wing1,fuselage1)])] INTENTIONAL STRUCTURE: Grosz&Sidner-like sp(i1) = i2 dom(i1) = i2

  46. Grounding • As in proposals such as Clark and Schaefer (1989) and Traum (1994), establishment of common ground (‘G’) modeled in terms of CONTRIBUTIONS, or DISCOURSE UNITS, that may be ACKNOWLEDGED or REPAIRED

  47. DU1 …. DU17 MCE1 CE1 MCE2 CE2 MCE3 … ACK(DU17) mce1:utter(Inst,“so"),ce1:take-turn(Inst), generate(mce1,ce1)]; DU17 = MCE4:”Du”:Pro U1:S CONT(DU17) U2:NP MCE3:”nimmst”:V U3:NP REPAIR(DU17) Discourse Units and Grounding Acts

  48. An IAM-analysis of the BTPC example • What leads Cnst to produce “eine Schraube”? • What is a situation model in this domain? • “the key dimensions encoded in situation models are SPACE, TIME, CAUSALITY, INTENTIONALITY, and REFERENCE to the MAIN INDIVIDUALS … “ (p. 7)

  49. Preliminary conclusions • PTT provides the technical tools to formalize a crucial feature of sentence cooperations: coordination at the micro conversational event level • Mind-reading always difficult, but Tuomela’s theory of we-intention goes some way towards formalizing one of the possible motivations for completions, in terms of “help” • A preliminary investigation of the alignment route also possible

More Related