Jan Alexandersson (Tilman Becker, Markus Löckelt, Norbert Pfleger)

1. Overlay – a non-monotonic operation for discourse processing Jan Alexandersson (Tilman Becker, Markus Löckelt, Norbert Pfleger) German Research Center for Artificial Intelligence – DFKI GmbH Stuhlsatzenhausweg 3, Geb. 43.8 - 66123 Saarbrücken phone: (+44) 681 302 5347 Email: janal@dfki.de http://www.dfki.de/~janal

2. What is this Talk about • Dialogue systems must deal with utterances relating elliptically to previous dialogue System: „at what time would you like to start recording?“ User:„half past five“ User: „I would like to start recording athalf past five“ • ... or utterances only partially related to the previous discourse User: „what films are on TV tonight?“ System: [Showing a list of films] „Here is a list of films“ User: ``Thats a boring program. I‘d rather go to the movies´´ User: ``Thats a boring program. I‘d rather go to the movies tonight´´

3. What is this Talk about? • There are similar elliptical phenomena like one-anaphora ``...an expensive large portrait of the queen by Wainright hanging in the library´´  ``and one of the princess over the mantel´´  ``and an expensive large portrait of the princess hanging over the mantel´´

4. Overview • SmartKom • Domain Model • Overlay • Discourse Modelling • Action Planning • Processing Partial Utterances • Conclusion

5. What is the challenge? • General tasks for a dialogue manager • Enrichment: • Given incomplete information, enrich it from other sources in order to proceed. • Validation: • Validate hypotheses against the current discourse state

6. Our Approach • One general operation – Overlay • Domain representation: • (Typed) Feature Structures • Non-monotonic: • Always succeeds • Computes a score: • Reflecting the structural consistency of the two arguments of overlay

7. SmartKom-Mobile SmartKom-Public SmartKom-Home/Office SmartKom: Task-Oriented Dialogue System Application Layer MM Dialogue Back- Bone Public: Cinema, Phone, Fax, Mail, Biometrics Mobile: Car and Pedestrian Navigation Home: Consumer Electronics EPG

8. Module Overview on the SmartKom Control GUI

9. The True Story

10. The DFKI Dialogue Back-Bone Analysers • Speech • Graphics • Gesture Generators • Speech • Gesture Communication pools Main data flow Context information Modality Fusion Discourse Modelling DiM Action Planning AP Presentation Manager External Services

11. Domain Model • Used for communication in the back-bone • Frame-based ontology; representation as Typed Feature Structures in M3L (XML) CinemaReservation theater: MovieTheater movie: Movie reservationNumber: PositiveInteger address: Address seats: SeatStructure… name: String director: Person cast: PersonList yearOfProduction: PositiveInteger… firstName: String lastName: String… • Application objects composed of subobjects • Slots: Feature paths meaningful for the dialogue (entities that can be talked about / referenced to); e.g. movie:director:lastName in a CinemaReservation object • Slots can recursively contain other slots

12. Discourse Modeling Intention lattice • Lattice of intention hypothesis sequences from Modality Fusion • Enrichment and Validation • compare and enrich with a selected number of discourse states: • fill in consistent information • compute a score • for each hypothesis - background pair: • Overlay (covering, background) • Overlay gives a score representing how well the covering fits the background. • Scores of overlay, speech interpretation etc. give overall score • One augmented hypothesis sequence with best score is passed on to Action Planning Covering: Background: Selected augmented hypothesis sequence

13. Example for Overlay with TFS User: What films are on TV tonight? System: [presents list of films] User: That‘s a boring program, I‘d rather go to the movies. How do we inherit “tonight” ?

14. The Domain Model A named entertainment at some time A named TV program at some time on some channel A named Movie at some time at some cinema

15. Films on TV tonight Go to the movies Unification Simulation Fail – type clash

16. Films on TV tonight Overlay Simulation Assimilation Background Go to the movies Covering

17. ``Formal´´ Definition Overlay T • Let • co be covering • bg be background • Step 1: • Assimilate(co,bg) • Step 2: • Overlay(co,assimilate(co,bg)) • If co and bg are frames: recursion • If co is empty: use bg • If bg is empty: use co • If conflict: use co bg co

18. Domain Models with Multiple Inheritance T • Assimilate(co,bg) • Compute the set of minimal upper bounds (MUB) • Specialize the MUBs • Unify the specialized MUBs MUB MUB co bg • Overlay remains untouched

19. Processing Partial User Contributions • Our Dialogues contain a lot of partial contributions • ``Three´´ • ``Three o‘clock´´ • ``At three o‘clock´´ • ``At three [] o‘clock´´ • ``Later´´ • ``Information []´´ • ... • Requirement • Advanced discourse modelling and action planning • Bidirection communication DiM - AP

20. Discourse Modelling -Three-Tiered Context Representation Domain layer DomainObject2 DomainObject1 Discourse layer DO1 DO2 DO10 DO11 DO12 DO3 DO9 . . . Modality layer LO4 LO5 LO6 LO2 GO1 LO3 LO1 . . . . . . reserve ticket first show list  heidelberg System: Here [pointing gesture] I show you a list of films running in Heidelberg. User: Reserve a ticket for the first!

21. Action Planner – Regression Planning • AP processes plan operators • Uniform description of mixed initiative with user or applications based on Games and Moves. Example: • User: request – System: response • System: request – Application: response • Processing steps • Constructs a plan from plan operators • A Plan specifies a partial order of games • Interprets the plan • Execute the games • Expectations are published during execution of the plan • Guide the interpretation of input

22. Goal provides() needs(X,Y) provides(C) needs() provides(A) needs(C) provides(X) needs(A,B) provides(B) needs(D) successor states provides(D) needs(E) Current State provides(?) needs(?) provides(F,G) needs(H) For Example: Slot D=y Action Planning - Regression planning provides(Y) needs(D) condition(D=x)

23. VCR_getChannel VCR_getStartTime VCR_getEndTime VCR_checkMedium VCR_rewindMedium AP - Example Plan: VCR Goal: VCR_record

24. Action Planning • Initiation of sub-dialogues to gather necessary information • Order proposed by the system, but the user may not follow the trail • Several cases of user responses to requests: • Providing a matching answer, • Supplying information not (yet) asked for, • Changing already established information • Acting uncooperatively or changing goals

25. Expectations • When AP initiates a sub-dialogue, it expects a certain answer • An Expectationstructure is published to help the analysis modules • Expected Slots: corresponding to the request • Possible Slots: plausible in context • Filled Slots: already provided • The currently active Application • Allows for a fine-grained distinction between types of possible utterances • Discourse Modelling can employ different rankings for each category

26. Integration of Partial • Partial Utterances are analyzed as subobjects • validation and enrichment include the integration into the preceding context (to achieve a coherent discourse) • we consider three types of partial utterances: • The system has the initiative and the user responds elliptically to a system request • An elliptical user utterance does not correspond to a request but can be interpreted in context • Misinterpretation or non-cooperative user behaviour

27. Integration of Partial - Possible Situations User contribution DiM processing AP processing

28. DiM - Processing User Responses • User provides expected slots • Bridging: • a new application object is created (of same type as the current application object) • for each subobject in the hypotheses its type is compared to the expected slots • matching subobjects are integrated into the new application object • remaining subobjects are interpreted as plausible partial utterances (next slide) • Overlay: • The extended application object is overlayed over the application object currently in focus

29. Processing Plausible Partial Utterances • User provides possible slots • processed like expected slots (but get a score penalty) • User changes filled slots • search for a discourse object DO of the same type in the discourse context • create application object AO1 of the same type as the application object AO2 within which DO was mentioned • overlay AO1 over AO2 • gets a score penalty based on position in discourse history • If integration fails, all subobjects are passed on unchanged (integration fails)

30. Example: Expected and Possible Slots • (1) User:I want to record a film. (2)System:When should I start recording? (3)User:1:30pm on channel two. • Analysis of (1) contains the setting of a goal VCR_record (empty application object of type VCR introduced) • AP triggers presentation (2) and publishes an expectation containing: • VCR as the type of the current application object • VCR_startTime as the (only) expected slot • some possible slots (e.g., VCR_endTime and VCR_channel) • and maybe some filled slots (e.g. VCR_mediumPresent) • Analysis of (3) contains two subobjects • the expected slot is filled by “1:30pm” • but also one possible slot is filled by “channel two”

31. Example: Filled Slots • (1) User:What is currently running on TV? • (2) System:The following [Smartakus points at a list of programmes] programmes are currently running. • (3) User:And tonight?

32. Domain layer Discourse layer DO3 DO1 DO2 DO3 DO4 Modality layer LO3 LO2 LO1 LO4 program currently show tonight Example: Filled Slots

33. Conclusion • Uniform, flexible and robust mechanism for interpreting possible partial utterances • Pragmatical classification of partial utterances from the point of view of the action planner • Modality independent • Approach integrates seamlessly with other aspects of discourse processing, e.g. anaphora resolution • Implemented and running in, e.g., a large multimodal dialogue system

34. Papers • Jan Alexandersson and Tilman Becker. Overlay as the Basic Operation for Discourse Processing in a Multimodal Dialogue System. In: Proceedings of the IJCAI Workshop ``Knowledge and Reasoning in Practical Dialogue Systems,´´ Seattle, 2001. • Norbert Pfleger, Jan Alexandersson, and Tilman Becker. Scoring Functions for Overlay and their Application in Discourse Processing. In Proceedings of ``KONVENS 2002,´´ Saarbrücken, Germany, 2002. • Markus Loeckelt, Tilman Becker, Norbert Pfleger and Jan Alexandersson Making Sense of Partial. In: Bos, Foster & Matheson (eds): ``Proceedings of the sixth workshop on the semantics and pragmatics of dialogue (EDILOG 2002),´´ Edinburgh, UK, Pages 101-107. • Jan Alexandersson and Tilman Becker. The Formal Foundations Underlying Overlay. Submitted to IWCS5

35. Thank you very much for your attention! Merci infiniment de votre attention!Babelfish.altavista.com 