Dialog Management

1. Dialog Management Intelligent Robot Lecture Note

2. Dialog System & Architectures Intelligent Robot Lecture Note

3. Dialogue System Intelligent Robot Lecture Note • A system to provide interface between the user and a computer-based application • Interact on turn-by-turn basis • Dialogue manager • Control the flow of the dialogue • Main flow • information gathering from user • communicating with external application • communicating information back to the user • Three types of dialogue system • finite state- (or graph-) based • frame-based • agent-based

4. Dialog System Architecture Intelligent Robot Lecture Note • Typical dialog system has following components • User Interface • Input: Speech Recognition, keyboard , Pen-gesture recognition .. • Output: Display, Sound, Vibration .. • Context Interpretation • Natural language understanding (NLU) • Reference resolution • Anaphora resolution • Dialog Management • History management • Discourse management • Many dialog system architectures are introduced. • DARPA Communicator • GALAXY Communicator • etc.

5. MIT AT&T SRI CMU DARPA CU Bell Lab BBN Dialog System Architecture Intelligent Robot Lecture Note The DARPA Communicator program was designed to support the creation of speech-enabled interfaces that scale gracefully across modalities, from speech-only to interfaces that include graphics, maps, pointing and gesture.

6. Galaxy Communicator Intelligent Robot Lecture Note • The Galaxy Communicator software infrastructure is a distributed, message-based, hub-and-spoke infrastructure optimized for constructing spoken dialogue systems. • An open source architecture for constructing dialogue systems • History • MIT Galaxy system • Developed and maintained by MITRE Corporation • Current version is 4.0

7. Galaxy Communicator Intelligent Robot Lecture Note The architecture

8. Galaxy Communicator Intelligent Robot Lecture Note Message Passing Protocol

9. CU Communicator Intelligent Robot Lecture Note • Dialogue management in CU Communicator • Event-driven approach • Current context of the system is used to decide what to do next • Do not need a dialogue script • A general engine operates on the semantic representations and the current context to control the interaction flow • Mixed-initiative approach • Not separate “user initiative” and “system initiative”

10. CMU Communicator • Dialogue management in CMU Communicator • Frame-based approach • Form-filling method • Not to specify a particular order in which slots need to be filled • Loosen the requirement for the system designed to correctly intuit the natural order in which information is supplied • Agenda-based approach • Treats the task as one of cooperatively constructing a complex data structure, a product • Uses a product tree which is developed dynamically • Supports topic shifts Intelligent Robot Lecture Note

11. Queen’s Communicator • Object-oriented architecture, distributed and inherited functionality: generic and domain-specific • Uses discourse history and confirmation status to determine how to confirm (explicit or implicit) Intelligent Robot Lecture Note

12. Dialog System Approaches Intelligent Robot Lecture Note

13. Dialog System approaches Intelligent Robot Lecture Note • There are many approaches to represent dialog • Frame based • Agent based • Voice-XML based • Information State approach

14. Example 1) System: What is your destination? User: London. System: What day do you want to travel? User: Friday Frame-based Approach • Frame-based system • Asks the user questions to fill slots in a template in order to perform a task (form-filling task) • Permits the user to respond more flexibly to the system’s prompts (as in Example 2.) • Recognizes the main concepts in the user’s utterance Example 2) • System: What is your destination? • User: London on Friday around 10 in the morning. • System: I have the following connection … Intelligent Robot Lecture Note

15. Frame-based Approach • Advantages • The ability to use natural language, multiple slot filling • The system processes the user’s over-informative answers and corrections • Disadvantages • Appropriate for well-defined tasks in which the system takes the initiative in the dialog • Difficult to predict which rule is likely to fire in a particular context • Related systems • CU Communicator • CMU Communicator Intelligent Robot Lecture Note

16. Agent-based Approach • Properties • Complex communication using unrestricted natural language • Mixed-Initiative • Co-operative problem solving • Theorem proving, planning, distributed architectures • Conversational agents • Examples User : I’m looking for a job in the Calais area. Are there any servers? System : No, there aren’t any employment servers for Calais. However, there is an employment server for Pasde-Calais and an employment server for Lille. Are you interested in one of these? • System attempts to provide a more co-operative response that might address the user’s needs. Intelligent Robot Lecture Note

17. Agent-based Approach • Advantages • Suitable to more complex dialogues • Mixed-initiative dialogues • Disadvantages • Much more complex resources and processing • Sophisticated natural language capabilities • Complicated communication between dialogue modules • Related Works • TRAINS project • TRIPS project Intelligent Robot Lecture Note

18. TRAINS project • TRAINS (1995~1997) • CISD research group in University of Rochester • http://www.cs.rochester.edu/research/cisd/projects/trains/ • Task • Finding efficient routes for trains • Goal • Robust performance on a very simple task • Approach • Speech Act, Plan reasoning • Demo • http://www.cs.rochester.edu/research/cisd/projects/trains/movies/TRAINS95-v1.3-Pia.qt.gz Intelligent Robot Lecture Note

19. TRIPS Project Intelligent Robot Lecture Note • TRIPS • The Rochester Interactive Planning System • http://www.cs.rochester.edu/research/cisd/projects/trips/ • Goal • An intelligent planning assistant (natural language + graphical display) • Extending TRAINS system to several domain • Domains (supported currently) • Pacifica - Evacuating people from an island • Airlift – Organization Airlift scheduling • TRIPS-911 – Managing the resources in small 911 emergency • Underwater Survey – Planning in collaboration with semi-autonomous robot agents • Demo (Pacifica) • http://www.cs.rochester.edu/research/cisd/projects/trips/movies/TRIPS-98_v4.0/200K/TRIPS-98_v4.0_200K.html

20. TRIPS Architecture The TRIPS System Architecture Intelligent Robot Lecture Note

21. VoiceXML-based System • What is VoiceXML? • The HTML(XML) of the voice web. • The open standard markup language for voice application • Can do • Rapid implementation and management • Integrated with World Wide Web • Mixed-Initiative dialogue • Able to input Push Button on Telephone • Simple Dialogue implementation solution Intelligent Robot Lecture Note

22. Dialogue by VoiceXML Intelligent Robot Lecture Note • Most VoiceXML dialogues are built from • <menu> • <form>  form based dialog • Form-based dialogue is similar to “Slot & Filling” system • Limiting User’s Response • Goal • Verification, and Help for invalid response • Good speech recognition accuracy

23. Example - <Menu> Browser : Say one of: Sports scores; Weather information; Log in. User : Sports scores <vxml version="2.0" xmlns="http://www.w3.org/2001/vxml"> <menu><prompt>Say one of: <enumerate/></prompt> <choice next="http://www.example.com/sports.vxml">Sports scores </choice> <choice next="http://www.example.com/weather.vxml">Weather information </choice> <choice next="#login">Log in </choice> </menu> </vxml> Intelligent Robot Lecture Note

24. Example – <Form> Browser : Please say your complete phone number User : 800-555-1212 Browser : Please say your PIN code User : 1 2 3 4 <vxml version="2.0" xmlns="http://www.w3.org/2001/vxml"> <form id="login"> <field name="phone_number" type="phone"> <prompt>Please say your complete phone number </prompt> </field> <field name="pin_code" type="digits"> <prompt>Please say your PIN code </prompt> </field> <block> <submit next=“http://www.example.com/servlet/login” namelist=phone_numberpin_code"/> </block> </form> </vxml> Intelligent Robot Lecture Note

25. Information State Approach • A method of specifying a dialogue theory that makes it straightforward to implement • Consisting of following five constituents • Information Components • Including aspects of common context • (e.g., participants, common ground, linguistic and intentional structure, obligations and commitments, beliefs, intentions, user models, etc.) • Formal Representations • How to model the information components • (e.g., as lists, sets, typed feature structures, records, etc.) Intelligent Robot Lecture Note

26. Information State Approach • Dialogue Moves • Trigger the update of the information state • Be correlated with externally performed actions • Update Rules • Govern the updating of the information state • Update Strategy • For deciding which rules to apply at a given point from the set of applicable ones Intelligent Robot Lecture Note

27. Example Dialogue Intelligent Robot Lecture Note

28. Example Dialogue Intelligent Robot Lecture Note

29. Example Dialogue Intelligent Robot Lecture Note

30. Example Dialogue Intelligent Robot Lecture Note

31. Example Dialogue Intelligent Robot Lecture Note

32. Example Dialogue Intelligent Robot Lecture Note

33. Reading Lists • B. Pellom, W. Ward, S. Pradhan, 2000. The CU Communicator: An Architecture for Dialogue Systems, International Conference on Spoken Language Processing (ICSLP), Beijing China. • Rudnicky, A., Thayer, E., Constantinides, P., Tchou, C., Shern, R., Lenzo, K., Xu W., Oh, A. 1999. Creating natural dialogs in the Carnegie Mellon Communicator system. Proceedings of Eurospeech, 531-1534. • Ian M. O’Neill and Michael F. McTear. 2000. Object-Oriented Modelling of Spoken Language Dialogue Systems Natural Language Engineering, Best Practice in Spoken Language Dialogue System Engineering, Special Issue, Volume 6 Part 3. • George Ferguson and James Allen, July 1998. TRIPS: An Intelligent Integrated Problem-Solving Assistant," in Proceedings of the Fifteenth National Conference on Artificial Intelligence (AAAI-98), Madison, WI, 26-30, pp. 567-573. Intelligent Robot Lecture Note

34. Reading Lists • S. Larsson, D.R. Traum. 2001. Information state approach to dialogue management. Current and New Directions in Discourse & Dialogue, Kluwer Academic Publishers. • S. Larsson, D.R. Traum. 2003. Information state and dialogue management in the TRINDI dialogue move engine toolkit. Natural Language Engineering. Intelligent Robot Lecture Note

35. Dialog Modeling Techniques Intelligent Robot Lecture Note

36. Training Info = desired (target) outputs Supervised Learning System Outputs Inputs (Feature, Target Label) Objective: To minimize error (Target Output – Actual Output) Training Info = evaluations (“rewards”/”costs”) RL System Outputs (“actions”) Inputs (State, Action, Reward) Objective: To get as much reward as possible Reinforcement Learning Intelligent Robot Lecture Note

37. Stochastic Modeling Approach Cimeasures the effectiveness and the achievement of application goal Intelligent Robot Lecture Note • Stochastic Dialog Modeling [E. Levin et al, 2000] • Optimization Problem • Minimization of Expected Cost (CD) • Mathematical Formalization • Markov Decision Process • Defining State Spaces, Action Sets, and Cost Function • Formalize dialog design criteria as objective function • Automatic Dialog Strategy Learning from Data • Reinforcement Learning

38. Mathematical Formalization Dialog Manager Dialog State Dialog Action (Prompts, Queries, etc.) Cost (Turn, Error, DB Access, etc.) Environment (User, External DB or other Servers) Intelligent Robot Lecture Note • Markov Decision Process (MDP) • Problems with cost(or reward) objective function are well modeled as Markov Decision Process. • The specification of a sequential decision problem for a fully observable environment that satisfies the Markov Assumption and yields additive rewards.

39. Dialog as a Markov Decision Process dialog history noisy estimate of user dialog act user dialog act user goal Speech Understanding State Estimator Reward User machine state Reinforcement Learning Speech Generation Dialog Policy Optimize MDP machine dialog act [S. Young, 2006] Intelligent Robot Lecture Note

40. Month and Day Example Intelligent Robot Lecture Note • State Space • State St represents all the knowledge of the system at time t (values of the relevant variables). • St=(d, m) where d=-1,…,31 and m=-1,..,12 • 0 : not yet filled • -1 : completely filled • (0,0) = Initial State • (-1,-1) = Final State

41. Day:1 - - Month:1 Day:1 Month:1 Day:30 - - Month:11 Day:30 Month:12 Day:31 - - Month:12 Day:31 Month:12 Month and Day Example - - - - 1 (initial) + 12(months) + 31(days) + 365(dates) + 1(final) Total Dialog State : 410 states Intelligent Robot Lecture Note State Space

42. Which month? (Am) Which day? (Ad) Which date? (Adm) Thank you. Good Bye.(Af) St Month and Day Example Intelligent Robot Lecture Note • Action Set • At each state, the system can choose an action at. • Dialog Actions • Asking the user for input, providing a user some output, confirmation, etc.

43. SYSTEM : Which month? - Month: 1 - - - Month: 11 - Month: 12 Month and Day Example New state might depend on external inputs: Not Deterministic Transition Probability: PT(St+1|St,at) Intelligent Robot Lecture Note • State Transitions • When an action is taken the system changes its state.

44. Month and Day Example SYSTEM : Which month? - Month: 1 - - Cost Distribution: Pc(Ct|St,at) - Month: 11 - Month: 12 Intelligent Robot Lecture Note • Action Costs and Objective Function • A cost Ctis associated to action at at state St.

45. Month and Day Example Strategy 1. Good Bye. - - - - Strategy2. Which date ? Good Bye. - - Day Month - - Strategy 3. Which day ? Which month? Good Bye. - - Day - Day Month - - Optimal strategy is the one that minimizes the cost. Strategy 1 is optimal if wi + P2* we - wf> 0  Recognition error rate is too high Strategy 3 is optimal if 2*(P1-P2)* we - wi> 0  P1 is much more high than P2 against a cost of longer interaction Intelligent Robot Lecture Note

46. Policy a1 a2 a0 S0 S1 S2 … r1 r2 r0 Goal : Learn to choose actions that maximize the reward function. discount factor Intelligent Robot Lecture Note • The goal of MDP is to learn a policy, π : S→A • But we have no training examples of form <s,a> • Training examples are of form <s,a,s’,r> • For selecting it next action at based on the current observed state st.

47. Policy Intelligent Robot Lecture Note • Discounted Cumulative Reward • Infinite-Horizon Model • γ=0 : Vπ(st) =rt • Only immediate reward considered. • γ closer to 1 : Delayed Reward • Future rewards are given greater emphasis relative to the immediate reward. • Optimal Policy (π*) • Optimized policy π that maximize Vπ(s) for all state s.

48. Q-Learning Intelligent Robot Lecture Note • Define the Q-Function. • As evaluation function. • Rewrite the optimal policy. • Why is this rewrite important? • It shows that if the agent learns the Q-function instead of the V* function. • It will be able to select optimal actions even when it has no knowledge of the function r and δ.

49. Q-Learning Intelligent Robot Lecture Note • How can Q be learned? • Learning the Q function corresponds to learning the optimal policy. • The close relationship between Q and V* • It can be written recursively as • This recursive definition of Q provides the basis for algorithm that iteratively approximate Q. • It can updates the table entry for Q(s,a) following each such transition, according to the rule.

50. Q-Learning Intelligent Robot Lecture Note Q-Learning algorithm for deterministic MDP.