Final Exam Information

1. Final Exam Information • Exam is Friday, December 13 11AM-1PM • Exam will be cumulative, slightly emphasizing material since midterm • Extra credit (and advantage) for problems submitted by email that we use • Final review on Thursday --- come with your questions • Also: term papers due Thursday

2. In 1985: computers used by scientists & engineers little attention to user interfaces or CHI almost no “social” apps little interaction between CS and psych. little attention to design principles web did not exist In 2002: most Americans have web/Internet popular culture and CS mixed together Technology and society increasingly intertwined many apps are social “civilians” can generate rich technology CHI a rich research area AI and CHI The Societal Impact of Computer (and Cognitive) Science Today: Case studies in spoken language interfaces and AI

3. Spoken Dialogue Systems • Provide automated telephone access to DB • Front end: ASR + TTS • Back end: DB • Middle: dialogue strategy is key component ASR DB user spoken dialogue system TTS

4. State-Based Design • System state: • info attributes perceived so far • individual and average ASR confidences • other dialogue history info • data on particular user • Dialogue strategy: mapping from current state to system action • Typically hundreds of states, several reasonable actions from each state

5. Typical System Design:Sequential Search • Choose state and action spaces • Choose and implement a particular, “reasonable” dialogue strategy • Field system, gather dialogue data (system logs, dialogue trajectories) • Do simple statistical analyses • Re-field improved dialogue strategy • Can only examine a handful of strategies

6. Issues in Dialogue Strategy Design • Initiative strategy • Confirmation strategy • DB query strategy • Criteria to be optimized

7. Facts About ASR • Inputs: audio file; grammar/language model; acoustic model • Outputs: utterance matched from grammar, or no match; log-likelihood • Performance precision-recall tradeoff: • “small” grammar --> high accuracy on constrained utterances, lots of no-matches • “large” grammar --> match more utterances, but with lower confidence • ASR community pushing these barriers

8. Initiative Strategy • System initiative vs. user initiative: • “Please state your departure city.” • “Please state your desired itinerary.” • “How can I help you?” • Influences user expectations • ASR grammar must be chosen accordingly • Best choice may differ from state to state! • May depend on user population & task

9. Confirmation Strategy • High ASR confidence: accept ASR match and move on • Moderate ASR confidence: confirm • Low ASR confidence: re-ask • How to set confidence thresholds? • Early mistakes can be costly later

10. 2 0.8 1 1.0 3 0.2 4 Markov Decision Processes • System state s (in S) • System action a in (in A) • Transition probabilities P(s’|s,a) • Reward function R(s,a) (stochastic) • Fast algorithms for optimal policy • Our application: P(s’|s,a) models the population of users

11. SDSs as MDPs Initial system utterance Initial user utterance Actions have prob. outcomes + system logs a e a e a e ... 1 1 2 2 3 3 estimate transition probabilities... P(next est. state | current est. state & action) ...and rewards... R(current est. state, action) ...from set of exploratory dialogues Violations of Markov property! Will this work?

12. The RL Approach • Build initial system that is deliberately exploratory wrt state and action space • Use dialogue data from initial system to build a Markov decision process (MDP) • Use methods of reinforcement learning to compute optimal strategy of the MDP • Re-field (improved?) system given by the optimal policy

13. Why Reinforcement Learning? • ASR output is noisy; user population leads to stochastic behavior • Design choices have long-term impact; temporal credit assignment problem • Many design choices can be fixed, but - Initiative strategy - Confirmation strategy • Many different performance criteria

14. Caveats • Must still choose states and actions • Must be exploratory with taste • Data sparsity • Violations of the Markov property • A formal framework and methodology, hopefully automating one important step in system design

15. The Application • Dialogue system providing telephone access to a DB of activities in NJ • Want to obtain 3 attributes: • activity type (e.g., wine tasting) • location (e.g., Lambertville) • time (e.g., morning) • Failure to bind an attribute: query DB with don’t-care

16. current attribute (A = 1,2,3) value (V = 0,1) confidence (C = 1,2,3,4,5) tries (T = 0,1,2,3) grammar (G = 0,1) “trouble” history bit (H = 0,1) The State Space N.B. Non-state variables record attribute values; state does not condition on previous attributes! Will this work?

17. Sample Actions • Initiative (when T = 0): • open or constrained prompt? • open or constrained grammar? • N.B. might depend on H, A,… • Confirmation (when V = 1) • confirm or move on or re-ask? • N.B. might depend on C, H, A,… • Only allowed “reasonable” actions • Results in 42 states with (binary) choices • Small state space, large policy space

18. The Experiment • Designed 6 specific tasks, each with web survey • Gathered 75 internal subjects • Split into training and test, controlling for M/F, native/non-native, experienced/inexperienced • 54 training subjects generated 311 dialogues • Exploratory training dialogues used to build MDP • Optimal strategy for objective TASK COMPLETION computed and implemented • 21 test subjects performed tasks and web surveys for modified system generated 124 dialogues • Did statistical analyses of performance changes

19. Reward Functions • Objective task completion: • -1 for an incorrect attribute binding • 0,1,2,3 correct attribute bindings • Binary version: • 1 for 3 correct bindings, else 0 • Other reward measures: ASR confidence (obj), perceived completion, user satisfaction, future use, perceived understanding, user understanding, ease of use • Optimized for objective task completion, but predicted improvements in some others

20. Main Results On all dialogues: • Objective task completion: • train mean ~ 1.722, test mean ~ 2.176 • two-sample t-test p-value ~ 0.0289 • Binary task completion: • train mean ~ 0.515, test mean ~ 0.635 • two-sample t-test p-value ~ 0.05 On expert dialogues 3-6: • Binary task completion • - train mean ~ 0.456, test mean ~ 0.682 • - two-sample t-test p-value ~ 0.001

21. Results for Other Rewards • ASR performance (0-3): • train ~ 2.483, test ~ 2.671, p ~ 0.0375 • User satisfaction (“move to the middle” effect): • %good: train ~ 0.459, test ~ 0.251, p ~ 0.06 • %bad: train ~ 0.278, test ~ 0.138, p ~ 0.07 • Similar significant MTM results for ease of use • Many insignificant instances of MTM • Objectives improve, subjectives MTM

22. Comparison to Human Design • Fielded comparison infeasible, but exploratory dialogues provide a Monte Carlo proxy of “consistent trajectories” • Test policy performance, binary completion: 0.67 (12) • Policy SysNoConfirm: -0.08 (11), significant win • Policy SysConfirm: -0.6 (5), significant win • Policy UserNoConfirm: -0.2 (15), significant win • Policy Mixed: -0.077 (13), significant win • Insignificant: difference with W99, similar to test • Even this is a potential advance...

23. Cobot: A Software Agent • User/client of LambdaMOO, a well-known Internet text chat and programming environment • Software chat agent providing “social statistics” • Previous functionality: • Extensive logging of human user behavior and interaction • Provision of social statistics and comparisons • Rudimentary chat based on IR applied to large documents • Proactive social behavior via reinforcement learning • This work: • Construction, fielding, and testing of a dialogue system providing spoken natural language access to Cobot and LambdaMOO via telephone, using speech recognition and text-to-speech

24. Sample Dialogue HFh waves to Buster. Buster bows gracefully to HFh. Buster is overwhelmed by all these paper deadlines. Buster begins to slowly tear his hair out, one strand at a time. HFh comforts Buster. HFh [to Buster]: Remember, the mighty oak was once a nut like you. Buster [to HFh]: Right, but his personal growth was assured. Thanks anyway, though. Buster feels better now. Standard verbs and emotes: directed and broadcast speech, hug, wave, bow, nod, eye, poke, zap, grin, laugh, comfort, ...

26. Calling Cobot • Provided a dozen or so “friendly” LambdaMOO users with access to a toll-free CobotDS number • Users call with LambdaMOO user name, numeric password; then enter main CobotDS command loop • Cobot announces phone call & user in LambdaMOO • From LambdaMOO to phone user: • MOO users direct arbitrary utterances or verbs to Cobot, prefixed by text “phone:” • Via TTS, Cobot passes verb or utterance directly to phone user • Virtually no noise on this channel • From phone user to LambdaMOO: Cobot passes on utterances, verbs from phone user (with attribution) • Mixed in with Cobot’s other behavior and activities • But this channel is very noisy (due to ASR), so…

27. Basic Phone Commands • 38 standard LambdaMOO verbs, directed or not • “Say” command with multiple ASR grammars: • Smalltalk grammar: 228 useful phrases & exclamations • Social pleasantries, assertions of mood • Statements of whereabouts • Cliché grammar: 2950 English witticisms and sayings • User specific personal grammars, periodically updated/modified • User can control grammar via the “grammar” command • “Listen” command: • At every dialogue turn, CobotDS will attempt to describe all activity taking place in LambdaMOO • Provides phone user richer view, allows passive participation • User has no scrollback • Pace of activity can quickly outrun TTS rate • Thus filter activity, including via social rules

29. Other Useful Phone Commands • “Where” and “who” commands • “Summarize” command • Intended for use in non-listening mode • Provides summary of last n minutes of activity • Describes which users have generated most activity • Characterizes interactions via verb usage

30. Sample Transcript

31. A (Very) Small User Study • 5 LambdaMOO users generated 31 dialogues • Averaged 65 turns per dialogue • Some findings: • Great variation in usage styles, verbs invoked • Popularity of “listen” command, often in “radio” mode • Effectiveness of “listen” filtering • Use of grammar command and personal grammars • Evolution of personal grammars to express limitations • ASR problems!

33. The Media Equation: Media = Real Life[Reeves and Nass] • Politeness • Interpersonal distance • Flattery • Personality • Media and Evolution • Lessons for interface design • Turing Test relevance?