Spoken dialog

1. Spoken dialog

2. Lexical & syntactic entrainment In dialog, people tend to re-use the same words and sentence structure. Socrates: Please select command mode. Student: Please find an author named Octavia Butler Socrates: Invalid Folio command: Please

3. Two people in search of a perspective A: ah boy this one ah boy alright it looks kinda like, on the right top there's a square that looks diagonal B: uh huh A: and you have sort of another like rectangle shape, the like a triangle, angled, and on the bottom it's ah I don't know what that is, glass shaped B: alright I think I got it A: it's almost like a person kind of in a weird way B: yeah like like a monk praying or something A: right yeah good great B: alright I got it

4. (Later, same card) Trial 2 (~ 11 cards later) B: 9 is that monk praying A: yup Trial 3 (~ 11 cards later) A: number 3 is the monk B: ok

5. Referring expressions become much shorter and more efficient upon re-use • This does not happen when people speak into a tape recorder! (where there is no feedback from an addressee) • This happens because people acquire common ground.

6. Grounding in interaction: • the evidence people give each other that they understand so far • This evidence allows them to coordinate and stay “in sync”. • If one person doesn’t provide enough evidence, the other one may seek evidence out (take initiative).

7. A: ah boy this one ah boy alright it looks kinda like, on the right top there's a square that looks diagonal B: uh huh A: and you have sort of another like rectangle shape, the like a triangle, angled, and on the bottom it's ah I don't know what that is, glass shaped B: alright I think I got it A: it's almost like a person kind of in a weird way B: yeah like like a monk praying or something A: right yeah good great B: alright I got it

8. Variability in language use "a bat" "the candle" "the anchor" "the rocket ship" "the Olympic torch" "the Canada symbol" "the symmetrical one" "shapes on top of shapes" "the one with all the shapes" "the bird diving straight down" "the airplane flying straight down" "the angel upside down with sleeves" "the man jumping in the air with bell bottoms on”

9. The Vocabulary Problem Exercise: Choose a name for a program that provides info about recreational activities in a major metropolitan area. Make this name intuitively obvious, one that other people would be able to remember or guess.

10. Words index perspectives. There’s no such thing as a synonym! Pushy, according to you. Assertive, according to me!

11. The Vocabulary Problem Probability of two people generating the same name for a text-editing command: # of commands in editor Probability 5 7% 25 11% (Furnas, Landauer, Gomez, & Dumais, 1987)

12. The Vocabulary Problem If you provide as many as 20 synonyms for an item’s name, then on average only 80% of people would guess correctly. Not only does this not solve the problem, but it actually can make it worse....

13. The Vocabulary Problem Probability that 2 people using the same name mean the same command by it: # of commands in editor Probability 5 41% 25 15% (Furnas, Landauer, Gomez, & Dumais, 1987)

14. The Vocabulary Problem The problem is even worse for speech and natural language dialogs than it is for command dialogs!

15. “the car” “the snazzy red car” “your Maserati”

16. The term you choose matters! The mapping of term to meaning is made collaboratively and is not the same for all speakers - it depends on common ground.

17. Entrainment: • Partners in conversation tend to converge on the same terms to show that they believe they’re talking about the same thing. • When a speaker abruptly uses a different term, the addressee assumes she means something different

18. A premise: • People transfer some (but not all) of their expectations about human dialog to human-computer dialog. • They expect consistency in terminology. • They expect turns to be connected. • They expect partners to keep track of what they just discussed. • They need to be able to repair problems.

19. Revisit: Adjacency Pairs A: Are you coming tonight? B: Can I bring a guest? A: Male or female? B: What difference does that make? A: An issue of balance. B: Female. A: Sure. B: I’ll be there.

20. Error messages should provide specific information that helps people repair the problem! (Examples from HPNL, a natural language database query program)

21. Revisit: Answering questions Show me Starry Night. > Here is Starry Night. (shows picture) Are there any self-portraits in the Museum of Modern Art? > Yes, there are 6 self-portraits in the Museum of Modern Art, namely, <list>

22. Where Van Gogh paint Starry Night? > Please try rephrasing that. Where is Strry Night? > “Strry”? That’s an unknown word. How large is Van Gogh? > Sorry, that’s not in the database.

23. Show me a chart of Van Gogh’s works. > How do you want that? - by when they were created? - by where they were created? - by current location?

24. Problems are inevitable! • To initiate a repair, the user needs to first recognize the problem • Neither the user nor the computer is omniscient - they each have to rely on the evidence they have about what is going on. • Positive evidence (that things are on track) is just as important as negative evidence (or error messages)!

25. Give error messages at the level at which the system “breaks”, so that the user can interrupt or figure out how best to recover.

26. Susan: you don’t have any nails, do you? Bridget: <pause> fingernails? Susan: no, nails to nail into the wall. <pause> when I get bored here I’m going to go put up those pictures. Bridget: no.

27. What B knows: A: you don’t ..? B: (pause) fingernails? What A knows: A: you don’t ..? B: Built-in asymmetry: One partner notices a problem before the other does.

28. Built-in asymmetry: One partner notices a problem before the other does. Susan: you don’t have any nails, do you? Bridget: <pause> fingernails? Susan: no, nails to nail into the wall. <pause> when I get bored here I’m going to go put up those pictures. Bridget: no.

29. Clark & Schaefer’s model An addressee, B, can be in different states w/ respect to a speaker’s (A’s) utterance, U State 0: B didn’t notice that A uttered any U State 1: B noticed U. State 2: B correctly heard U. State 3: B understood what A meant by U.

30. C&S’s model extended(Brennan & Hulteen, 1995) State 0: Not attending State 1: Attending (but not hearing) State 2: Hearing(but not parsing) State 3: Parsing (but not interpreting) State 4: Interpreting (but not intending) State 5: Intending (but not acting) State 6: Acting (but not reporting) State 7: Reporting

31. Different error messages at each level: “Call Eric” State 1: What was that again? State 2: Sorry, I didn’t get that - call who? State 3: Do you mean Eric Hulteen or Eric Stubbs? State 4: Sorry, I don’t know how to reboot Eric. State 6: I can’t call Eric - the phone line is down. State 7: Eric’s line is busy. Shall I keep trying?

32. Implications for HCI • Give error messages at the level at which the system “breaks”, so that the user can interrupt or figure out how best to recover. • Positive feedback (that things are going well) is just as important as negative feedback (error messages)!

33. Positive feedback at each level: State 1: I’m still listening (or: an alert face) State 2: I heard something (or: face changes, etc.) State 3: I heard you say Call Eric. State 4: Do you want me to Call Eric? State 5: OK, I’m calling Eric. State 6: (touch-tone dialing sounds) State 7: (Eric answers the phone) Hello?

34. Set a grounding criterion • If certain errors are common (e.g., recognition), then give default feedback at that level. I heard you say call Eric. (Level 3) • If certain errors are destructive (e.g., calling the wrong person), then set the grounding criterion higher: Do you want me to call Eric? (Level 4)

35. Adaptive feedback(Brennan & Hulteen, 1995) Select the level for positive feedback depending on: • the task model (is a command risky?) • a model of the user (expertise, preferences) • a model of how the dialog is going so far

36. Repair strategies in dialogue • Conversation isn’t just passing messages back and forth • In conversation, people ground their utterances - that is, they seek and provide evidence about what they understand. • This can take different forms… Consider these example dialogues from an automated airline reservation system (from Voice User Interface Design by Cohen, Giangola, & Balogh, 2004)

37. Dialogue with confirmations What’s the departure city? • San Francisco. San Francisco. Is that correct? • Yes. To what city? • New York. New York. Is that right? • Yes. Great. What day do you want to leave? • I wanna leave March fifth. March fifth. Is that correct? • Yes. (Cohen et al., 2004, p. 223)

38. Grouped confirmations What’s the departure city? • San Francisco. To what city? • To New York. And on what date? • I’m leaving on March fifth. Now let me confirm that: San Francisco to new York on March fifth. Is that correct? • Yes. (Cohen et al., 2004, p. 223)

39. Failed error recovery • Order status To better assist you in checking the status of your order, we need your order number. Please say your order number. • I don’t know what it is. I’m sorry, I didn’t understand. Please say or key in your 8-digit order number. You can say, “Help” for more details. • Help. OK, here’s some help. In order to assist you, we need your order number. Please speak naturally and say the 8 digits of your order number. You can also enter them on your touchtone keypad. • <hangs up> (Cohen et al., 2004, p. 225)

40. Prosody and previous information System: I heard "Tennessee." Is that RIGHT? Caller: No. System: What about "Texas"? Is THAT right? (Cohen et al., 2004)

41. Prosody and previous information System: I heard "Tennessee." Is that RIGHT? Caller: No. System: What about "Texas"? Is THAT right? (Cohen et al., 2004) New information is stressed; given information is de-stressed.

42. Prosody and given/new information Caller: Bill payment. System: What bill would you like to PAY? Caller: uhhh… MasterCard. System: How much would you like to PAY? Caller: Two hundred and fifteen dollars. System: On what date would you like it PAID? (Cohen et al., 2004)

43. Prosody and given/new information Caller: Bill payment. System: Sure, bill payment. What bill would you like to PAY? Caller: MasterCard? System: How MUCH would you like to pay? Caller: Two hundred and fifteen dollars. System: And on what DATE would you like it paid? (Cohen et al., 2004)

44. More on handling errors

45. Cohen et al., Ch. 13, #3 System:What's the flight number? Caller: Sixteen fifty-five. System:Now is this for arrival or departure information? Caller: Arrival. System:Let me confirm that: Flight 6355 arriving on- Caller: No. System:Sorry, which part should I change, the flight number or the date? Caller: The flight number.... System:What's the flight number? Caller: Sixteen fifty-five. System:Let me confirm that: Flight 6355 arriving on- Caller: No! <hang up>

46. Cohen et al., Ch. 13, #4 Caller: Order status. System:To better assist you in checking the status of your order, we need your order number. Please say your order number. Caller: I don't know what it is. <reject> System:I'm sorry, I didn't understand. Please say or key in your eight-digit order number. You can say, "Help" for more details. Caller: Help. System:Okay, here's some help. In order to assist you, we need your order number. Please speak naturally and say the eight digits of your order number. You can also enter them on your touchtone keypad. Caller: <hang up>

47. Cohen et al., Ch. 13, #5 System:What's the flight number? Caller: . . . <reject> System:What was that? Caller: <reject> System:I still didn't get it. Let's try another approach. What's the origin city of the flight? Caller: Boston. System:And what's the destination? Caller: New York, Kennedy Airport. System:Okay. And what time is it due? Caller: Around noon. System:Got it. American Airlines flight 135 will be arriving at JFK on time at 12:15 PM at gate 3.

48. Cohen et al., Ch. 13, #6 System: What's the date of travel? Caller: Um, I gotta get there by next Saturday. <reject> System: I'm sorry, I didn't understand. Please say the date of travel-for example, "January eighth" or "April twenty-third."