Methodology & Basics of Authoring TuTalk Dialogue Agents

1. Methodology & Basics of Authoring TuTalk Dialogue Agents Pamela Jordan University of Pittsburgh Learning Research and Development Center

2. Session Team Members • Pam Jordan • Yue Cui (Jenny) • Moses Hall

3. Agenda • Quick review of what TuTalk is • Methodology for creating dialogue agents • Some lessons learned on authoring computer-mediated dialogues • Cover authoring with TuTalk authoring tool • Basic authoring • Authoring alternative phrasings for responses • Authoring multi-part responses • Next steps for projects

4. What is TuTalk? Dialogue system construction tools for learning applications • Authoring tools for creating, testing and maintaining an artificial dialogue partner/agent • Dialogue agent server that: • Conducts dialogues with multiple students and are either • agent-led (system initiates all topics) or • mixed-initiative dialogues (student or system can initiate topics) • Provides communication protocol for connecting to application specific interface • Provides experiment management tools

5. As a runner pushes a ball away, what horizontal forces act on it? Good! <subdialog> <subdialog > After the push ends, what forces….? Finite state dialogue management {gravitational, *} <anything else> {runner’s}

6. T: Let’s consider the forces acting on objects after they have been thrown. As a runner pushes a ball away, what horizontal forces are acting on it while she is pushing it? S: Gravity? T: In what direction does gravity act? S: downward T: Yes. So is a downward force a horizontal or vertical force? S: It is vertical T: Right. I asked for the horizontal forces instead and the only one is from the runner. After the push ends, what horizontal forces are acting on the ball? Example dialogue Subdialogue

7. What do you have to do to create a dialogue agent? • Write domain content in form of natural language dialogue turns (e.g. elicit or tell) • Write expected student responses (default agent best at short answer) • Write subdialogues for responses that are: • Vague • Incorrect • Overly specific

8. Methodologies for authoring • Corpus-based • Theory-based • Corpus-inspired • Follow any of above methodologies with incremental refinement

9. Corpus-based authoring • Collect corpus of humans interacting on task • Computer mediated • Non-interruptible turns • Analyze for goals/topics & adjust for learning objectives • Analyze goals/topics identified for student responses, look for answer categories of: • Partially correct/incomplete • Partially incorrect • Overly vague • Overly specific • Correct but premature • Identify tutor tactics for each answer category • Analyze language used in student responses to support authoring of response concepts

10. Identify tutoring tactics (from ProPl)

11. Form tactics • Pump: can you say more about X? • Hint & reask: fill in a possible missing piece then try again • Socratic: lead through line of reasoning • Simulation: lead through an example & abstract • For additional ones, see chapters 7 & 8 of Evens & Michael (2006), One-on-One Tutoring by Humans and Computers

12. Applying tactics in ProPl

13. Analyze student language (from ProPl) Use to define response concepts Strategy: pick a minimal set of key words that will distinguish between responses

14. Theory-based authoring • Similar to corpus-based in that • Based on theories about domain & learning • But skips corpus collection and analysis phase • Examples of theoretical conceptual tactics: • Definitions & applications of concepts (e.g. distinguish technical & lay senses of terms) • Conceptual variant of a domain principle (e.g. boundary conditions) • Variant of problem

15. Corpus-inspired authoring • Combination of corpus-based & theory-based • Locate related corpus • Identify theoretical goals & refine based on those goals that can find in corpus • Identify theoretical expected responses & refine relative to responses find in corpus

16. Incremental refinement • Author main-path dialogues w/ correct answers • Refine by adding answer categories: • Partially correct/incomplete • Partially incorrect • Overly vague • Overly specific • Correct but premature • Author response to answer categories & attach to answer categories • Pilot dialogues • Analyze logs & refine authored dialogues

17. Some lessons learned on computer-mediated dialogues • Students prone to “refusal to answer”: I don’t know, who cares • Don’t always bottom out • Prod to try • Avoid interrogation: • remember coherency w/ short recaps, turn and topic transitions, • make some abstractions, meta-info explicit • Assess understanding: • Avoid explicit “do you understand” • Use trick questions, after success check strength of assertion • Are you sure? • What other forces (when answer is no more) • Don’t be interactive for sake of being interactive but for sake of adapting to individual • Interact in order to diagnose what the student needs! • Dialogue slow if cover everything

18. Basic authoring • Create templates to address/cover goals • Create concepts to use in templates • used to recognize student responses/turns • used to generate tutor turns/responses • Template for a goal has one to many steps • Step is an • Initiation that is optionally paired with an expected Response • Can be a pointer to another recipe

19. Examples of concepts (abstract) ask_share_appetizer [So, should we share an appetizer?] [I’d like to share an appetizer. What looks good to you?] skip_appetizer [I don’t want an appetizer] [Let’s skip the appetizer]

20. Example template for a dialogue covering a goal (abstract) Goal name Goal: select-appetizer step:enthuse_about_appetizers step:ask_share_appetizer [agree_to_share_appetizer] [skip_appetizerabort, ask-soup] [unknownabort, loose-temper] step:agree-on-appetizer Concept to realize or recognize initiation possible responses Response action: push to subdialogue for this goal Push to subdialogue for this goal

21. The need for alternative phrasings for concepts • The language recognizer uses simple technique of minimum edit distance • For each concept (a set of alternative NL phrases) in all responses for the current question, find the minimum edit distance for the student input • Minimum edit distance is minimum number of “edits” (insertions/deletions) needed to student response for it to match one of phrases in concept definition • Find concept with smallest minimum edit distance • If that edit distance is within threshold (default of < .5) then take that concept as the response • Else the student response is not recognized so follow the unanticipated response path • Strategy for authoring a response: pick a minimal set of key words that will distinguish between responses for a question

22. Extending concept definitions

23. Extending concept definitions

24. Multi-part responses • Gives student credit for partial responses and seeks just what is missing: • Example: T; What are the forces on a set of keys on top of a table? S: gravity T: Almost. There is another force. The keys aren’t moving so the net force must be zero. What force balances out the force due to gravity? S: normal force

25. Multi-part responses

26. Next steps for projects • Identify dialogue goals • Search for example of those goals in a corpus or • Write down some ideas for how to cover the goals and engage someone in such a dialogue • Try authoring some of those goals with the authoring tool