Methodology for Authoring Dialogues

1. Methodology for Authoring Dialogues Pamela Jordan University of Pittsburgh Learning Research and Development Center

2. Agenda • Methodology for authoring dialogues • Some lessons learned on authoring computer-mediated dialogues • Next steps for projects & discussion

3. Authoring preparation methodologies • Corpus-based • Theory-based • Corpus-inspired • Incremental refinement

4. Corpus-based authoring • Collect corpus of humans interacting on task • Computer mediated • Non-interruptible turns • Analyze for goals/topics & adjust for learning objectives • Analyze goals/topics identified for student responses, look for answer categories of: • Partially correct/incomplete • Partially incorrect • Overly vague • Overly specific • Correct but premature • Identify tutor tactics for each answer category • Analyze student language

5. Tutoring tactics in ProPl

6. Form tactics • Pump: can you say more about X? • Hint & reask: fill in a possible missing piece then try again • Socratic: lead through line of reasoning • Simulation: lead through an example & abstract • For additional ones, see chapters 7 & 8 of Evens & Michael (2006), One-on-One Tutoring by Humans and Computers

7. Applying tactics in ProPl

8. ProPL student language analysis • Use to define response concepts • Strategy: pick a minimal set of key words that will distinguish between responses

9. Authoring preparation methodologies • Corpus-based • Theory-based • Corpus-inspired • Incremental refinement

10. Theory-based authoring • Based on theories about domain/task & learning • Examples of theoretical conceptual tactics: • Definitions & applications of concepts (e.g. distinguish technical & lay senses of terms) • Conceptual variant of a domain principle (e.g. boundary conditions) • Variant of problem

11. Authoring preparation methodologies • Corpus-based • Theory-based • Corpus-inspired • Incremental refinement

12. Corpus-inspired authoring • Combination of corpus-based & theory-based • Locate related corpus • Identify theoretical goals • Search for some of those goals within a corpus & refine relative to what can find • Identify theoretically expected student responses • Refine relative to those response instances can find in corpus

13. Authoring preparation methodologies • Corpus-based • Theory-based • Corpus-inspired • Incremental refinement

14. Incremental refinement • Author main-path dialogues w/ correct answers • Refine according to answer categories • Author responses to answer categories • Pilot dialogues • Analyze logs & refine authored dialogues

15. Agenda • Methodology for authoring dialogues • Some lessons learned on authoring computer-mediated dialogues • Next steps for projects & discussion

16. Recognizing student responses • Language recognizer uses simple technique of minimum edit distance • Minimum edit distance is smallest number of “edits” (insertions/deletions) needed for student response to match a response phrase • For each set of alternative NL phrases (concept) for all responses for the current question, find the minimum edit distance • Select concept with smallest minimum edit distance • If that edit distance is within threshold (default of < .5) then select that concept as the response • Else the student response is not recognized, so follow the unanticipated response path • Beginnings & endings of unmatched parts of responses are not penalized • Stop words (e.g. of, the) are not penalized • Strategy for authoring a response: pick a minimal set of key words that will distinguish between responses for a question

17. Advice on computer-mediated dialogues • Students prone to refusal to answer e.g., “I don’t know”, “who cares” • Don’t always bottom out • Prod student to try (e.g. “Make your best guess”) • Avoid interrogation: • remember to address coherency; include short recaps, turn and topic transitions, • make some abstractions, meta-information explicit e.g., “Let’s break it down some more”, “First, we’ll identify the givens”. • Assess understanding: • Avoid explicit “do you understand?” • Use trick questions; after success check strength of assertion • “Are you sure?” • “What other forces are there?” (when answer is no more) • Don’t be interactive just for sake of being interactive, instead use it to adapt to individual • Interact in order to diagnose what the student needs • Dialogue slow if cover everything; figure out what can be skipped

18. Agenda • Methodology for authoring dialogues • Some lessons learned on authoring computer-mediated dialogues • Next steps for projects & discussion

19. Next steps for projects • Look at dialogue samples/corpus for yur project and identify goals to cover in dialogue • available corpora: http://andes3.lrdc.pitt.edu/TuTalk/corpora/ • For each goal author main path with only correct responses and unanticipated response follow-ups

20. Discussion & questions • Describe your projects • What help/advice do you anticipate needing?