Spoken Dialogue for the Why2 Intelligent Tutoring System

1. Spoken Dialogue for the Why2 Intelligent Tutoring System Diane J. Litman Learning Research and Development Center & Computer Science Department University of Pittsburgh Pittsburgh, PA 15260 USA

2. CIRCLE (Center for Interdisciplinary Research on Constructive Learning Environments) is an NSF-funded research center located at Pitt and CMU. CIRCLE's mission is to determine why highly effective forms of instruction, such as human one-on-one tutoring, work so well, and to develop computer-based constructive learning environments that foster equally impressive learning. In the intelligent tutoring system community, research suggests that dialogues which encourage students to do as much of the work as possible, lead to increased learning gains. Why Dialogue Tutoring?

3. Motivation Promote learning gains by enhancing communication richness Benefits for Intelligent Tutoring Systems Naturalness and ease of use New sources of evidence regarding dialogue and pedagogy Complement to current talking heads Hands-free aspect can support multimodal interaction Benefits for Spoken Dialogue Systems Tutoring is a challenging and largely unexplored application Why Spoken Dialogue Tutoring?

4. Spoken dialogue technology has come of age Commercially available products Freely available toolkits A flurry of recent activity Project LISTEN (CMU, reading tutor) Computer-Assisted Language Learning (CMU, pronunciation) Just in time training (NASA) COVE (BBN, shiphandling training) DC-TRAIN (Stanford, post-training critiquing) Arithmetic tutoring (Umass/Rochester, planned) Why Now?

5. What are the advantages – and disadvantages – of using speech over text? Can prosody be used to infer pedagogically significant information? Can the tutoring system make use of such inferences? Testbed - Why2: an existing University of Pittsburgh (deep NLP) text-based dialogue tutoring system Research Questions

6. Why2-Atlas screen shot 4. A rock is thrown straight upward with a velocity v. What is its acceleration at the top of its path? What is its velocity when it comes back to the starting point? Why? Tutor: What is the velocity just before it hits the ground? Student: That will depend on the time that the ball is in the air/distance ball traveled. Tutor: How will it compare with the velocity with which it was thrown up? At the very top of the path, where the velocity is equal to 0, the acceleration will be -9.8 m/(s*s), the acceleration of gravity. It will no longer have any upward acceleration. When it comes back the its starting point (assuming that it is the ground) its velocity will be equal to 0 because its downward progress will be stopped by the ground They will be equal.The height the ball reaches will depend on the initial velocity

7. What can we learn from human-human and wizard system dialogues? How do systems detect when a dialogue is `going wrong'? How do they define such conditions? What factors are the key indicators of `bad' dialogues? Can we devise adaptive strategies to identify patterns of error and respond accordingly? What sorts of behavior do users exhibit when faced with system errors? What measures (better prompts, anticipation of likely error, better help information) can be taken to minimize possible errors? Detecting and Recovering from Problems in Spoken Dialogue

8. Prediction of dialogue and utterance level problems Automatically computable features (prosody, ASR, history, efficiency, quality, lexical) can accurately predict annotatedturn classes (misrecognitions - 93%; aware sites - 89%; corrections -84%) Prosodic features include pitch, energy, duration, pause, tempo, and silence Adaptation Task success improved from 23% to 65% by detecting and reacting to problematic subdialogues My Previous Work (handouts)

9. Example with Annotations S: Hi, this is AT&T Amtrak schedule system. This is TOOT. How may I help you? U: I'd like to find a train from Ch- to Chicago from Baltimore. MISRECOGNITION S: Sorry, I can't understand you. Please repeat your utterance. U: I'd like to find a train from Baltimore to Chicago. MISRECOGNITION, AWARE, CORRECTION S: I heard you say “leave today around 5 thirty in the morning ." Which city do you want to go to? U: I would like to leave Saturday at eight in the morning. MISRECOGNITION, AWARE, CORRECTION S: You can say the arrival city name, such as “New York City." U: Baltimore to Chicago. AWARE, CORRECTION

10. Hypothesis Similar features will prove useful for predicting and adapting to problematic pedagogical situations Challenge What pedagogical situations should (and can) be annotated? Starting Points “Tutorial-level misrecognitions” (e.g., a correct student answer is understood to be incorrect by the system) Problematic affective states (e.g., confusion, boredom, anger, frustration [Evens 2002]) Subjective language (Wiebe) Off-talk Initiative, Dialogue acts / discourse structure Implications for Tutorial Dialogue

11. Neutral July 30 Yes Disappointed/tired No Amused/surprised No Annoyed Yes Late morning (HYP) Frustrated Yes No No, I am … (HYP) There is no Manila... Emotion and Prosody (Shriberg et al. 2001) 3 1 8 2 4 6 5 9 7 10

12. Currently collecting a corpus of human-human, spoken tutoring dialogues Implementation of human-computer system in progress Lots of opportunities for manual and automated dialogue annotation, analysis via machine learning, and incorporation of insights back into the system Status

13. Adding spoken dialogue to tutoring systems provides both opportunities and challenges Expected Contributions Empirical comparisons with text-based tutoring dialogue systems Annotation schemes for dialogue states of potential pedagogical interest Use of prosodic and other features to predict such states Exploitation of such predictions by the tutoring system Summary

14. ITSpoke (or, your idea for a name here) Architecture: Sphinx ASR + Festival TTS + Why2 Demo