Speech and Language Processing for Adaptive Training

Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development Center University of Pittsburgh

Outline • The State of the Art: A Brief Survey • Adaptation to Student Uncertainty in Tutorial Dialogue: A Case Study • ITSPOKE: System and Corpora • Uncertainty Detection • System Adaptation • Experimental Evaluation • Summing Up

What is Natural Language Processing? • “The goal of this new field is to get computers to perform useful tasks involving human language, tasks like enabling human-machine communication, improving human-human communication, or simply doing useful processing of text or speech.” [Jurafsky and Martin 2008] • Many names and facets • Speech and Language Processing • Human Language Technology • Computational Linguistics

Relevance for Adaptive Training • Knowledge of Language is often needed to • trigger adaptation • personalize training, using the enormous amount of machine-readable text and audio that is now available • Conversational Agents are becoming an important form of human-computer interaction

Phonetics and Phonology: speech sounds Morphology: words and their internal composition Syntax: the structuring of words into larger units Semantics: the meaning of words and larger units Pragmatics: interpretation in situational context Discourse: interpretation in context of previous utterances Knowledge of Language

Computational Models (and Associated Algorithms) • State Machines • Formal Rule Systems / Grammars • Logic-Based Formalisms • Models of Uncertainty

A Brief Survey of Applications NLP Applications to Education

A Brief Survey of Applications NLP Applications to Education Learning Language (reading, writing, speaking) Tutors Scoring

A Brief Survey of Applications NLP Applications to Education Learning Language (reading, writing, speaking) Using Language (to teach everything else) Tutors Conversational Tutors / Peers Scoring CSCL

A Brief Survey of Applications NLP Applications to Education Learning Language (reading, writing, speaking) Processing Language Using Language (to teach everything else) Readability Tutors Conversational Tutors / Peers Questioning & Answering Scoring CSCL Discourse Coding Lecture Retrieval

Tutorial Dialogue Systems • Why is one-on-one tutoring so effective? “...there is something about discourse and natural language (as opposed to sophisticated pedagogical strategies) that explains the effectiveness of unaccomplished human [tutors].” [Graesser, Person et al. 2001] • Currently only humans use full-fledged natural language dialogue

SpokenTutorial Dialogue Systems • Most human tutoring involves face-to-face spoken interaction, while most computer dialogue tutors are text-based • Can the effectiveness of dialogue tutorial systems be further increased by using spoken interactions?

Potential Benefits of Spoken Dialogue: I • Conversation provides a learning environment that promotes student activity • Self-explanation correlates with learning and occurs more in speech

Potential Benefits of Spoken Dialogue: II • Speech contains prosodic information, providing new sources of information about the student for adaptation • A correct but uncertain student turn • ITSPOKE: How does his velocity compare to that of his keys? • STUDENT: his velocity is constant

Potential Benefits of Spoken Dialogue: III • Spoken computational environments may foster social relationships that may enhance learning

Potential Benefits of Spoken Dialogue : IV • Some applications inherently involve spoken language • Conversational skill training • Others require hands-free interaction • e.g., NASA

Back-end is Why2-Atlas system [VanLehn, Jordan, Rose et al. 2002] • Sphinx2 speech recognition and Cepstral text-to-speech

Three Types of Tutoring Corpora • Human Tutoring • 14 students / 128 dialogues (physics problems) • 5948 student turns, 5505 tutor turns • Computer Tutoring • 77 students / 385 dialogues • both synthesized and pre-recorded tutor voices • Wizard /Computer Tutoring • 81 students / 405 dialogues • human performs speech recognition, semantic analysis • computer performs dialogue management

Experimental Procedure • College students without physics • Read a small background document • Took a multiple-choice Pretest • Worked 5-10 problems (dialogues) with tutor • Took an isomorphic Posttest • Goal was to optimize Learning Gain • e.g., Posttest – Pretest

Standard Empirical Detection Methodology • Manual annotation of user states that will trigger system adaptation • Naturally-occurring spoken dialogue data • Prediction via machine learning • Use speech and language processing to automatically extract features from user turns • Use extracted features and annotations to learn a model for predicting user state(s) in new data • Significant reduction of baseline error

Example Features • What a user says • words (speech recognition), stems (morphology) • part-of-speech, syntactic constituents (parsing) • correctness (semantic analysis) • dialogue moves (pragmatics and discourse) • How a user says it • acoustic-prosodic analysis

Extracting Pitch Features

Extracting Energy Features

Temporal Features • Duration = end time - begin time • Tempo (speaking rate) = #syllables/duration

Detecting Neg/Pos/Neu in ITSPOKE • Baseline Accuracy via Majority Class Prediction

Detecting Neg/Pos/Neu in ITSPOKE • Use of prosodic (sp), recognized (asr) and/or actual (lex) lexical features outperforms baseline

Detecting Neg/Pos/Neu in ITSPOKE • As with other applications, highest predictive accuracies are obtained by combining multiple feature types

System Adaptation: How to Respond? • Our initial focus: responding to student uncertainty • Most frequent user state in our data • Focus of other studies • .62 Kappa • Approaches to adaptive system design • Theory-based • Data-driven

Theory-Based Adaptation:Uncertainty as Learning Opportunity • Uncertainty represents one type of learning impasse, and is also associated with cognitive disequilibrium • An impasse motivates a student to take an active role in constructing a better understanding of the principle. [VanLehn et al. 2003] • Astate of failed expectations causing deliberation aimed at restoring equilibrium. [Craig et al. 2004] • Hypothesis: The system should adapt to uncertainty in the same way it responds to other impasses (e.g, incorrectness)

Data-Driven Adaptation: How Do Human Tutors Respond? • An empirical method for designing dialogue systems adaptive to student state • extraction of “dialogue bigrams” from annotated human tutoring corpora • χ2analysis to identify dependent bigrams • generalizable to any domain with corpora labeled for user state and system response

Example Human Tutoring Excerpt S: So the- when you throw it up the acceleration will stay the same? [Uncertain] T: Acceleration uh will always be the same because there is- that is being caused by force of gravity which is not changing. [Restatement, Expansion] S: mm-k. [Neutral] T: Acceleration is– it is in- what is the direction uh of this acceleration- acceleration due to gravity? [Short Answer Question] S: It’s- the direction- it’s downward. [Certain] T: Yes, it’s vertically down. [Positive Feedback, Restatement]

Findings • Statistically significant dependencies exist between students’ state of certainty and the responses of an expert human tutor • After uncertain, tutorBottoms Out and avoids expansions • After certain, tutor Restates • After mixed, tutorHints • After any non-neutral, tutor increases Feedback • Dependencies suggest adaptive strategies for implementation in computer tutoring systems

Manipulate tutor responses to student uncertainty and investigate impact on learning and efficiency Experimental-Basic: treat all uncertain turns as incorrect (theory) Experimental-Empirical: for uncertain or incorrect turns, provide original content but vary dialogue act (human tutor analysis) Control-Norm: ignore uncertainty (as in original system) Control-Random: ignore uncertainty, but treat a percentage of random correct answers as incorrect (to control for additional tutoring) Experimental Design: 4 Conditions

TUTOR: Now let’s talk about the net force exerted on the truck. By the same reasoning that we used for the car, what’s the overall net force on the truck equal to? STUDENT: The force of the car hitting it? [uncertain+correct] TUTOR (Control-Norm):Good [Feedback] … [moves on] TUTOR (Experimental-Basic): Fine. [Feedback] We can derive the net force on the truck by summing the individual forces on it, just like we did for the car. First, what horizontal force is exerted on the truck during the collision? [Remediation Subdialogue] Same tutor response if student had been incorrect Treatments in Different Conditions

Modified version of ITSPOKE Dialogue manager adapts to uncertainty system responses based on combined uncertainty and correctness Full automation replaced by some Wizard of Oz (WOZ) components human wizard recognizes student speech human also annotates uncertainty and correctness provides upper-bound speech and NLP performance Platform: Adaptive WOZ-TUT System

WOZ-TUT Screenshot

20-21 subjects in each condition Native English speakers with no college physics Procedure: 1) read background material, 2) took pretest, 3) worked training problem with WOZ-TUT, 4) took user Brief Survey, 5) took posttest Experimental Procedure

Experimental Results Two-way ANOVA indicated students learned (F(1,77) = 271.214, p = 0.000, MSe = 0.009) Amount depended on condition (F(3,77) = 3.275, p = 0.025, MSe = 0.009) One-way ANOVA with post-hoc Tukey tests determined which conditions learned more

In Addition… • Learning Efficiency also improved • Two Efficiency Measures • (Normalized Learning Gains) / (Total Student Turns) • (Normalized Learning Gains) / (Total Time in Minutes) • Experimental-Basic > Control-Norm (p < .05) • Current Directions • New evaluation of Experimental-Basic • fully-automated ITSPOKE • New methods for designing Experimental-Empirical • educational data mining using reinforcement learning • Other student states

Summing Up: I • Spoken Dialogue Systems for Adaptive Training • Natural language dialogue is a key aspect of human one-on-one training • Using presently available technology, successful conversational computer training environments are now being built • Evidence that more adaptive versions of such systems will further enhance performance

Summing Up: II • Adaptive Training in turn provides many other opportunities and challenges for researchers in Speech and Natural Language Processing

Speech and Language Processing for Adaptive Training