Why2 Tutors that teach mental models using natural language dialog

Why2Tutors that teach mental models using natural language dialog Kurt VanLehn & the Natural Language Tutoring Group at the University of Pittsburgh Art Graesser & the Tutoring Research Group at the University of Memphis

1. Can natural language tutorial dialog add pedagogical value? • Expert Human tutors • Computer mediated communication in print • Spoken computer mediated communication • Computer tutors • Why2/AutoTutor • Why2/Atlas • Reading text • Reading textbook • Focused lectures

2. Can deep linguistic and dialog techniques add pedagogical value?

3. How feasible is a deeplinguistic tutoring system? • Existing deep dialogue systems (e.g., Trains) are not applied • Existing applied dialogue systems (e.g., call centers) are not deep • Tutoring is not easy • Non-trivial knowledge (i.e., worth teaching) • Vague, incomplete, ill-formed language • Operate in real time

Task domain: Writing qualitative physics explanations • Why explanations? • Generating explanations > studying explanations • But coaching and feedback may be necessary • Why conceptual physics? • Inert knowledge (e.g., F=m*a) may exist already but may be difficult to apply to problems • Correct principles can be identified objectively for problems, rather than being unconstrained • Subject matter is verbal and requires some analytical precision, but is not mathematics • Persistent misconceptions

An example essay Question: Suppose you are running in a straight line at constant speed. You throw a pumpkin straight up. Where will it land? Explain why. Student’s explanation: Once the pumpkin leaves my hand, the horizontal force that I am exerting on it no longer exists, only a vertical force (caused by my throwing it). As it reaches it’s maximum height, gravity (exerted vertically downward) will cause the pumpkin to fall. Since no horizontal force acted on the pumpkin from the time it left my hand, it will fall at the same place where it left my hands.

Evaluation framework • Pretest • Training:For each problem/question, do: • Student enters initial explanation • Tutor and student discuss it • Student enters final explanation • Tutor presents ideal explanation • Posttest Only this varies with the condition

How might better discussion increase learning? • Better NLU => better dialogue goals • Better misconception detection • Doesn’t adopt remediation goal for correct statements • Better ways to achieve dialogue goals • Increase student explanation by better hints, prompts… • Managing complexity by postponing new goals • Maintain engagement

Research plan • Develop tutoring systems with natural language dialog • Why2/AutoTutor • Why2/Atlas • Develop several control conditions • Human tutors • Reading text • Doing nothing • Compare computer tutors and controls • Hypothesis: learning is proportional to linguistic and dialog competence • Conduct research on computational models and mechanisms of tutorial dialog

Broad Outline • Evaluation (60 min) • Why2-AutoTutor R&D (60 min) • Coffee break (30 min) • Why2-Atlas R&D (60 min)

Next Evaluation Outline • Tutors • Human tutoring (Pam Jordan) • Why-Atlas (Pam) • Why-AutoTutor (Art) • Lecturer (Kurt) • Experiments • 1 (Kurt) • 2 (Art) • 3 (Tanner Jackson) • 4 (Pam) • 5 (Kurt) • Discussion (Kurt)

Student’s screen for human tutoring Problem Dialogue history Student’s essay Student’s turn in the dialogue

Example (slide 1 of 4) • Suppose you are running in a straight line at constant speed. You throw a pumpkin straight up. Where will it land? Explain. • Essay: The pumpkin will land behind me Problem Student’s explanation

Example (slide 2 of 4) • Tutor: Please give reasons for your answer. • Student: The pumpkin will land behind me because the gravitational force will cause the pumpkin to land behind the position where I threw it Some “explanations” are incomprehensible even for us

Example (slide 3 of 4) • Tutor: Let us argue systematically. What was the pumpkins velocity just bfore it was thrown? • Student: 0 m/s Ignores “explanation.” Starts long line of reasoning with a short-answer question wrong

Example (slide 4 of 4) [Many lines omitted] • Tutor: Ok. If you have no questions I shall show you the expert essay? • Student: ok • Tutor: The man holding the pumpkin is running with constant horizontal velocity. In order to throw the pumpkin vertically up, the man will have to exert an upward vertical force on the pumpkin such that the net vertical force on it causes it to accelerate upward and …[20 more lines] • Student: ok Ideal essay

Student’s screen for Why-Atlas Problem Dialogue history Student’s essay Student’s turn in the dialogue

Why2-Atlas dialogue: outer loop • Identify flaws in the student’s essay • Misconceptions • Missing expectations (= points that a good essay is expected to cover) • Discussion: • Select a flaw • Select a KCD (knowledge construction dialogue) plan • Run it • Invite student to fix the essay • Repeat until essay is flawless

Why2-Atlas dialogue inner loop: running a KCD plan • A KCD plan has a main line of reasoning • Each tutor turn ends in a question • Branch on student’s response: • Correct => positive feedback, may restate answer • Recognized incorrect or partial => specific remediation • Otherwise => generic remediation • Repeat until main line of reasoning is finished

Example (slide 1 of 4) • Suppose a man is running in a straight line at constant speed. He throws a pumpkin straight up. Where will it land? Explain. • Essay: The pumpkin would land in front of me because of the horizontal speed that is exerting on the pumpkin.

Example (slide 2 of 4) • Tutor: I see some opportunities for growth. Let's chat. So let's discuss a point that seems to be missing in your essay. Can you tell me what forces are acting on the pumpkin after it is released? • Student: Gravity force Picks a missing expectation & starts KCD Correct, so goes to next step of the KCD

Example (slide 3 of 4) • Tutor: Good. Since we know that gravity is the only force acting on the pumpkin, let me ask you this. As the gravitational force pulls the pumpkin down, does it also pull the pumpkin horizontally a little bit? • Student: Yes. • Tutor: Let's look at this more closely. In which direction does the earth's gravitational force act? • Student: Vertical • Tutor: Let's consider this some more. Is that vertical upwards or vertical downwards? • Student: Vertical downwards Wrong, so subdialogue Incomplete, so subdialogue

Example (slide 4 of 4) • [many lines omitted] • Tutor: Fine. I think this is an important point to include. See if you can add some sentences that convey this. Modify your essay please. • Essay: Since there is no gravity acting on the pumpkin vertically downward, then there is no force acting on the pumpkin when it is in the air. So the pumpkin does not land horizontally, but it lands in the same direction that i threw it up at. • Tutor: Terrific. Although we're making progress, I can see a few more things I'd like to discuss…

Talking head • Gestures • Synthesized speech • Presentation of the question/problem Student input (answers, comments, questions) • Dialog history with • tutor turns • student turns

Distinctive features ofWhy2-AutoTutor • Animated agent • Attempt to have mixed initiative dialogue rather than a coached construction of an essay • Final request to generate essay in a single turn rather than ongoing modification of an essay • Accommodates spontaneous student questions (which they rarely ask) • Prompts for one student question (per problem) • Asks one diagnostic question about a misconception (per problem)

Managing One AutoTutor Turn • Short feedback on the student’s previous turn • Advance the dialog by one or more dialog moves that are connected by discourse markers • End turn with a signal that transfers the floor to the student • Question • Prompting hand gesture • Head/gaze signal

Dialog Moves • Positive immediate feedback: “Yeah” “Right!” • Neutral immediate feedback: “Okay” “Uh huh” • Negative immediate feedback: “No” “Not quite” • Pump for more information: “What else?” • Hint: “How does tossing the pumpkin affect horizontal velocity?” • Prompt for specific information: “Vertical acceleration does not affect horizontal _______.” • Assert: “Vertical acceleration does not affect horizontal velicity.” • Correct: “Air resistance is negligible” • Repeat: “So, once again, how does tossing the pumpkin affect horizontal velocity?” • Summarize: “So to recap, [succinct summary].”

Cycle fleshes out one expectation at a time Exit cycle when: LSA-cosine(S, E ) > T S = student input E = expectation T = threshold Hint-Prompt-Assertion Cycles to Cover Good Expectations Hint Prompt Assertion Hint Prompt Assertion

The Lecturer • Intended as control condition • Monologue not dialogue • Everything else is the same • Content is the same

What the student does • Read the problem • Write an initial explanation/essay • Read several minilessons (text only) • Revise the explanation/essay • Study the ideal essay

Controlling for content • Each KCD converted to a minilesson • T: What direction is the gravitational force?S: Straight down • => The direction of the gravitational force is straight down • Ignoring the student’s essay, the Lecturer presents every relevant minilesson • Student may not read (or reread) it.

Example (slide 1 of 4) • Suppose you are running in a straight line at constant speed. You throw a pumpkin straight up. Where will it land? Explain. • Essay: Once the pumpkin leaves my hand, the horizontal force that I am exerting on it no longer exists, only a vertical force (caused by my throwing it). As it reaches it's maximum height, gravity (exerted vertically downward) will cause the pumpkin to fall. Since no horizontal force acted on the pumpkin from the time it left my hand, it will fall at the same place where it left my hands.

Example (slide 2 of 4) • Lecturer: Here is an important point that a good explanation should cover. The relationship between the velocity of the runner and the velocity of the pumpkin which the runner is holding, before the runner releases the pumpkin, is that the two velocities are the same. As an example let's consider in a different scenario the relationship between the velocities of two objects that move together. • Student Click

Example (slide 3 of 4) • Lecturer: A woman is carrying a suitcase through an airport. The suitcase does not move relative to the woman's body. Therefore, over any interval of time, the displacement of the woman compared to the displacement of the suitcase is equal to the displacement of the suitcase. • Student Click • Lecturer: Velocity is defined as the displacement vector over a time interval divided by that time interval. Thus, the relationship between the velocity of the woman and the velocity of the suitcase is that the two velocities are the same.

Example (slide 4 of 4) • [Many lines omitted] Lecturer: Please revise your essay now. • Essay: As I am running with the pumpkin, the velocity of the pumpkin is equal to my velocity. …. • Student Click • Lecturer: Now that you've finished your essay, here is an essay that covers everything needed to fully answer the problem. Student edits initial essay => final essay Ideal Essay goes here

Similar format for all 4 conditions • Pretest • Training:For each problem/question, do: • Student enters initial explanation • Tutor and student discuss it • Student enters final explanation • Tutor presents ideal explanation • Posttest Only this varies with condition But same expectations & misconceptions per essay

Experiment 1 details • Participants had college physics • 10 Training problems • Tests (two equivalent forms) • 40 multiple choice questions • 4 essay questions • Componential scoring • Wholistic scoring

All four groups learned Essays

All groups learned the same amount • Measures • Multiple choice test: Raw, normalized, malleable • Essay test: Lenient, stringent, wholistic • Combined measure of principle/misconceptions • Statistical power was sufficient • Subpopulations • Test items that are similar to training vs. not • Subjects with low vs. high pretests

Why did all 4 groups learnthe same amount? • Content alone controls learning, and all 4 groups covered the same content equally diligently? • Experiment 2 • Tests are not deep enough? • Experiment 3 • Not really “learning”, just “reminding”? • Experiment 4

Experiment 2: AutoTutor versus Reading a Textbook Three tutoring conditions • Why-AutoTutor • Read textbook control • Read nothing 63 participants knowledge of physics

Phases of Experiment • Survey on participant information • Pretest • Multiple choice test • 4 essay questions • Training on 10 problems vs. control • Posttest • Multiple choice test • 4 problems – essay answers

Multiple Choice Scores

Content: Textbook  Others Transcript = one tutor, one student, all 10 training problems For each pair of transcripts, compute the LSA cosine, with “object” substituted by problem-specific objects

Conclusions from Experiment 2 • Learning from Why2-AutoTutor is better than reading chapters from a textbook for nearly the same amount of time. • Reading from the textbook is no different from doing nothing at all. • The textbook may cover slightly different content than Why2-AutoTutor • Reading the textbook may not be particularly engaging.

Why2 Tutors that teach mental models using natural language dialog