Supporting Students Working Together on Math with Social Dialog

1. Supporting Students Working Togetheron Math with Social Dialog Rohit Kumar, Gahgene Gweon, Mahesh Joshi, Yue Cui, Carolyn Rosé Language Technologies Institute Carnegie Mellon University

2. Introduction • We are investigating the effectiveness of methods to SUPPORTSTUDENTS • Conversation as Dynamic Feedback • Two types of Conversational Agents • Instructive behavior (e.g. Offering Help) • Social Behavior (e.g. Promoting Collaboration) DynamicFeedback vs. Static Instruction CollaboratingTeams vs. Individuals i n t r o d u c t i o n : a p p r o a c h : e x p e r i m e n t : r e s u l t s : c o n c l u s i o n

3. Introduction – Motivation • We are exploring the space of strategies for conversational agents • Instructive as well as Social Strategies • Conversational Agents / Tutorial Dialogue shown to be effective support for Individual learning • Working collaboratively has been shown to change the way students conceptualize a learning task and respond to feedback i n t r o d u c t i o n : a p p r o a c h : e x p e r i m e n t : r e s u l t s : c o n c l u s i o n

4. Introduction – Challenges • What should be the basis for these conversational strategies ? • Corpus Analysis • Theory from related disciplines • Iterative Design • Is there a general purpose framework for implementing and deploying these conversational strategies? • How do you evaluate the effectiveness of these strategies ? • Metrics: Standardized or Task Specific i n t r o d u c t i o n : a p p r o a c h : e x p e r i m e n t : r e s u l t s : c o n c l u s i o n

5. Approach – Task Domain • Mathematics Problem Solving • Sixth Grade • Fractions • Addition • Subtraction • Multiplication • Division CTAT Problem Solving Interface with Cognitive Tutors based feedback [Aleven, Koedinger] (shared by students) Conversation Interface (shared by students & conversational agents) i n t r o d u c t i o n : a p p r o a c h : e x p e r i m e n t : r e s u l t s : c o n c l u s i o n

6. Approach • Two Conversational Strategies explored • Strategy 1: • Instruction offering dialog • Strategy 2: • Social Agent encouraging participation i n t r o d u c t i o n : a p p r o a c h : e x p e r i m e n t : r e s u l t s : c o n c l u s i o n

7. Approach – Strategy 1 (Instructive) Tutor Dividing by two means you take the whole and break it into two parts. What would it mean to divide by three instead? Student5 you would break it into 3 parts Tutor Correct. What do you do differently if you divide by 1/3 instead? Student13 6 Tutor Actually, you need to multiply by 3. You should think about why you need to multiply. We'll talk about it more later. Example • Motivated by the intention of knowledge construction • Six knowledge construction dialogs triggered at fixed time intervals • Agent do not interrupt during problem solving: Policy based on past experiments • Authored using TuTalk • TuTalk integrated as an Conversational Agent in our framework i n t r o d u c t i o n : a p p r o a c h : e x p e r i m e n t : r e s u l t s : c o n c l u s i o n

8. Approach – Strategy 2 (Social) Tutor:Student1, If you had to choose between a long flight and a longcar ride, which seems less uncomfortable? Student1: I’ll take the car ride Tutor:Ok Student1. Tutor:Student2, Which are more entertaining – books or movies? Student2: definitely books! Tutor:Ok Student2. Tutor:Please work on the problem shown in the side panel. Example • Motivated by the intention of engaging students • By showing interest in their personal preferences • Goal of this social conversation is to make the students feel that they worked together to construct the problem statement • Comes up every time the students are about to start solving a new problem • Implemented as a conversational agent in our framework Jan packed several books to amuse herself on a long car ride to visit her grandma. After 1/5 of the trip, she had already finished 6/8 of the books she brought. How many times more books should she have brought than what she packed? i n t r o d u c t i o n : a p p r o a c h : e x p e r i m e n t : r e s u l t s : c o n c l u s i o n

9. Approach – Infrastructure Strategy1 (Instructive) Uses TuTalk Dialog Engine at the back end Strategy1 (Instructive) Trigger is a timer and problem completion in the CTAT Problem solving interface Strategy2 (Social) Conversational Agent built into the framework Strategy2 (Social) Trigger is ready to start new problem start in the CTAT Problem solving interface i n t r o d u c t i o n : a p p r o a c h : e x p e r i m e n t : r e s u l t s : c o n c l u s i o n

10. Approach – How to Evaluate ? • Methods • Cognitive Evaluation • Learning Gains Analysis • Social Evaluation • Questionnaire • Analysis of Conversation • Conversational Strategy 1 (Instruction Offering) • Learning Gains Analysis • Conversational Strategy 2 (Social Dialog) • Questionnaire • Analysis of Conversation i n t r o d u c t i o n : a p p r o a c h : e x p e r i m e n t : r e s u l t s : c o n c l u s i o n

11. Experiment – Design • Participants • 30 sixth grade students from a suburban elementary school • Dependant Measures • Test, Quizzes, Questionnaires, Analysis of Conversation i n t r o d u c t i o n : a p p r o a c h : e x p e r i m e n t : r e s u l t s : c o n c l u s i o n

12. Experiment – Procedure • Day 1 (Wednesday) • Pre-test • Day 2 (Thursday) • 45 minute lab session • Quiz 1 • Day 3 (Friday) • 45 minute lab session • Quiz 2 • Questionnaire • Day 4 [separated from Day 3 by a weekend] (Monday) • Post-test i n t r o d u c t i o n : a p p r o a c h : e x p e r i m e n t : r e s u l t s : c o n c l u s i o n

13. Experiment - Questionnaire • Eight Questions about • Perceived problem solving competence of self • Perceived problem solving competence of partner • Perceived benefit of collaboration • Perceived help received • Perceived help provided • All questions were answered on a 6 point Likert scale • 0 (strongly disagree) …. 5 (strongly agree) i n t r o d u c t i o n : a p p r o a c h : e x p e r i m e n t : r e s u l t s : c o n c l u s i o n

14. Results - Questionnaire Students perceived significantly higher help offering by their partners in the Experimental condition Students perceived they offered more help to their partners in the Experimental condition i n t r o d u c t i o n : a p p r o a c h : e x p e r i m e n t : r e s u l t s : c o n c l u s i o n

15. Results – Analysis of Conversation • To investigate if students really offered more help, conversations were hand-coded • [R] Help Requests “Help me”, “I’m stuck”, “I don’t get it.” • [P] Help Provisions “Find the common denominator”, “Try the flip and multiply strategy” • [C] Can’t help “I don’t know”, “I’m stuck too” • [D] Deny help “ask the teacher”, “you’re an idiot”, “press the help button” • [N] Nothing i n t r o d u c t i o n : a p p r o a c h : e x p e r i m e n t : r e s u l t s : c o n c l u s i o n

16. Results – Analysis of Conversation • Observations • Average number of Help Provisions [P] not significantly different across conditions • More help related episodes per problem in the Experimental condition Mean (Control) = 0.30 Mean (Experimental) = 0.69 F(1, 15) = 16.8 p < 0.001 • More episodes of [D] Deny Help in Control condition Mean (Control) = 40.2 Mean (Experimental) = 24.7 F(1, 62) = 3.46 p = 0.001 • Students displayed more negative attitude in Control conditions • Insults like “you stink”, “stupid” occurred only in Control condition i n t r o d u c t i o n : a p p r o a c h : e x p e r i m e n t : r e s u l t s : c o n c l u s i o n

17. Results – Learning Gains Analysis • Weak evidence in favor of Experimental condition • Consistent (but non-significant!) trend for students in the Experimental condition to learn more • Marginal advantage for Experimental condition on Lab day 2 on Interpretation problems (p=.06, effect size=0.55 s.d.) N1 N2 S D1 D2 i n t r o d u c t i o n : a p p r o a c h : e x p e r i m e n t : r e s u l t s : c o n c l u s i o n

18. Conclusions • Social Conversational Agents can play important (and different) role in supporting collaborative learning • Not just Extraneous Entertainment • It is important to study these besides instruction / help providing tutorial dialog • A strategy we explored changed the students attitude towards each other • Improved perceived help giving and help receiving among partners in a collaborative learning environment i n t r o d u c t i o n : a p p r o a c h : e x p e r i m e n t : r e s u l t s : c o n c l u s i o n

19. Conclusions – Directions • Continued exploration of more social conversational strategies motivated by theories of social processes in learning • Development of our general purpose architecture to implement, deploy and conduct experiments with new conversational agents Thank Youfor listening to my Talk For More Information:rohitk@cs.cmu.edu, cprose@cs.cmu.edu i n t r o d u c t i o n : a p p r o a c h : e x p e r i m e n t : r e s u l t s : c o n c l u s i o n