Learning Sciences: Supporting Group Work with Language Technologies

1. Learning Sciences:Supporting Group Work with Language Technologies Carolyn Penstein Rosé Language Technologies Institute and Human-Computer Interaction Institute With funding from the National Science Foundation and the Office of Naval Research

2. ? Technology Learning Sciences Linguistics Human-Computer Interaction

3. L I S T E N !

4. Technology Linguistics Human-Computer Interaction

5. Outline • Overview • Technology for Dynamic Collaborative Learning Support • Study One: Agents Offering Student Control • Study Two: Agents Behaving Socially • Conclusions and Current Directions

6. Outline • Overview • Technology for Dynamic Collaborative Learning Support • Study One: Agents Offering Student Control • Study Two: Agents Behaving Socially • Conclusions and Current Directions

7. Helping students learn together in on-line groups… Machine Learning Middle School Engineering Higher Education Math Earth Sciences Psychology High School

8. Collaborative Design Task • Instructional Goal: • Apply thermodynamics to design of a power plant based on the Rankine Cycle paradigm • Each pair turns in exactly one design • Competing Student Goals: • Power:Design a power plant that achieves maximum power output • Motivated by economic concerns • Green: Design a power plant that has the minimum impact on the environment • Motivated by environmental concerns

9. Dynamic Collaborative Learning Support • Support for collaborative learning is like training wheels • Effective support allows learners to achieve better collaboration • Unnecessary support can be demotivating • Fading support is ideal • But too little support can be detrimental as well • Human facilitators are able to achieve the right balance

10. Outline • Overview • Technology for Dynamic Collaborative Learning Support • Study One: Agents Offering Student Control • Study Two: Agents Behaving Socially • Conclusions and Current Directions

11. Technology for Dynamic Collaborative Learning Support • Identify conversational interactions that are valuable for learning • Automatic conversation analysis • Automates assessment of group processes • Enables context sensitive triggering of support • Interactive support technologies

12. Classroom Studies Chat room style interaction ConcertChat 2nd year Undergraduate Mechanical Engineering students Classroom session During the semester Instruction Pretest  Session  Posttest  Questionnaire 12 > Experimental Design >Study Procedure 12

13. ConcertChat Server ConcertChatActor ConcertChatListener MessageFilter PresenceFilter DiscourseMemory AnnotationFilter OutputCoordinator SocialController ActivityDetector ProgressDetector PlanExecutor RequestDetector T.TakingCoordinator IntroductionsManager PromptingManager TutoringManager TutoringActor IntroductionsActor PromptingActor Basilica Architecture (Kumar & Rosé, in press) • Easy integration with a variety of computer supported collaborative learning environments

14. Labeled Texts Labeled Texts TagHelper Behavior Unlabeled Texts A Model that can Label More Texts Time Monitoring Collaboration with Machine Learning Technology Download tools at: http://www.cs.cmu.edu/~cprose/TagHelper.html http://www.cs.cmu.edu/~cprose/SIDE.html <Triggered Intervention>

15. Using Conversational Agents to Support Collaboration

16. Overview of Findings • Dialogue agents more effective than hints (Rosé et al., 2001; Rosé et al., 2005; Kumar et al., 2006) • Collaborative learning with the support of dialogue agent significantly better than working alone (Kumar et al., 2007) • 1 S.D. = 1 letter grade • Working with a partner and an agent is 1.24 S.D. better • Working with a partner is 1 S.D. better • Working alone with an agent is 1 S.D. better

17. Overview of Findings • Students learn more when they have more control (Chaudhuri et al., 2009) • Students learn more from agents that employ social strategies (Kumar et al., 2010; Ai et al., 2010) • Students explore design space more completely when agents show more openness to alternatives ideas (Kumar et al., under review)

18. Outline • Overview • Technology for Dynamic Collaborative Learning Support • Study One: Agents Offering Student Control • Study Two: Agents Behaving Socially • Conclusions and Current Directions

19. Student Control • Student: so i think the best thing to go with first is to increase the pressure entering the connector • Tutor: You might want to review what you saw earlier in the sensitivity analysis investigating the effect of changing P-max on steam quality and efficiency, from the top half of page 13 in your book. • Tutor: type: HELP WITH CHANGEPMAX if you want to discuss it with me. • Student: help with changepmax • Tutor: Remember the sensitivity analysis you have completed where you manipulated P-Max and observed it's effect on the efficiency of the cycle. You can refer back to Figure 11 on page 13 of your booklet, which …

20. Student Control • Hypotheses: • Dialogue Agents increase conceptual depth • Students will be more engaged when they are offered control over timing • Pointer Hints • Suggestion that students think about a certain issue, with a pointer to a page in their book. • Dialogue Agents • Interactive “walk through” a line of reasoning found on the corresponding page in the book No Support Pointer Only Dialogue Only Pointer+ Dialogue

21. Example Interaction

22. Experimental Procedure • Consent form and Pre-Questionnaire (10 minutes) • Training (15 minutes) • Instruction on Rankine Cycles and Green Design (65 minutes) • Pretest (15 minutes) • Collaborative Design Exercise (40 minutes) • Packaging up the design (10 minutes) • Posttest and Post-Questionnaire (20 minutes)

23. Pointer Only • Student: so i think the best thing to go with first is to increase the pressure entering the connector • Tutor: You might want to review what you saw earlier in the sensitivity analysis investigating the effect of changing P-max on steam quality and efficiency, from the top half of page 13 in your book. • …

24. Dialogue Only • Student: so i think the best thing to go with first is to increase the pressure entering the connector • Tutor: Remember the sensitivity analysis you have completed where you manipulated P-Max and observed it's effect on the efficiency of the cycle. You can refer back to Figure 11 on page 13 of your booklet, which I have pasted onto your whiteboard for your convenience. You can delete it after this conversation with me if it is in your way. Having seen how changing P-Max, the Maximum Pressure at the Heater Inlet, effects efficiency and Steam, do you think as a general principle the Maximum pressure should be increased ? • Student: Yes • Tutor: So the Efficiency of a Cycle increases by…

25. Pointer + Dialogue • Student: so i think the best thing to go with first is to increase the pressure entering the connector • Tutor: You might want to review what you saw earlier in the sensitivity analysis investigating the effect of changing P-max on steam quality and efficiency, from the top half of page 13 in your book. • Tutor: type: HELP WITH CHANGEPMAX if you want to discuss it with me. • Student: help with changepmax • Tutor: Remember the sensitivity analysis you have completed where you manipulated P-Max and observed it's effect on the efficiency of the cycle. You can refer back to Figure 11 on page 13 of your booklet, which …

26. Results • ANCOVA Model • Dependent Variable: Posttest score • Independent Variables: Dialogue and Pointer • Covariate: Pretest score • Significant effect of Pointer Hints • F(1,71) = 7.5, p < .01, effect size .57 s. d. • Dialogue with and without Pointer hints, .8 s. d.

27. Results • Only Pointer + Dialogue has significant pre to post test gain • Only Pointer + Dialogue is significantly better than No Support

28. What was the impact of the experimental manipulation on displayed reasoning? • ANCOVA Model • Dependent Variable: Number of Reasoning Expressions • Independent Variables: Pointer and Dialogue • Covariate: Total amount of talk in dialogue

29. What was the impact of the experimental manipulation on displayed reasoning? • Significant effect of Dialogue on Amount of reasoning • F(1,71) = 7.9, p < .01 • Effect size .35 s.d.

30. Outline • Overview • Technology for Dynamic Collaborative Learning Support • Study One: Agents Offering Student Control • Study Two: Agents Behaving Socially • Conclusions and Current Directions

31. Experimental Design • Social Behavior • Frequent, Infrequent, None • Goal Alignment • Green, Power, Neutral • Hypotheses • Students will be more engaged with agents displaying social behaviors • Students will be sensitive to tutor goal orientation • Interaction effect Frequent Green Infrequent Green None Green Frequent Power Infrequent Power None Power Frequent Neutral Infrequent Neutral None Neutral > Experimental Design >Manipulations 31

32. An Example of Displaying Bias Green Bias • Green: What is bad about increasing heat input to the cycle is that more waste heat is rejected to the environment. • Neutral and Power: Increasing heat input to the cycle increases waste heat rejected to the environment. Power Bias: • Power: What is good about increasing heat input to the cycle is that more power output is produced. • Neutral and Green: Increasing heat input to the cycle increases power output produced. > Experimental Design >Agent Design 32

33. Example of Social Behaviors Adapted from Bales’ IPA (Bales, 195) Tutor: Let’s Introduce ourselves. My name is Avis. Tutor: Be nice to your teammates! Tutor: I’m happy to work with our team :-) Tutor: m-hmm (showing attention) 33 > Experimental Design >Agent Design 33

34. Experimental Design • Social Behavior • Frequent, Infrequent, None • Goal Alignment • Green, Power, Neutral • Hypotheses • Students will be more engaged with agents displaying social behaviors • Students will be sensitive to tutor goal orientation • Interaction effect Frequent Green Infrequent Green None Green Frequent Power Infrequent Power None Power Frequent Neutral Infrequent Neutral None Neutral > Experimental Design >Manipulations 34

35. Results • ANCOVA Model • Dependent Variable: Post-test score • Independent Variables: Social and Match • Covariate: Pre-test score • Significant effect of Social behavior • F(2,94) = 5.27, p < .01 • Infrequent is significantly better than the other two (effect size .83 s.d.) • Marginal interaction between Social Behavior and Match • Infrequent Social only better when Match is true Match: For Green students: Green = Match Power = NoMatch Neutral = Neutral For Power students: Green = NoMatch Power = Match Neutral = Neutral Frequent Green Infrequent Green None Green Frequent Power Infrequent Power None Power Frequent Neutral Infrequent Neutral None Neutral

36. Other Effects of Social Behavior • For each student, computed number of contributions that were: • Positive social, Off-task, Negative comments about tutor • Students contribute more Positive social contributions in Social conditions • Students contribute more Off-task turns in the NoSocial condition • Students contribute more Negative comments about the tutor in the Frequent social conditions

37. Measuring Student Bias Using a topic modeling tool – ccLDA [Paul and Girju, 2009] Corpus Collection1 Collection2 Topic1 Topic 2 Topic 3 37 > Experimental Design >Displaying Bias 37

38. Example of Extracted Topics TOPIC 1 TOPIC 2 Background Green Power Background Green Power > Experimental Design >Displaying Bias 38

39. Bias Measurement Metrics • Max Topic-word bias: count the number of words in the list of the N most strongly associated words, and take the maximum across topics • Average Topic-word bias: count the number of words in the list of the N most strongly associated words, and take the average across topics • Weighted Topic-Word Average bias: Same but weight each word by its association within the background model first • All three measures highly correlated both for Green and for Power perspectives • Demonstrates convergent validity • Students in the Green condition got higher Green scores on average than Power scores and vice versa in the Power condition • Demonstrates face validity of construct

40. Mapping Influence • Per student, Green and Power scores were negatively correlated • Demonstrates divergent validity • Within pairs, Green scores of both students positively correlated, same with Power scores • Suggests positive collaboration • Students displayed more bias when the agent was biased • Suggests that Agent bias enhances the effect of the student level goal manipulation > Experimental Design >Displaying Bias 40

41. What did we learn about designing dialogue agents? • Dialogue agent technology is effective for supporting collaborative learning • Can be triggered through automatic text analysis • Increases amount of explanation • Increases learning • Design of agent behavior must be sensitive to social concerns • Students are more receptive when they have control over timing • Students engage better with agents that are on their side • Social behavior is effective when it is not too frequent

42. Outline • Overview • Technology for Dynamic Collaborative Learning Support • Study One: Agents Offering Student Control • Study Two: Agents Behaving Socially • Conclusions and Current Directions

43. Current Directions • Further exploration of bias and influence in collaborative discussions • Tutor could always appear to be on the student’s side • Complex Management of Attention • Engagement between student and agent • Engagement between student and partner • New! Exploration insights from theories of Pragmatics and Sociolinguistics for detecting social dynamics from speech

44. The (Near!) Future of Online Group Work

45. Assessment from speech *Using prosodic features rather than content Speech Prediction Report

46. Encoding Insights from Linguistics into Representation of Speech STYLE VALUE Level Markedness Shift State of the Art Pragmatic Sociolinguistic TIME

47. Prior work on analysis of social meaning from speech • Social aspects that are directly connected to independent prosodic features • Emotions e.g. annoyance, anger, sadness, boredom [Ang et al 02, Lee&Narayanan 02, Liscombe et al 03] • High accuracy (70 ~ 90%) • Social aspects that are not directly connected • Personality e.g. charisma, extroversion [Rosenberg & Hirschbert 05, Mairesse et al. 07] • Low correlation values reported (r=0.2 ~ 0.45)

48. What about idea Co-Construction? • Definition: Process of taking up, transforming, or building on an idea expressed earlier in a conversation • Example: I understand that plastic is cheaper, but paper is better for the environment Let’s use plastic since it’s so cheap. ✔ No, that won’t work ✖

49. What about idea Co-Construction? 28 MFCC (power) Segment duration • 92 prosodic features • Random baseline • Recall: 0.17, precision: 0.15, f-score: 0.16, kappa: 0.04 • Result • Recall: 0.72, precision: 0.24, f-score: 0.35, kappa: 0.23 4 Pitch related Speaker change 4 Energy (intensity) related 48 Phonemes 4 Amplitude related 3 phoneme related

50. L I S T E N !