Lessons Learned in Teaching Mathematics with Adaptive Tutoring Software

1. Lessons Learned in Teaching Mathematics with Adaptive Tutoring Software Ivon Arroyo University of Massachusetts Amherst To Erica

2. Wayang Outpost --Math Tutoring SystemGrades 7,8,9,10 and community colleges http://Wayangoutpost.com

3. Empirical Learning Results since 2003 After short exposure (3-4 hours) MCAS passing% Wayang MCAS passing% No Wayang 77% 60% ** 34% 24% * d=0.25 92% 76% * d=0.24 Control No Wayang Wayang Posttest 76% 67% ** d=0.52 Expanding to 2000 students in 2011

4. Wayang Outpost --Math Tutoring SystemStandardized-test math problems with multimedia help http://Wayangoutpost.com Modality Animation Contiguity More Help

5. What have we learned about how to teach math with advanced technologies? Many things…. Some are supported by experimental evidence Some are conjectures and anecdotes…

6. What have we learned about teaching math? Adaptive Problem Selection Training Math Fluency Affect Offering Help Showing Progress

7. Lesson learned 1 Adaptive Math Tutoring that maintains students within a “zone of proximal development” improves learning.

8. What kind of adaptivity? Murray, T.; Arroyo, I. (2002) Toward Measuring and Maintaining the Zone of Proximal Development in Adaptive Instructional Systems, Lecture Notes in Computer Science, 2002, Volume 2363/2002, 749-758 Too much effort Fatigued Frustrated Little Effort Can we understand when we are outside of the ZPD?

9. Learning what is high and low effort In any problem pi i=1, .., N N=Total problems in system Incorrect Attempts Hints Time (each bar=5seconds) E(Ii) E(Hi) E(Ti) IL IH HL HH TH TL 0 1 2 3 4 0 1 2 3 4 5 6 7 Odd behavior: too much effort, or too little effort Attempts < E(Ii) — IL Hints > E(Hi) + HH Time < E(Ti) — TL Few Inc. Attempts Lots of Hints Little Time < > <

10. Scenarios outside of the ZPD

11. Where the ZPD is Murray, T.; Arroyo, I. (2002) Toward Measuring and Maintaining the Zone of Proximal Development in Adaptive Instructional Systems, Lecture Notes in Computer Science, 2002, Volume 2363/2002, 749-758 Too much effort dHIGH Fatigued Frustrated dLOW Little Effort Disengaged

12. How to increase/decrease problem difficulty? Arroyo, I., Mehranian, H., Woolf, B. (2010) Effort-based Tutoring: An Empirical Approach to Intelligent Tutoring. Proceedings of the 3rd International Conference on Educational Data Mining. Pittsburgh, PA.

13. Does this adaptivity improve learning? Randomized Controlled Experiment (N=56) Spring 2004 Raw percent Correct (Pre and Posttest) Accuracy over attempted problems ANCOVA for Posttest Score F(55,1)=8.4, p=.006 Arroyo, I., Mehranian, H., Woolf, B. (2010) Effort-based Tutoring: An Empirical Approach to Intelligent Tutoring. Proceedings of the 3rd International Conference on Educational Data Mining. Pittsburgh, PA.

14. Lesson learned 1 Adaptive Math Tutoring that attempts to maintain students within a “zone of proximal development” improves learning. • Being adaptive over smaller “chunks” of similar problems (instead of the full set of problems) yields higher learning • Being “gentle” at increasing difficulty yields higher learning

15. Lesson learned 2 Training Basic Arithmetic, not only for accuracy but for Speed to respond, enhances mathematics learning in combination with Wayang Outpost.

16. Math Facts Retrieval (fluency) training Royer, J. M., & Tronsky, L. N. (1998). Addition practice with math disabled students improves subtraction and multiplication performance. In T. E. Scruggs and M. A. Mastropieri (Eds.), Advances in Learning and Behavioral Disabilities (Vol 12). Greenwich, Conn.: JAI Press, Inc.

17. Results on Standard Math Test (Hard Items) True Means and SD for HARD items of the standardized pretest and posttest ANCOVA for Hard Posttest with Hard Pretest as covariate; MFR,Wayang fixed factors: Wayang F(222,1)=6.8, p=.01; WayangxMFR F(222,1)=6.8, p=.009 Post-Hoc Contrasts: Wayang > no-Wayang? Yes. Wayang-MFR > Wayang-NoMFR? Yes.

18. Royer&Tronsky (1998) Royer et al. (1999) But why? • Problem solving takes place in a cognitive system constrained by a limited capacity of working memory Working Memory Capacity When Solving a Math Problem Basic Math Basic Math Strategy • Doing Math is like speaking a language. • If you are fluent, you will concentrate better on the message. • Math Fluency • Math Fluency helps predict performance at state-wide tests • “Basic Math” could be anything… such as solving easy equations…

19. Lesson learned 2 Training Basic Arithmetic, not only for accuracy but for Speed to respond, enhances mathematics learning in combination with Wayang Outpost.

20. Lesson learned 3 Showing students their historical improvement (not just their mastery level) at math problem solving improves engagement in subsequent problems and increases learning.

21. Progress Monitoring InterventionsSelf-monitoring feedbackSelf-referenced-feedback (McColskey and Leary (1985)) Last 5 problems 5 problems earlier Similar to open learner models, but focus on progress

22. Progress Monitoring InterventionsSelf-monitoring feedbackSelf-referenced-feedback (McColskey and Leary (1985)) Last 5 problems 5 problems earlier

23. +16% +13% ANCOVA Dependent Learning (Posttest-Pretest score) Group Covariate Pretest score F=4.23, p=.043 Effect size: Cohen’s d=0.4 Mathematics Learning Results Means and Standard Deviations Percentages +0% +7%

24. Changes in time per problemMedians and Quartiles for time spent in the problem (in pairs of subsequent problems) S e c o n d s s p e n t i n a p r o b l e m 7 0 m e l b o 6 0 r p r e p 5 0 t n e p s 4 0 e m i t r o 3 0 f s e l i t r 2 0 a u q / n a 1 0 i d B e f o r e I n t e r v e n t i o n e M 0 A f t e r I n t e r v e n t i o n N = 3 0 3 3 0 3 5 2 9 5 2 9 Tutor-Intervention Tutor-Control C o n t r o l M o t i v a t i o n a l

25. Lesson learned 3 Showing students their historical improvement (not just their mastery level) at math problem solving improves engagement in subsequent problems and increases learning.

26. Lesson learned 4 Emotions/Attitudes/Affect as a more important long-term outcome than learning. In general, students are really bored about mathematics. Also, there are important group differences in students emotions, before tutoring.

27. Who needs more affective supportLow achieving students (95% of students with IEPs)

28. Who needs more affective supportHigh School Girls Figure 1: Results for a pre-tutor survey in two public schools: Girls develop negative feelings for mathematics, including decreased confidence (left) and increased frustration (right), between middle and high school.

29. Lesson learned 4 Emotions/Attitudes/Affect as a more important long-term outcome than learning. In general, students are really bored about mathematics. Also, there are important group differences in students emotions, before tutoring.

30. Lessons learned 5 We can understand and TRACE students at the affective level, understanding their emotions. How? from very recent behaviors, and with the help of physiological sensors.

31. Students Self-Report Emotions Every 5 minutes, students would report emotions

32. PredictingEmotionsin Real-Time Affective Tracing Linear Models to Predict Emotions from last Problem SeenModels Created using Stepwise Regression Tutor Context Variables (for the last problem) Seconds to 1st Attempt Seconds To Solve # Hints Seen Solved? 1st Attempt # Incorrect attempts Time in Tutor Character Present? R2=0.19 R2=0.3 R2=0.15 R2=0.18 Anxiety Frustration Enjoyment Boredom 86% 88% 78% 83% R2=0.38 R2=0.31 R2=0.40 R2=0.44 “Concen- trating” SitForward Stdev SitForward Mean “Interest” Min Max Pressure Accuracy of a YES/NO prediction of each emotion, compared to TRUE self-report Possible that looking at longer episodes of recent history will achieve as good accuracy as sensors.

33. Lessons learned 5 We can understand and TRACE students at the affective level, understanding their emotions. How? from very recent behaviors, and with the help of physiological sensors.

34. Lessons learned 6 Affective Characters that talk about the importance of Effort and Perseverance improve affect towards math for all, particularly for girls and low achieving students. However, they don’t impact learning.

35. Animated Pedagogical Agents “Cognitive” Pedagogical Agents Affective Pedagogical Agents ? Cognitive Outcomes (Retention, Transfer) Affective Outcomes (Motivation, Attitudes, Emotions)

36. Human-Like Affective Learning CompanionsAffective Experts, cognitive peers Train the idea of “Malleability of Intelligence” Dweck, C.S., (1999) Self-Theories: Their role in motivation, personality and development

37. Human-Like Affective Learning CompanionsAffective Experts, cognitive peers Correct No Effort Quick-guess incorrect Praise Effort and Time on Hints

38. Summary of ResultsAnalysis of Covariance Affective Learning Companions are good for all

39. Impact of Affective LCsfor all Lessfrustrationreported within the tutor withJane. **F(213,2)=6.1,p=.003 More Frustrated Neutral Frustration Level Less frustrated

40. Impact of Affective LCsfor all For N~95 students, comparing LCs vs. no-LCs Less boredom for math at posttest time in LC condition. +F(94,1)=3.4,p=.07 More Interested Neutral Interest Level More Bored

41. Impact of Affective LCsfor LOW ACHIEVING For N~95 students, comparing LCs vs. no-LCs More CONFIDENCE for math at posttest timeFOR LOW ACHIEVING.

42. Lessons learned 6 Affective Characters that talk about the importance of Effort and Perseverance improve affect towards math for all and for low achieving students. However, they don’t impact learning.

43. Lessons learned 7 There are important gender differences that suggest girls have more productive use of Wayang Keep in mind that you might be designing for a subset of the population

44. Summary of Results charactersAnalysis of Covariance Affective Learning Companions are good for all Benefit for all, but effect is stronger when considering Girls alone

45. Impact of Affective LCs on GIRLS Frustration Level More Frustrated Reduced Frustration With “Jane” Neutral Frustration Level Less frustrated See dotted line for increasedfrustration without companions

46. Impact of Affective LCs on BOYS Frustration Level More Frustrated Neutral Frustration Level Less frustrated

47. Summary of Results about charactersAnalysis of Covariance Affective Learning Companions are good for all Overall benefit, but the effect is stronger for Girls alone. Girls are the ONLY ones benefitting, and boys are clearly not.

48. Perceptions of Wayang Outpost Liked it? Did you Learn? Was Wayang concerned about your learning? Was it helpful? Means and S.E. for overall Perception of Wayang Outpost Positive perception Boys report a better experience Without ALCs. Girls report a better learning experience With ALCs. Neutral Perception (neither positive nor negative) Negative perception Males Females Males Females Learning Companion No Learning Companion

49. Gender differences in Accepting/Rejecting help Help Offered Help Accepted Help Rejected

50. Gender differences in attitudes and behaviors in Wayang