Meta-Cognition, Motivation, and Affect

1. Meta-Cognition, Motivation, and Affect PSY504Spring term, 2011 April 13, 2011

2. Gaming the System(Baker, Corbett, Koedinger, & Wagner, 2004; Baker et al., 2006)

3. Gaming the System “Attempting to get correct answers and advance in a curriculum by taking advantage of the software’s help or feedback, rather than by actively thinking through the material”(Baker et al., 2004, 2006)

4. Gaming in Intelligent Tutors • Help Abuse (cf. Wood and Wood, 2000; Aleven, 2001; Aleven et al., 2004)

5. Gaming in Intelligent Tutors • Systematic Guessing (cf. Baker et al., 2004)

6. Gaming in Intelligent Tutors • Systematic Guessing (cf. Baker et al., 2004) • Alternately conceptualized as “hasty guessing”/”rapid guessing” (Beck, 2005; Muldner et al., 2010)

7. Gaming in Intelligent Tutors • Intentional Rapid Mistakes (cf. Murray & VanLehn, 2007; Baker, Mitrovic, & Mathews, 2010)

8. Gaming the System • First documented in 1972(Tait, Hartley, & Anderson, 1972) • Has been documented in many learning environments • Intelligent Tutors (Baker et al, 2004; Beck et al, 2005; Murray & VanLehn, 2005; Walonoski & Heffernan, 2006; Johns & Woolf, 2006) • Puzzle-Solving Games (Rodrigo et al, 2007) • Collaborative Games (Magnussen & Misfeldt, 2004) • Graded-Participation Newsgroups (Cheng & Vassileva, 2005)

9. Methods for Assessing Gaming • Quantitative Field Observations • Text Replays • Machine-Learned/Data-Mined Models • Rational/Knowledge-Engineered Models

10. Quantitative Field Observations(Baker et al., 2004) • Each student’s behavior observed several times as they used tutor (by 2-3 observers) • Pre-determined coding categories and observation order • Peripheral vision • 20 second observation window

11. Quantitative Field Observations • Historically unsynchronized • Recent work in Baker’s lab has developed methods for synchronizing field observations with log files using Android handhelds(personal communication, Ryan Baker)

12. Inter-rater reliability • k = 0.74 (Baker, Corbett, & Wagner, 2006) • k = 0.71, 0.78 (Baker, D’Mello, Rodrigo, & Graesser, 2010)

13. Text Replays(Baker, Corbett, & Wagner, 2006)

14. Text replays • Pretty-prints of student interaction behavior from the logs

15. Real Examples

21. Inter-rater reliability (for gaming) • k = 0.58 (Baker, Corbett, & Wagner, 2006) • k = 0.80 (Baker, Mitrovic, & Mathews, 2010)

22. Machine Learned/Data Mined Models • Developed using data from text replays and quantitative field observations

23. Accuracy • The latest model versions can • Distinguish a gaming student from a non-gaming student 96% of the time (not relevant for off-task; in the USA, almost everyone goes off-task sometimes) • Achieve a correlation to frequency of gaming behavior of 0.9, and a correlation to off-task behavior of 0.62 • Distinguish off-task behavior from when a student is talking to the teacher • Determine exactly when gaming behavior and off-task behavior occurred, 40% better than chance • Predict student behavior accurately for new students and new tutor lessons • Not yet clear if detectors can transfer between tutors, but gaming detectors developed for high school Algebra can predict learning gains in college Genetics

24. Rational Model(Gong, Beck,& Heffernan, 2010) • “Rapid Guessing: submit answers less than 2 seconds apart at least twice in a row. • Rapid Response: perform any action after a hint or starting a problem before a reasonable amount of time has passed (where “reasonable” is a fast reading speed for the content of the hint or problem body. We chose a reading rate of 400wpm). • Repeatedly Bottom-out Hinting: reach a bottom out hint on three consecutive problems.”

25. Rational Model(Muldner, Burleson, Van de Sande, VanLehn, 2010) • “Skipping a hint: the tutor presents a hint and the student skips the hint by quickly asking for another hint (under 3 seconds) • Copying a hint: the tutor presents a bottom-out hint and the student quickly generates a solution entry, suggesting a shallow copy of the hint instead of learning of the underlying domain principle (under 4 seconds) • Guessing: after the tutor signals an incorrect entry, the student quickly generates another incorrect entry, suggesting s/he is guessing instead of reasoning about why the entry is incorrect (under 4 seconds) • Lack of planning: after the tutor signals a correct entry, the student quickly asks for a hint, suggesting reliance on hints for planning the solution (under 4 seconds)” • Thresholds set by visual inspection of data

26. Other Rational Models • Aleven et al. (2004, 2006) • Beck (2005) • Johns & Woolf (2006) • Beal, Qu, & Lee (2007)

27. Advantages/Disadvantages • What are the advantages and disadvantages of each method of assessing gaming?

28. Gaming and Learning

29. Gaming and Learning in Cognitive Tutors(5 field observation studies, 2003-2005)

30. Beck (2005) • Gaming studied in Project LISTEN reading tutor for elementary school students • Measured using rational model • Gaming associated with significantly lower learning

31. Aleven et al. (2006) • Gaming studied in Cognitive Tutor • measured using rational model • Gaming associated with significantly lower learning

32. Walonoski & Heffernan (2006) • Gaming studied in ASSISTments for mathematics • measured using data mined model constructed from field observations • Trend towards gaming being associated with lower learning, but not significant

33. Gong et al. (2010) • Gaming studied in ASSISTments for mathematics • measured using rational model • Gaming associated with significantly lower learning

34. Gobel (2010) • Gaming studied in college classroom in Japan using ESL software • measured using quantitative field observations • No relationship to learning (p value and correlation not reported)

35. Special Case

36. Shih, Koedinger, & Scheines (2008)

37. Shih, Koedinger, & Scheines (2008) • If student games the system, obtains the answer, enters the answer • And • Then pauses for a substantial period of time • Interpreted as self-explanation • This behavior is associated with successful learning (in Cognitive Tutor for Geometry)

38. Factors Predicting Gaming

39. Factors Predicting Gaming: State or Trait? • Baker (2007) found that gaming varies more by lesson-level factors than student-level factors • Using machine-learned detector • Muldner et al. (2010) and Gong et al. (2010) found that gaming varies more by student-level factors than problem-level factors • Using rational models

40. Factors Predicting Gaming: Trait • Baker et al. (2008) studied gaming and student characteristics using gaming detector, in Cognitive Tutor and ASSISTments • Disliking mathematics (r=0.19), lack of grit (r=0.22) significantly associated with gaming • Performance goals and other constructs not associated

41. Factors Predicting Gaming: State • Baker et al. (2009) developed taxonomy of 79 tutor features • Labeled data from 58 students and 20 lessons in terms of taxonomy and gaming (using text replays) • Split taxonomy using factor analysis to produce 6 factors for tutor design • 1 factor statistically significant predictor of gaming

42. (Some) tutor lesson features associated with gaming • Hints are not associated with better future performance (more gaming) • Proportion of hints in each hint sequence that refer to abstract principles (more gaming) • Average number of words in problem statements not directly related to math (less gaming) • No problem statement (less gaming) • Not immediately apparent what icons in toolbar mean (to teacher coder with tutor experience) (more gaming)

43. Final Model • Achieves r2 = 0.56

44. Next Class (APRIL 18) • Off-Task Behavior and Carelessness • Readings • Karweit, N., Slavin, R.E. (1982) Time-On-Task: Issues of Timing, Sampling, and Definition. Journal of Experimental Psychology, 74 (6), 844-851. • Baker, R.S.J.d. (2007) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068. • Clements, M.A. (1982) Careless Errors Made by Sixth-Grade Children on Written Mathematical Tasks. Journal for Research in Mathematics Education, 13 (2), 136-144.