Jose Carlo A. Soriano

1. Effective Help-seeking BehaviorAmong Students Using an Intelligent Tutoring System for Math: a Cross-Cultural Comparison Jose Carlo A. Soriano

2. Break down • Effective Help-seeking Behavior Among Students • Using an Intelligent Tutoring System for Math: • A Cross-Cultural Comparison

3. Intelligent Tutoring Systems • Seeks to simulate the effectiveness of a good personal human tutor • Individual tutoring is more effective than classroom instruction by 2 standard deviations (Bloom, 1984) • Knows what specific skills the student is having trouble with • Able to offer appropriate help at an appropriate time • Objective: help the student learn • ITS are better by 1 standard deviation (Koedinger et al, 1998; Corbett et al., 2001)

4. Intelligent Tutoring Systems

5. Help-Seeking in ITS • The kind of help, and how the help is offered, affects learning (Aleven et al, 2003) • Students who use High-level help most frequently have the least learning(Matthews and Mitrovic, 2008) • Help-seeking is a Meta-cognitive skill • Meta-cognition is “cognition about cognition” • One’s knowledge of the processes in play • One’s active control of it during learning

6. Help-seeking Behavior • Both the EU and UNESCO declared: • developing metacognitive skills, or ‘teaching students how to learn’ should be among the highest educational priorities (Louizidu and Kotselini, 2007) • Help-seeking behavior is an achievement-related behavior (Karabenick and Knapp, 1991) • Higher-achieving students were more likely to ask for help when encountering personal difficulties (Taplin et al, 2001)

7. Effective Help-Seeking • Student is more likely to learn when: • Student seeks for help when encountering personal difficulties • Student knows what kind of help is needed such that student can work effectively on his/her own • Student knows how to ask for help • Student is not dependent on help • Student spends time understanding help given

8. Help-seeking in ITS • However, students generally do not know when they need help (Aleven and Koedinger, 2000) • Students “game the system” (Baker et al) • Meta-cognitive tutors have been developed by Aleven et al • “Scooter the tutor” developed by Baker et al • To teach students “how to learn”

9. The Problem • Is “effective help-seeking” the same across cultures? • Very few cross-cultural comparisons • Comparing ITS use between USA and Latin American students(Ogan et al, in press) • Comparing Disengaged behavior between USA and Filipino students (Rodrigo et al, 2010) • Implications on Meta-cognitive tutors

10. Method • Find out if effective help-seeking behavior is the transferrable across cultures, or are significantly different • Might encourage more cross-cultural comparisons • Implications on future efforts on meta-cognitive tutoring

11. Scatterplot tutor

12. Data • Costa Rica • Mexico • USA • Philippines

13. Feature Engineering 1. Helpavoidance 2. Nothelpavoidance 3. Helpnonuse 4. Unneededhelp 5. BugmsgLongpause 6. BugmsgShortpause 7. HintmsgLongpause 8. HintmsgShortpause

14. Feature Engineering 9. HintmsgLongpauseCorrect 10. HintmsgShortpauseCorrect 11. NothelpavoidanceShortpause 12. NothelpavoidanceLongpause 13. UnneededhelpShortpause 14. UnneededhelpLongpause 15. ShortpauseHintmsg 16. LongpauseHintmsg 17. FirstattemptHintmsg

15. Feature Optimization • Most features require a threshold, either p-know or a time threshold • Brute-force grid search: • For p-know thresholds, grid-size is 0.05 • For pause thresholds, grid-size is 0.5 seconds • Single-parameter linear regression for each threshold for each feature in grid

16. Feature Selection • Cross-validated r wasused as the goodness criterion • Correlation between the predicted learning values and the actual learning value • The threshold with the best cross-validated r becomes the threshold for each feature • As an additional control against over-fitting, features whose best threshold had negative cross-validated r is dropped from model creation

17. Model Creation and Evaluation • Models were created using Forward Selection • Models were evaluated by applying each country’s model to each country’s data set • A model were created after combining the four data sets

18. Brute-Force Grid Search

19. Analysis • FirstattemptHintmsg – only feature that was not able to pass in any country • Different nature of skills • Some can be hard at the very start, some can be easily understood from the start • Most skills may require a large number of actions • NothelpavoidanceLongpause – had positive cross-validated correlation in all five data sets • In contrast to theory, this feature has negative directionality for most countries • This might be because students who exhibit the behavior simply do not understand the skill

20. Forward Selection

21. Analysis • Helpavoidance – negative directionality for CR and PH • reinforccesAleven et al’s findings that avoiding help is negatively correlated to learning (Aleven et al., 2006) • ButmsgShortpauseand NothelpavoidanceShortpause also has negative directionality for MX and US, and PH and US. • Possibly effect of not spending enough time to understand bug or hint message provided

22. Cross-Cultural Evaluation Rows are models, columns are data sets applied to

23. Analysis • Eighteen out of 25 model applications produced positive correlation between model’s predicted learning and the actual learning • Negative correlation means model did worse than chance at predicting learning • LOOCV r values are very high compared to r when applied to other countries • Reinforces hypothesis that help-seeking might not transfer across countries

24. Analysis • MX and US – performance on each other’s data sets are low (-0.009 and -0.164) • This means that our model of effective help-seeking is not effective when applied to the other country • Reinforces findings in (Ogan et al, in press) which compares differences in behavior of USA and Mexico students • Collaborative nature of students from Mexico may be the reason why help-seeking is different • The help they ask from the tutor will only be help that they do not get from other students

25. Analysis • CR and MX – did not perform well on each other’s data sets (-0.285, 0.04) • Though collaborative tendencies might be common between Costa Rica and Mexico, help-seeking behavior with the ITS may differ • US and PH – performed very well on each other’s data set (0.146 and 0.228) • Reinforces findings in (San Pedro, 2011) wherein a carelessness detector is generalizable between the two countries • In contrast to (Rodrigo, 2010) which shows disengaged behavior is different between the two countries • But Help-seeking and Disengaged behavior are two different sets of behaviors

26. Conclusion • Results did not expose that effective help-seeking as a whole is very culture-specific (18 out of 25 applications returned positive r) • However, it is not apparent that effective help-seeking is transferrable across countries • Big difference between LOOCV r and cross-cultural evaluations

27. Conclusion and Recommendations • There are pairs of countries wherein effective help-seeking fail to generalize to each other’s data set. • Meaning effective help-seeking from one country may not necessarily be effective in another • Single effective help-seeking models used by meta-cognitive tutors may be effective in one culture but not in others • Future meta-cognitive tutors might have to use a more generalizable model • May have to switch models, depending on the culture where the ITS is used