Real-time interactions between attention and behavior in multimedia learning environments

1. Real-time interactions between attention and behavior in multimedia learning environments Susan Letourneau Postdoctoral Fellow, CREATE Lab NYU & CUNY Graduate Center LearnLab Summer Workshop August 4, 2012

2. How can multimedia technology be made more effective for learning? CREATE Lab research includes: • Systematic investigation of design principles that may support learning • Iterative development of educational games and simulations

3. Interactivity and Engagement • Students interact and “engage” with multimedia materials in different ways: • By acting and doing • By looking and thinking • By reacting and feeling • How can we capture attention, cognition, emotion, in addition to behavioral activity? • Multiple measures: • Activity logs • Eye-tracking • Physiological responses

4. Eye-tracking measures of visual attention • Benefits • Remote, noninvasive • Objective • Continuous recording • Measures include: • Location of gaze • Duration of fixations • Fixation Sequences

5. Integrating Activity Logs & Eye-tracking • Synchronized recordings of behavior and attention using common timestamp • Data analysis approaches: • Behaviors as individual events • Behaviors as markers or dividers to parse eye-tracking data • Sequences of gaze and behavior over time

6. Study 1: Gaze and Activity in a Chemistry Simulation • 26 high school students • Measures: • Eye-tracking, activity logs • Pre/post-tests of chemistry knowledge

7. Gaze transitions between multiple representations correlated with learning outcomes • Controllers-Axes: =.54, t(20)=2.88, p=.01, Container-Graph: =.46, t(20)=2.38, p=.02 • Students often looked to these key areas immediately after changing a variable in the simulation

8. Study 2: Using visual scaffolds to guide attention • 28 high school students, using simulation with or without scaffolds • Examined gaze patterns following interactions with the controllers

9. Scanpaths follow the path of the scaffolds. • Students with more transitions show higher learning outcomes • [Controllers-Axes, r=.56, p<.01]

10. Study 3: Attention during experimentation. • 32 high school students planned and executed experiments in a chemistry simulation • Activity logs used to divide eye-tracking data into three types of activities: • Adjusting variables • (planning experiment) • Watching ongoing experiment • Experiment completed

11. Students directed attention to different parts of the simulation during different activities. Attention to the graph area specifically while students planned an experiment was correlated with post-test scores [=0.49, t(22)=2.51, p=.02].

12. Planning Watching End of Experiment

13. Ongoing work: Physiological measures of cognitive and affective responses • Cognition: • Eye-tracking • EEG • Emotion: • Skin conductance • Heart rate

14. Triangulating multiple measures Physiological measurements can be synchronized with eye-tracking and behavioral recordings. Measurements can be time-locked with any channel of information.

15. Current Research Directions • Controlled comparisons of responses to tasks Cognitively Engaging Behaviorally Engaging Affectively Engaging

16. Acknowledgments • CREATE Lab • PIs: Jan Plass, Bruce Homer, Catherine Milne • Lizzie Hayward, Ruth Schwartz • Institute of Education Sciences, IPORT Fellowship