Eye Movements, Attention,and Working Memory in Natural Environments

1. Eye Movements, Attention,and Working Memory in Natural Environments Mary Hayhoe University of Rochester

2. Selecting information from visual scenes What controls the selection process?

3. Fundamental Constraints Attention is limited. Visual Working Memory is limited. Humans must select a limited subset of the available information in the environment. Only a limited amount of information can be retained. What controls these processes?

8. The Question How do attentional and memory limitations play out in natural behavior? Need to understand Usage, ie mechanisms than control allocation of gaze, attention, and memory, not justCapacity - Natural behavior : sequences of operations over several sec - selection and timing under observer’s control. - Trial structure of standard paradigms : repeated instances of a single operation. Experimenter controls timing and nature of selection.

9. Developments in Eye Tracking Difficulty: optical power of eye + observer movement Head fixed (restricted): Contact lenses: magnetic coils, Dual Purkinje Image tracker Head Free: Head mounted IR video-based systems Scene camera

10. Investigation of natural tasks with head-mounted eye-trackers Scene camera on head provides video record of scene + eye position

11. Advantages of Natural Behavior Need active interaction with environment - not just passive viewing of images. Task structure allows interpretation of role of fixations.

12. Viewing pictures of scenes is different from acting within scenes. Heading Obstacle avoidance Foot placement

13. Eye Movements During Natural Behavior

14. (Hayhoe et al, 2003)

15. Other Tasks Driving (Land & Lee, 1992) Table Tennis (Land & Furneaux, 1997) Piano (Land & Furneaux, 1997) Toy models (Pelz et al, 2000) Cricket (Land & Macleod, 2000) Walking (Patla & Vickers, 1997,Turano et al 2003)

16. Saliency vs Tasks Image properties eg contrast, chromatic saliency etc can account for some proportion of the observed fixations when viewing images of scenes (Itti & Koch, 2001; Parkhurst & Neibur, 2003; Mannan et al, 1997). However, only modest role for image saliency in interactive tasks.

17. Insights from natural behavior 1. Fixations tightly linked to task: “just-in-time” strategy.

18. Fixations tightly linked to actions.

19. Timing of fixations linked to current action fixations Hand path Gaze arrives at critical point just before needed and departs when goal achieved. Johanssen et al 2001

20. “Just-in-time” strategy eye Model hand Workspace Resource Area (Ballard et al 1995)

21. Insights from natural behavior 1. Fixations tightly linked to task: “just-in-time” strategy. 2. Fixations patterns reflect learning at several levels: what objects are relevant/where information is located/order of sub-tasks/properties of world.

22. Cognitive Goal Learn task sequence Make PBJ sandwich Micro-task Learn where to fixate Get jelly Fixation Fixate jelly jar Acquire Info

23. Gaze distribution is very different for different tasks.Subjects have learnt that traffic signs are mostly at intersections Learning Where to Look (Shinoda et al, 2001) Follow 15% Time fixating Intersection. Obey traffic rules 45%

24. Learning optimal location: Fixate tangent point while driving around a curve Fixation density (Land & Lee, 1994) Gaze angle relative to body gives steering angle - gaze angle= “control variable”

25. Need to learn optimal location for control of pouring regulate flow monitor level (Land, Mennie, Rusted 1999)

26. Neural Substrate for Learning where to Look

27. Saccadic eye movement circuitry target selection LIP:lateral intra- parietal saccade decision saccade command inhibits SC signals to muscles

28. Neural Substrate for Learning where to Look Hikosaka et al, 2000: Caudate cell responses in basal ganglia reflect both upcoming saccade and expected reward. Regulates fixation & timing of saccades. Schultz, 2000: dopaminergic neurons in basal ganglia signal expected reward. Cortical saccade-related areas sensitive to reward: LIP - Platt & Glimcher, 1999; Sugrue et al, 2004 Supplementary eye fields - Stuphorn et al, 2000

29. Note: Targeted hand movements show similar rapid learning and influence of reward (Trommershauser et al, 2003)

30. Learning Properties of World

31. Eye movements in cricket bowler Bounce point batsman Land & MacLeod, 2000

32. Learning Properties of World Batsman anticipate bounce point Better batsman arrive earlier saccade pursuit Eye movements in cricket: Anticipation implies internal model of ball’s expected path bounce Land & MacLeod, 2000

33. The need for internal models Internal models of body’s dynamics mitigate problem of sensory feedback delays. (eg Wolpert et al, 1998) Less evidence for internal models of environment. eg - evidence for minimal memory representations However, need to plan movements and predict state of environment to counter visual delays. For example, memory of spatial structure of scene is necessary for coordinated movements (eg Chun & Nakayama, 2000 Hayhoe et al, 2003)

34. Eye movements when catching

35. X smooth pursuit saccade X Catching: Gaze Patterns X Thrower Catcher

36. Catching: GazeAnticipation 61 ms X X -53 ms X Thrower Catcher Timing of departure and arrival linked to critical events

37. Scatter Plot of Fixations near Bounce Relatively tight lateral clustering implies Ss target likely location of bounce. 20 deg bouncepoint 2D elevation Subjects fixate above the bounce point

38. Poor tracking when ball is unexpectedly bouncy

39. Better tracking 2 trials later- subjects have revised model

40. Pursuit accuracy following bounce tennis ball bouncy ball 5 subjects Pursuit improves rapidly with repeated trials

41. Earlier Arrival at Bounce Point Over Trials tennis ball bouncy ball Subjects continue to adjust saccade timing

42. Predicting the location for contact with racquet

43. Predictive Saccade Anticipation: 183 +/- 35 ms Ball Anticipatory saccade to predicted location 183 msec before ball.

44. Predictive Saccade ctd Racquet Error = 2.6 deg Fixation after saccade Duration: 250 +/- 21 ms Ball Ball arrives at fixation point

45. Internal Models Allow Predictive Vision Anticipatory saccades, head movements, and pursuit movements reveal that acquisition of visual information is planned for a predicted state of the world. Predictions may be based on some kind of internal model of events. Subjects rapidly adjust this model when errors occur. Rapid adjustment of performance suggests that prediction is a ubiquitous feature of visually guided behavior.

47. Cognitive Goal Make PBJ sandwich Micro-task Get jelly Fixation Fixate jelly jar How selective? Acquire Info Object? Feature? What is stored? Fixations alone don’t Specify what information is selected.

49. Insights from natural behavior 1. Fixations tightly linked to task: “just-in-time” strategy. 2. Fixations patterns reflect learning at several levels: what objects are relevant/where information is located/order of sub-tasks/properties of world. 3. Duration of fixations reflect time required to complete the current visual operation: implies specialized computations

50. Different fixation durations for different tasks Pelz et al, 2000