Eye Movements and Working Memory Marc Pomplun Department of Computer Science

1. Eye Movements andWorking Memory Marc Pomplun Department of Computer Science University of Massachusetts at Boston E-mail: marc@cs.umb.edu Homepage: http://www.cs.umb.edu/~marc/

2. Overview: Image Processing: Convolution Filters Iconic Memory Representations for Visual Search Working Memory Use in a Natural Task The Working Memory - Eye Movement Tradeoff Eye Movements andWorking Memory

3. 1 6 3 2 9 1/9 1/9 1/9 2 11 3 10 0 1/9 1/9 1/9 5 10 6 9 7 1/9 1/9 1/9 3 1 0 2 8 4 4 2 9 10 Convolution Filters Averaging Filter: Grayscale Image:

4. 0 1 6 0 0 3 0 2 0 9 1/9 1/9 1/9 2 0 11 3 10 0 0 1/9 1/9 1/9 5 0 10 6 9 0 7 1/9 1/9 1/9 0 3 1 0 2 0 8 0 4 0 4 2 0 0 9 10 0 Image Processing Filtered Image: Original Image: 5 value = 11/9 + 61/9 + 31/9 + 21/9 + 111/9 + 31/9 + 51/9 + 101/9 + 61/9 = 47/9 = 5.222

5. 0 1 6 0 0 3 0 2 0 9 1/9 1/9 1/9 2 0 11 3 10 0 0 1/9 1/9 1/9 5 0 10 6 9 0 7 1/9 1/9 1/9 0 3 1 0 2 0 8 0 4 0 4 2 0 0 9 10 0 Image Processing Filtered Image: Original Image: 5 7 value = 61/9 + 31/9 + 21/9 + 111/9 + 31/9 + 101/9 + 101/9 + 61/9 + 91/9 = 60/9 = 6.667

6. 1 6 3 2 9 0 0 0 0 0 2 11 3 10 0 0 0 5 10 6 9 7 0 0 3 1 0 2 8 0 0 4 4 2 9 10 0 0 0 0 0 Image Processing Filtered Image: Original Image: 5 7 5 5 6 5 4 5 6 Now you can see the averaging (smoothing) effect of the 33 filter that we applied.

7. 1 • 4 • 7 • 4 • 1 • 4 • 16 • 26 • 16 • 4 • 7 • 26 • 41 • 26 • 7 • Discrete version:1/273 • 4 • 16 • 26 • 16 • 4 • 1 • 4 • 7 • 4 • 1 Gaussian Filters • implement decreasing influence by more distant pixels

8. original 33 99 1515 Gaussian Filters Effect of Gaussian smoothing:

9. -1 0 1 1 2 1 -2 0 2 0 0 0 -1 0 1 -1 -2 -1 Sx Sy Sobel Filters • Sobel filters are an example for edge detection filters. • Two small convolution filters are used successively:

10. Sobel Filters Sobel filters yield two interesting pieces of information: • The magnitude of the gradient (local change in brightness): • The angle of the gradient (tells us about the orientation of an edge):

11. Sobel Filters Original image (left) and result of calculating the magnitude of the brightness gradient with a Sobel filter (right)

12. Rao, Zelinsky, Hayhoe & Ballard (2002):Eye Movements in Iconic Visual Search Question: How do people represent items in their memory for efficient visual search? Idea: Iconic (appearance-based) multiscale representation Such representations were modeled using spatiochromatic convolution filters of different scales and orientations.

13. Rao, Zelinsky, Hayhoe & Ballard Convolution filters used for the model.

14. Rao, Zelinsky, Hayhoe & Ballard According to the model, iconic visual search proceeds as follows: • The first saccade is aimed at the point in the visual scene whose low-frequency features have the best match with the low-frequency features of the memorized object. • For the programming of the following saccades, higher and higher frequencies are included, until the target is found.

15. Rao, Zelinsky, Hayhoe & Ballard Coarse-to-fine scanning mechanism

16. Rao, Zelinsky, Hayhoe & Ballard Conclusion: Good correspondence between modeled and empirical scanpaths

17. Ballard, Hayhoe & Pelz (1995):Memory Representations in Natural Tasks Task: Copy a pattern of colored blocks

18. Ballard, Hayhoe & Pelz Possible strategies for completing the block copying task. Participants performed the following operations:(M)odel inspection, (P)ickup, and (D)ropoff.

19. Ballard, Hayhoe & Pelz Typical hand and gaze trajectories for a single copying step

20. Ballard, Hayhoe & Pelz Empirical frequency of individual strategies in the block copying task

21. Ballard, Hayhoe & Pelz Conclusions: • In the block copying task, working memory is only sparsely used. • Instead, subjects prefer to make additional eye movements. • Because eye movements are “inexpensive”, subjects use the visual scene as an “external memory” rather than building an internal representation of it.

22. Based on the previous study by Ballard et al, it seems that using working memory is clearly more “expensive” than performing eye movements. So maybe a “cost model” is an adequate way of describing and predicting behavior in visual tasks. Eye Movement - Working MemoryTradeoff (Inamdar & Pomplun, 2003)

23. The basic idea is that the visual system (including the cognitive mechanisms that are required for performing the task) optimizes visual behavior, i.e. minimizes its effort (cost). Is there such a tradeoff between the use of working memory and eye movements? If so, what exactly is minimized? Can this be quantified? Inamdar & Pomplun

24. Inamdar & Pomplun • Let subjects perform a visual task that requires eye movements and extensive use of visual working memory. • Vary the “cost” of eye movements. • Hypothesis: If the assumed tradeoff between eye movements and working memory exists, costlier eye movements should lead to increased use of working memory.

25. Stimuli in Experiment 1 • Subjects were presented with two columns of simple geometrical objects in three different colors and three different shapes. • The columns were identical except for one object that differed in either its color or its shape (in target-present trials). • Subjects had to indicate whether such a target was present or not. • The objects in the non-attended hemifield were always masked. • The cost of eye movements was varied by changing the distance between the two columns.

26. Stimuli in Experiment 1

27. Stimuli in Experiment 1

28. Stimuli in Experiment 1

29. Stimuli in Experiment 1

30. Eye Movements in Experiment 1

31. Eye Movements in Experiment 1

32. Results of Experiment 1

33. Results of Experiment 1

34. Results of Experiment 1

35. Experiment 2 • What happens if the capacity limit of visual working memory is reached? • By just varying the distance between columns, the cost of eye movements cannot be dramatically increased. • Idea: “Artificially” increase the cost of eye movements in the present paradigm by delaying the unmasking of objects after any gaze switch between hemifields.

36. Stimuli in Experiment 2 • We used the same stimuli as in Experiment 1, but only those for the “medium-distance” condition. • Three visibility delays were used: 0ms, 500ms, and 1000ms. • During the delays, objects in both hemifields were masked.

37. Results of Experiment 2

38. Results of Experiment 2

39. Results of Experiment 2

40. Conclusions • There clearly is a cost-minimizing behavior with regard to eye movements and working memory. • However, the current data does not allow to build a quantitative model of this phenomenon. • It seems that people slightly overestimate their working memory capacity when they are forced to heavily increase their working memory load.