Eye Movements & Change Blindness: What do they tell us?

1. Eye Movements& Change Blindness:What do they tell us? Ben J. Schlorholtz Psyc 736 Eye Movements, Theory & Applications 4-20-06

2. What is Change Blindness? • The inability to notice changes to visual scenes, sometimes even very large changes (Rensink et al. 1997) • Flicker paradigm demo http://www.usd.edu/psyc301/ChangeBlindness.htm

3. Change Blindness examples • Failure to detect changes to actors of central interest in … • short films (Levin & Simons, 1997) • real-world conversations (Simons & Levin, 1998) • http://viscog.beckman.uiuc.edu/djs_lab/demos.html • Perhaps the richness of the visual world is not as “detailed” as once thought

4. The missing link… • How rich are our mental representation across fixations? • Early studies simulated visual disruptions caused by blinks and saccades, BUT • There was no way to tell where the eyes were actually fixated • Perhaps changes were missed because people weren’t looking at the immediate area of change? • Remedied this by recording eye movements

5. Grimes (1996) • Can results from EM and reading studies be extended to more naturalistic scenes • McConkie & Zola (1979) – subjects were practically oblivious to changes made during saccades, but reading accuracy was maintained • Implemented a procedure similar to the flicker paradigm, used a Dual Purkinje Image Eye Tracker, and a saccade contingent paradigm • Subjects viewed 2-D magazine photographs for 10 secs and were instructed to… • Inspect each photo for a later memory test and report any changes detected while viewing • During a single saccade, a change occurred and remained for the duration of the viewing period

6. Grimes (1996) • Examples of changes included two men changing hats of a different style, a large parrot changing from bright red to bright green, 2 cowboys exchanging heads, a child enlarged by 30% in a playground scene, etc…

7. Grimes (1996), Results • 80% of photos changed however subjects only reported that 10-30% changed • Because EM’s were recorded, Grimes was able to separate out those who never fixated the object and those who did • For those that did fixate the object before and after a change, most were still oblivious to any changes (note: didn’t report any statistics) • Results were consistent with previous EM & reading literature • suggested that the visual detail retained across saccades was minimal • Grimes suggests that it’s simply more economical to mentally represent the overall object information rather than intricate details. • If more details are needed, one can simply re-fixate the object for deeper analysis

8. McConkie & Currie (1996) • Common criticism of the visual perception literature is the overuse of ecologically valid stimuli • Could results from research using simple dot patterns be extended to more naturalistic, color photographs • Ultimate goal was to determine if detection of changes made during saccades was the result of specific, local info acquired from the saccade destination; or the result of more global info regarding the mere position of the entire image

9. McConkie & Currie (1996) • Conducted a series of experiments using a fifth generation Dual Purkinje eye tracker (saccade detection within 10ms of initiation) • Instructed subjects to inspect scenes for a later recognition test and report any changes believed to be occurring

10. McConkie & Currie (1996) • In Exp. 1, the entire image was displaced either horizontally or vertically during predetermined saccades • Only finding replicated from earlier studies using simple stimuli was a decrease in detection as a function of increasing saccade distance

11. McConkie & Currie (1996) • In Exp 2, images now increased and decreased in overall size by 10 or 20% • Results suggest that the larger the change in image size, the increased likelihood of change detection • Overall, the authors conclude that findings where the result of inconsistent local information expectations. • Go on to suggest the saccade target theory which states that the visual system assumes consistency in the world during saccades. Changes are detected when there are any discrepancies between this assumption and the saccade target. • But with increasing saccade distance, the variability of the target area also increases • Important to the CB lit in the sense that they provide an explanation for visual stability across saccades using more complex stimuli than traditional visual perception research

12. Hayhoe, Bensinger, & Ballard (1997) • How do task demands influence fixation durations during performance of a sensorimotor task? • Subjects have 3 distinct areas to monitor and choose from • Model Area • Resource Area • Workspace Area • Task is to inspect the model and recreate it in the workspace using a computer mouse to drop-and-drag blocks from the resource area

13. Hayhoe, Bensinger, & Ballard (1997) • Capitalized on stereotypical EM’s • Model – Resource – Model - Workspace • A single blocks color was changed during 2 saccade conditions • Before Pickup • After pickup

14. Hayhoe, Bensinger, & Ballard (1997) • In the Before pickup condition, • If color information in the model is maintained through peripheral observation or memory from previous fixations, these changes should be noticed and evidenced by increased fixation durations • In the After pickup condition, • The color of the block just selected by the subjects becomes inconsistent with the model thus allowing the authors to observe any interference that might occur. • Because the manipulations occurring in each condition involve the exact same block changing color, the authors were… • Able to observe how the different task demands of each condition affect performance for identical fixation positions

15. Hayhoe, Bensinger, & Ballard (1997) • For the Before pickup condition, critical color information was not encoded prior to fixation • But for the After pickup condition, the color of the block being held was retained in memory to confirm its final location • Changing the color disrupted performance

16. Hayhoe, Bensinger, & Ballard (1997) • The ability to detect a change at identical fixation positions depends upon the immediate demands of the task at hand • The fact that we can detect a change in one circumstance, but not another lends evidence to the claim that our visual system does not represent everything mentally, but rather only those items which are immediately necessary

17. What is lacking thus far? • Up to this point in the CB literature, the verification of fixation position relative to the changing targets had not been adequately controlled for • Henderson & Hollingworth (1999) attempted to control for this by introducing changes which were entirely dependent upon the current fixation position

18. Henderson & Hollingworth (1999) • Using a dual Purkinje image eye tracker (1000Hz), changes were made to scene target areas in the following three conditions • Toward Condition • Object change occurred during the first saccade in which the eyes landed on the target • Away Condition • Object change occurred during the first saccade away from the target after it had been directly fixated • Control Condition • Object change occurred during the first saccade to a non-target area of the scene • 2 types of target changes were also included • Rotation Condition • 90 degrees on vertical axis • Deletion condition • Objects disappeared completely

19. Henderson & Hollingworth (1999) • Subjects were instructed to inspect each picture for a later memory test and respond immediately if any changes were detected • Hypothesized… • If fixation position is not involved in change detection as implied by the non-EM related CB lit, change detection rates should be equivalent among the 3 saccade conditions • BUT, if information is retained across previous fixations, detection performance should be better in the Away condition compared to the Towards and Control conditions, but this increase in performance should also decline as a function of saccade distance from the target

20. Henderson & Hollingworth (1999) • Poor change detection for Control condition • Suggests that the visual system does not create a global image representation by combining info from previous fixations • Many changes were not detected until re-fixation of the target region • Suggests that some info was maintained across saccades, but was inaccessible until eyes re-fixated the target

21. Henderson & Hollingworth (1999) • For object Deletions, as subjects saccaded away from target a steady decrease was apparent with increasing saccade length • Strikingly, this was not the case for saccades made towards the target area

22. Henderson & Hollingworth (1999) • For saccades toward the target, the fact that detection rates were equivalent for 1-8o suggests that the visual system is encoding certain information from upcoming saccade destinations. • This is also supported by low detection rates for both the saccades towards and away in the rotation condition • E.g. moderate changes in rotation do not appear to be encoded • Our visual system does not appear to recreate a global representation of scenes across saccades, but certain info is encoded and retained

23. O’Regan et al. (2000) • Can the changes people miss be replicated during blinks? • Blinks range from 100-200ms while saccades range from 20-70ms • Also, to what extent does attention to certain scene aspects affect the rate of change detection? • Operationalized attention by using 5 judges which rated objects in scenes • Central Interest object if chosen by 3 judges • Marginal Interest object if rated by none of the judges • Twas believed that by making changes to scene components in each interest category, they could manipulate the level of attention to certain objects • Objects of both interest groups were of comparable size

24. O’Regan et al. (2000) • Changes occurred during each blink, unbeknownst to the subject • Subjects were told to respond as soon as a change was detected • Used a dual Purkinje image eye tracker to determine how distant the eyes had to be from a change before it was missed

25. O’Regan et al. (2000), Results

26. O’Regan et al. (2000), Results • Authors used the info from these 2 graphs to form a ratio • # of changes detected after a blink at given eccentricity total # of changes/blinks at a given eccentricity

27. O’Regan et al. (2000), Results • Prob of detection as a function of eye position relative to the object • For eccentricity >2o, detection declines to only 10% and remains consistent • “seeing without looking” • BUT, prob of detection for direct fixation of changes was only 60% • “you don’t always see where you look” • Authors suggest that there must be other aspects of this fixated region to which attention is being allocated

28. Hollingworth, Schrock, & Henderson (2001) • Wanted to further examine the original CB studies by measuring EM’s • Conducted a series of experiments using a Flicker paradigm and a Dual Purkinje eye tracker to ask, “How does fixation position affect change detection performance?”

29. Hollingworth, Schrock, & Henderson (2001), Exp 1 • Subjects viewed naturalistic scenes and target objects were either • Deleted, then added • Rotated 90o • If detection is influenced by current fixation position, performance should improve the closer the eyes get to the target • Separated the scene into 4 visual regions • Target object region • Smallest circle around the object • Foveal region • 1o ring around the target object • Parafoveal region • 2.5o ring around the target object • Peripheral region • All other parts of the scene

30. Hollingworth, Schrock, & Henderson (2001), Exp 1 - Results • When the object was detected, the eyes were much more likely to be directly fixating it (74.5% across change conditions) compared to all other regions • At the moment of change detection, % of fixations directly on the target for Rotated objects (85%) was sig. higher than for Deleted/added objects (63%) • This suggests that Rotated objects were more difficult to identify and therefore required an increased # of direct fixations

31. Hollingworth, Schrock, & Henderson (2001), Exp 2 • Previous research has shown that visual attention is allocated to upcoming saccade target locations and thus is not entirely attributed to current fixation • Authors wanted to separate visual attention from current fixation position. How? • Used 2 conditions • No-movement • Maintain central fixation while searching for changes • Movement • Free movement of eyes to search for changes • If active fixation of a target is the single factor influencing detection performance, the no-movement condition should be practically impossible

32. Hollingworth, Schrock, & Henderson (2001), Exp 2 - Results • Subjects performed better in the Movement compared to the No-movement condition • More accurate (99% vs. 91%) • Quicker response latency (3.7sec vs. 6.3sec) • Fewer false alarms (0% vs. 18.1%) • Suggests that fixation position does play a significant role in change detection • BUT, the fact that performance was still well above chance suggests that fixation position is not the only critical factor utilized in flicker paradigm studies • For future studies, it’s imperative that fixation position be sufficiently controlled for

33. Hollingworth & Henderson (2002) • What types of info are preserved from objects that have previously been fixated? • There are some discrepancies between the CB lit and the Picture Memory lit • This area states that viewers are capable of encoding a substantial amount of visual info into a LTM store

34. Hollingworth & Henderson (2002) • Used a dual Purkinje eye tracker • Instructed subjects to inspect each scene for an upcoming memory test and press a button as soon as they detected a change • In this study a LTM test was actually conducted • 3 change conditions • Change after fixation • Change did not occur until the subject fixated the object, then saccaded away to a certain region • Change before fixation • Target object changed immediately prior to fixation • Control • No change occurred

35. Hollingworth & Henderson (2002)

36. Hollingworth & Henderson (2002) • To examine the type of information retained across saccades, objects changed via… • Type • Notepad changed to a floppy disk occupying same space • Token • Notepad changed to a spiral notebook • If the visual system maintains object representation after attention is withdrawn from that object, subjects should be able to detect both Type & Token changes

37. Hollingworth & Henderson (2002) Exp 1 - Results • Higher detection in the “Change After” compared to the “Change Before” fixation conditions • Type changes were detected sig. more than Token changes • Change detection appears to be highly dependent upon previous fixations • This might explain why detection performance was so poor in the original CB studies

38. Hollingworth & Henderson (2002) Exp 1 - Results • For the fixation duration data, sig. Positive correlation for the Token change detection • Influenced not only by previous fixation, but also by how long the target was fixated • For the Change After condition, 93% of fixations occurred upon re-fixation of the target • Authors suggest that re-fixating a target might serve as a trigger for retrieval of previously attended info • Poor performance in original CB studies might have been due to lack of re-fixation • LTM tests • 93% correct in the Type condition • 80% correct in the Token condition

39. Hollingworth & Henderson (2002) Exp 2 • Used only the Change After Fixation and Control Condition • Objects changed with respect to… • Token qualities • Rotational qualities • Only affects visual appearance and not object meaning • The ability of subjects to decipher between different rotational positions on LTM test would offer support for the maintenance of purely visual qualities in memory across fixations

40. Hollingworth & Henderson (2002) Exp 2 • Results were consistent with Exp 1 • Rotated objects were detected with just as much likelihood as Token changing objects • Purely visual information was retained after attention was distributed else where in the scene • Is it really? • Re-fixation of the target object sig. improved detection performance

41. Hollingworth & Henderson (2002) Exp 3 – Forced Choice Procedure

42. Hollingworth & Henderson (2002) Exp 3 – Results • Change detection performance was highly accuarate • Performance did NOT decrease as the # of fixations between target object and mask presentation

43. Hollingworth & Henderson (2002) Conclusions • Overall, data suggests that there is some buildup of visual information in some memory store that is retained across multiple fixations • Previous CB studies which used poor detection as evidence for a lack of representations across visual interruptions might not have been accurate • Visual representations do exist in some LTM store and it appears that re-fixating an object plays a significant role in detecting changes across scenes

44. Static vs. Dynamic CB • To this point, the stimuli used to investigate CB have been primarily static in nature (computer renderings, photographs, etc…) • Researchers have to wonder if the same types of findings would be found in more dynamic situations • Once such area could involve driving • “Looked, but didn’t see” accidents

45. Velichkovsky et al. (2002) • 3 types of occlusion modes • Blinks, blanks, and saccades • 2 viewing conditions • Static (still pictures of driving scenes) • Dynamic (driving simulator) • 2 types of changes • Relevant or Irrelevant (all were insertions or deletions) • Do the different occlusion methods influence detection performance under static and dynamic conditions • Used a SR-R Research Eyelink recorder • No hypotheses were provided

46. Velichkovsky et al. (2002) Results – Static condition • Detection performance was … • sig. higher for relevant changes (80%) compared to irrelevant changes (34%) • 150msec faster for relevant changes • No differences were found among the occlusion methods • Insertions were detected more often and quicker than deleted objects • Potentially hazardous objects are usually those that suddenly appear, although this could be argued • RT’s were 225msec longer during an occlusion compared to no occlusion • This suggests that critical events which take place during a visual disruption may require 1/5 sec more time to react

47. Velichkovsky et al. (2002) Results – Dynamic condition • Due to equipment delays, authors were only able to observe changes occurring during fixations & blanks (i.e. simulated blanks and saccades • Not an issue based on results of static condition

48. Velichkovsky et al. (2002) Results – Dynamic condition • Results were completely opposite what would be expected given the CB lit • Detection performance was… • More accurate for blanks than fixations (95% vs. 90%) • Faster for blanks than fixations (666ms vs. 702ms) • The fact that false alarm rates were also low served as evidence to the authors that the blank screen stimulus was not serving as a cue “when” to look for a change • Further research is needed to verify this finding

49. So what do EM’s tell us about CB? • Observed CB using very simple stimuli and 2-D naturalistic images and photos • Task Demands appear to play a sig. role • Allocation of attention changes quickly • Saccading toward and away from a target object can impact our ability to detect a change • Tells us more about the relationship between impending saccade targets and the allocation of attention • Continue to find our ability to detect a change decreases as a function of increasing fixation distance from a target

50. In my own opinion, • CB occurs not because people simply don’t look at changing objects • In O’Regan et al. (2000) subjects missed 40% of changes which were directly fixated • It’s not so much about visually inspecting a scene, but rather where in the scene attention is deployed • Perhaps our perceptual system uses attentional thresholds to regulate what is and is not explicitly detected • for changes not explicitly detected, subjects looked at some changing objects of a given quality more than others (Hollingworth et al., 2001) • The next logical step • CB applications in more dynamic environments