Why do things look the way they do?

1. Why do things look the way they do? • Because that’s they way they really are!? That question is misleading because “how things look” is ill defined in the absence of further theoretical analysis. • In this class we will look at a somewhat related question: • What does the way things appear to us in our conscious experience tell us about how vision and mental imagery works and about the nature of the representations we construct when we see or visualize? • LaGrande Illusion is the belief that we have access to the form of representation we construct in episodes of cognition • Recent example: Wegner, in The Illusion of Conscious Will, Argues that what we are conscious of is not what causes our voluntary acts • ZP: “…the ‘illusion’ is really that the experience we have of causing certain things to happen explains why it happened”

2. This is what our conscious experience suggests goes on in vision…

3. This is what the demands of explanation suggests must be going on in vision…

4. Subjective experience suggests that when we see, we are examining an inner display Are there any reasons (other than conscious experience) for taking this view seriously? • The completion of percepts from partial information • Blind spot, resolution and color sensitivity • ‘amodal completion’ (most scene objects are occluded) • Completions have real behavioral consequences • ‘illusory contours’ and amodal completions act in many ways like real figures • Visual representations appear to be panoramic, stable, and in an allocentric frame of reference • The superposition theory has a lot of intuitive appeal

5. Completions … Where’s Waldo?

6. Standard view of saccadic integration by superposition

7. The superposition view fails O'Regan, J. K., & Lévy-Schoen, A. (1983). Integrating visual information from successive fixations: Does trans-saccadic fusion exist? Vision Research, 23(8), 765-768.

8. Visual interpretation is local Partly Ambiguous Unambiguous #1 Unambiguous #2

9. Visual interpretation is local

10. Visual interpretation is local Coordinating the interpretations among parts is more difficult if they are not both visible at the same time

11. When temporal integration occurs is shows the effect of memory load Higher memory load Lower memory load

12. When temporal integration occurs is shows the effect of memory load

13. When temporal integration occurs order of presentation is important

14. Errors in recall suggest how visual information is encoded Children have very good visual memory, yet often make egregious errors of recall • Errors in relative orientation often take a canonical form • Errors in reproducing a 3D image preserve 3D information

15. Errors in recall suggest how visual information is encoded • Children more often confuse left-right than rotated forms • Errors in imitating actions is another source of evidence

16. Ability to manipulate and recall patterns depends on their conceptual, not geometric, complexity • Difficulty in superimposing shapes depends on they are conceptualizedLook at first two shapes and superimpose them in your mind; then draw (or select one) that is their superposition

17. Many studies have shown that memory for shapes is dependent on the conceptual vocabulary available for encoding theme.g., recall of chess positions by beginners and masters

18. Finally, the phenomenology of seeing (including its completeness, its filled-out details, and its panoramic scope) turns out to be an illusion! • We see far less, and with far less detail and scope, than our phenomenology suggests • Objectively, outside a small region called the fovea, we are colorblind and our sight is so bad we are legally blind. The rest of the visual field is seriously distorted and even in the fovea not all colors are in focus at once. • More importantly, we register surprisingly little of what is in our field of view. Despite the subjective impression that we can see a great deal that we cannot report, recent evidence suggests that we cannot even tell when things change as we watch.

19. What changes between flashes? Harborside Airplane Helicopter Dinner Farm scene Paris corner

20. Where does this leave us? • Should we conclude that seeing is a process of constructing conceptual descriptions? • Most cognitive scientists and AI people would say yes, although there would be several types of exception. • There remains the possibility that for very short durations (e.g. 0.25 sec) there is a form of representation very like visual persistence – sometimes called an ‘iconic storage’(Sperling, 1960). • From a neuroscience perspective there is evidence of a neural representation in early vision – in primary visual cortex – that is retinotopic and therefore “pictorial.” Doesn’t this suggest that a ‘picture” is available in the brain in vision? • We shall see later that this evidence is seriously misleading and does not support a picture theory of working memory or LTM • A major theme of this course will be to show that an important mechanism of vision is notconceptual but causal: Visual Indexes • Many people continue to hold a version of the “picture theory” of mental representation in mental imagery. More on this later.

21. Conceptual & Methodological Problems in the study of Visual Perception • There is a problem about what it means to see: Ordinary usage confounds: seeing X, seeing X as R, seeing that P, and believing that P. Distinguishing these is a cottage industry in philosophy. • Some examples of the prescientific sense of see and why it is a problem (Droodles, “looks like”) Examples of droodles • Ordinary language also confounds the content and the vehicle of perception: what perceptual representations are about and what properties the representations themselves have • There is a serious problem about what to make of phenomenology – about the conscious experience of seeing. In view of how misleading subjective experience can be, and in view of the fact that what progress there is cognitive science has been made by ignoring the distinction between processes of which we are consciously aware and those of which we have no conscious experience. Cognitive Science, unlike philosophy, has tended to shy away from the evidence of introspection.

22. Conceptual & Methodological Problems in the study of Visual Perception • Part of the puzzle about vision rests on the failure to make certain distinctions (such as those alluded to earlier: between seeing and believing, between content and form). These two distinctions will play a major role in our later discussion. • The question posed at the beginning of this lecture, Why do things look the way they do? asks why the appearance is conceptualized (or described) a certain way. The difficulty with this idea is dramatized by the Wittgenstein story.

23. What does “how things look” mean? • We speak of someone “looking sick,” or “looking happy” or of the weather “looking like it’s going to snow.” This informal sense of looking is too diverse to be useful. • A story due to Wittgenstein (as told by Tom Stoppard), goes: Meeting a friend in the hall, a philosopher says, “Tell me, why do people always say it was natural for men to assume that the sun went around the earth rather than that the earth was rotating?” His friend said, “Well, obviously, because it just looks as if the sun was going round the earth.” To which the philosopher replied, “Well, what would it have looked like if it had looked as if the earth was rotating?” • The problematic nature of How Things Look will become even more apparent when we discuss mental imagery later. As a take-home exercise you might think about “Why do mental images look like what they are images of?”

24. Conceptual & Methodological Problems in the study of Visual Perception • An important distinction that I will address only briefly (but is covered in Chapters 2 and 3) is between seeing and coming to have a belief about what one is seeing. This is crucial to my claim about the Cognitive Impenetrability of Vision or the view of vision as an architectural module. • The development of Signal Detection Theory alerted us to the fact that signal detection involves at least two stages: a detection stage and a response selection or decision stage, the latter of which is the only stage thatiscognitively penetrable.