Why do things happen the way they do in your imagination?

1. Why do things happen the way they do in your imagination? • Is it because of the format of your image or your cognitive architecture? Or because of what you know? • Did it reveal a capacity of mind? • Or was it because you made it do what it did? • Can you make your image have any properties you choose? Or behave in any way you want? Why not? • How about imagining an object from all directions at once, or from no particular direction? • How about imagining a 4-dimensional object? • Can you imagine a printed letter which is neither upper nor lower case? A triangle that is not a particular type?

2. More on Representation of space and arguments from neuroscience • The intuition that images are spatial (or, as some put it, that they “have space”) is an interesting one and deeper than most other questions being studied in laboratories. I have discussed this in excruciating detail in the last chapter of my “Things and Places”. • According to Kosslyn, the argument has now entered the third and final stage, where neuroscience evidence has put an end to the debate. I will look at some of this alleged debate-ending evidence and will show that it is characterized by desperation and lack of critical analysis.

3. A brief look at whether we may be storing information in the form of an icon • For present purposes I want to look at the issue of explanation and I will do this by considering the sense in which spatial information may (must?) itself be spatial • This matters to the question of what forms of representation we have in our minds and how these refer to things in the world.

4. Our studies of mental scanning (Pylyshyn & Bannon. See Pylyshyn, 1981) There is even reason to doubt that one can imagine scanning continuously (Pylyshyn & Cohen, 1998)

5. Major Question 2: Does visual imagery use the visual system? • It depends on what you mean by use and visual system • Touse vision can mean actually to perceive the image • Only early vision is relevant to this use of “perceive”, since general vision can involve all of cognition • This question is of interest to picture theorists because if vision is involved it may suggest that images are uninterpreted picture-like (or depictive) spatial displays. The visual module cannot be applied to already-interpreted data structures • Let’s assume that “the visual system” is “active” during mental imagery as some have suggested • What follows from that? • Why should we believe that vision may be involved? • In the last 15 years the main support for the assumption that vision is involved has come from neuroscience.

6. Reasons for thinking that images are interpreted by the visual system • Similar phenomenology of imagining & seeing • This reason overshadows all others • Superposition & interference studies • Visual illusions with projected images • The ubiquitous role of attention • Re-perceiving and novel construals • A large but very problematic literature

7. More demonstrations of the relation between vision and imagery • Images constructed from descriptions • The D-J example(s) • The two-parallelogram example • Amodal completion • Reconstruals: Slezak

8. Can images be visually reinterpreted? • There have been many claims that people can visually reinterpret images • These have all been cases where one could easily figure out what the combined image would look like without actually seeing it (e.g., the J – D superposition). • Pederson’s careful examination of visual “reconstruals” showed (contrary to her own conclusion) that images are never ambiguous (no Necker cube or figure-ground reversals) and when new construals were achieved from images they were quite different from the ones achieved in vision (more variable, more guessing from cues, etc). • The best evidence comes from a philosopher (Slezak, 1992, 1995)

9. Slezak figures Pick one (or two) of these animals and memorize what they look like. Now rotate it in your mind by 90 degrees clockwise and see what it looks like.

10. Slezak figures rotated 90o

11. Connect each corner of the top parallelogram with the corresponding corner of the bottom parallelogram Do this imagery exercise:Imagine a parallelogram like this one Now imagine an identical parallelogram directly below this one What do you see when you imagine the connections? Did the imagined shape look (and change) like the one you see now?

12. Amodal completion by images?

13. Is this what you saw?

14. Off-retinal info different from foveal info

15. Off-retinal info different from foveal info

16. Labels propagate over picture

17. Anorthoscope: Scan view How many contours are there?

18. Anorthoscope: Scan view What is the shape?

19. Vision is involved when images are superimposed onto vision • Many experiments show that when you project an image onto a display the image acts very much like a superimposed display • Shepard & Podgorny, Hayes, … • Interference effects (Brooks) • Mental scanning • Interaction with the motor system (Finke)

20. Shepard & Podgorny experiment Both when the displays are seen and when the F is imagined, RT to detect whether the dot was on the F is fastest when the dot is at the vertex of the F, then when on an arm of the F, then when far away from the F – and slowest when one square off the F.

21. Brooks’ spatial interference study Respond by pointing to symbols in a table or by saying the words left or right

22. Visual-Motor adaptation and motor adaptation to images • The basic prism adaptation setup: arm movement towards a target while wearing prism glasses • Now repeat with arm unseen but subject toldwhere it is (actually where it would have been in the prism case) • Get adaptation {Finke, R. A. (1979). The Functional Equivalence of Mental Images and Errors of Movement. Cognitive Psychology, 11, 235-264.} • But in the original experiment you don’t need to see a hand, any indicator of where the hand is will do as long as the subject believes it is where indicated

23. Standard view of saccadic integration by superposition Is it plausible? Is it true?

24. Superposition does not work(O’regan 1983)

25. This is what our conscious experience suggests goes on in vision… Kliban

26. This is what the demands of explanation suggests must be going on in vision…

27. Are there pictures in the brain? • There is no evidence for cortical displays of the right kind to explain visual or imaginal phenomena

28. Is there very short iconic storage? • Although the idea of pictorial long-term memory is not supported, there is some provisional evidence that sensory information outlasts the duration of the stimulus. Many people have studied these “sensory buffers” including George Sperling and Michael Posner.

29. Sperling’s partial report method for showing an iconic memory

30. Posner’s demonstration of short lasting shape information Fast Slower Slower Fast

31. So what does the visual system produce? • The best hypothesis so far (i.e., the only one that has not been shown to be clearly on the wrong track) is that the brain is a species of computer in which representations of the world are encoded in the form of symbol structures, and actions are determined by calculations (i.e., inferences) based on these symbolic encodings. • Therefore vision (and other sensory systems) compute symbolic expression for the rest of the (Turing) computing machinery to use.

32. So why does it not feel like we are doing computations? • The content of our conscious experience is a very poor guide to what is actually going on that causesboth our experiences and our behavior. Science is concerned with causes, not just correlations. • We have learned (since Galileo) that we can’t assume that the way things seem has much to do with how it works (consider the example of language understanding) • As in most sciences, the essential causes are far from obvious (e.g., How can the moon exert a pull on the earth without contacting it? What is this table made of? etc.). • In the case of cognition, what is going on is a delicate mixture of the obvious (what Granny or Shakespeare knew about people and why they do what they do) and the incredible.

33. We can’t even be sure that we have the right methods or instruments

34. Mental Scanning • Some hundreds of experiments have now been done demonstrating that it takes longer to scan attention between places that are further apart in the imagined scene. In fact the relation is linear between time and distance. • These have been reviewed and described in: • Denis, M., & Kosslyn, S. M. (1999). Scanning visual mental images: A window on the mind. Cahiers de Psychologie Cognitive / Current Psychology of Cognition, 18(4), 409-465. • Rarely cited are experiments by Pylyshyn & Bannon which I will summarize for you.

35. Studies of mental scanningDoes it show that images have metrical space? (Pylyshyn & Bannon. See Pylyshyn, 1981) • Conclusion: The image scanning effect is Cognitively Penetrable • i.e., it depends on goals and beliefs, or on Tacit Knowledge.

36. What is assumed in imagist explanations of mental scanning? • In actual vision, it takes longer to scan a longer distance because real distance, real motion, and real time is involved, therefore this equation holds due to natural law: Time =distance speed But what ensures that a corresponding relation holds in an image? The obvious answer is: Because the image is laid out in space! • But what if that option is closed for empirical reasons? • Imagists appeal to a “Functional Space” which they liken to a matrix data structure in which some cells are adjacent to other cells, some are closer and others further away, and to move from one to another it is natural that you pass through intermediate cells • Question: What makes these sorts of properties “natural” in a matrix data structure?

37. Thou shalt not cheat • There is no natural law that requires the representations of time, distance and speed to be related according to the motion equation. You could equally easily imagine an object moving instantly or according to any motion relation you like, since it is your image! • There are two possible answers why the relation Time =Representation of distance Representation of speed typically holds in an image-scanning task: • Because subjects have tacit knowledge that this is what would happen if they viewed a real display, or • Because the matrix is taken to be a simulation of a real-world display, as it often is in computer science

38. Mental Rotation has been one of the most cited demonstrations of all • Look at the following 3D figures and judge which pairs are the same except for orientation. The other pair are enantiomorphs – 3D mirror images so they can’t be put into correspondence by 3D rotation only.

39. Mental rotation Time to judge whether (a)-(b) or (b)-(c) are the same except for orientation increases linearly with the angle between them (Shepard & Metzler, 1971)

40. What do you do to judge whether these two figures are the same shape? Is this how the process looked to you? When you make it rotate in your mind, does it seem to retain its rigid 3D shape without re-computing it?

41. Thou shalt not cheat • What happens in ALL imagist accounts of phenomena from mental scanning to mental rotation is that they assume the properties of real space in order to provide a principled explanation, then retreat to something not-quite-real space when it is pointed out that they are assuming that images are laid out in real space-in-the-head. • This happens with mental rotation as well, even though the tacit knowledge account is not plausible there (it is an involuntary and universal way of solving the rotated-figure task so long as the task involves tokens of enantiomorphs).

42. The missing bit of logic: • According to Prinz (2002) p 118,“If visual-image rotation uses a spatial medium of the kind Kosslyn envisions, then images musttraverse intermediate positions when they rotate from one position to another. The propositional system can be designed represent intermediate positions during rotation, but that is notobligatory.” • Given that this happens in 3D so that it can’t be a literal brain space, the question arises: What makes this obligatory in “functional Space”?

43. Sources of obligatory constraint • It could be that there is a medium or a set of analog properties that together happen to simulate a virtual space, and the physical properties of this medium enforce the continuity of motion through it. This is very unlikely. • It could be that the constraint comes from the fact that it holds in the world. Then its transfer from the real to the mental world occurs either ; • Voluntarily, because we know how it would happen and we can use that fact to solve the problem • Naturally, because the constraint got built in over time through evolutionary pressures • Naturally, but not because the constraint is built in, but because of other properties of the architecture that make it more efficient to compute the rotated shape incrementally until there is a match

44. Do images have low-level psychophysical properties? • Bring the bars closer and closer together. In which can you see the bars when the spacing is closest? • In experiments, it was shown that the oblique effect occurs in mental images just as it does in vision • Kosslyn, Thompson & Ganis (2006) argued that this is because there are more vertical and horizontally tuned cells than obliquely tuned cells in visual cortex. Does this explain the finding?

45. Do images have low-level visual properties? • Imagine a grating in which the bars are: • Horizontal • Vertical • Oblique (45°)

46. A final point… • In Kosslyn, Thompson & Ganis (2007) the authors cite Ned Block to the effect that one does not need an actual 2D surface, so long as the connections upstream from the cortical surface can decode certain pairs of neurons in terms of their imagined distance. Imagine long stretchy axons going from a 2D surface to subsequent processes. Now imagine that the neurons are randomly moved around so they are no long on a 2D layout. As long as the connections remain fixed it will still behave as though there was a 2D surface. • Call this the “encrypted 2D layout” version of literal space.

47. The encrypted-spatial layout alternative • By itself the encrypted-layout alternative will not do because without referring to the original 2D locations, the relation between pairs of neurons and scan time is not principled. In the end the only principle we have is Time=distance/speed so unless the upstream system decrypts the neuron locations into their original 2D surface locations the explanation for the increase in time with increased imagined distance remains a mere stipulation. It stipulates, but does not explain why, when two points are further away in the imagined layout it takes longer to scan between them or why scanning between them requires that one visit intermediate locations along the way. • But this is what we needed to explain! One can apply such a mere stipulation to any form of representation. What was a principled explanation with the literal 2D display has now been given up for a mere statement of how it shall be.

48. The ‘Imagery Debate’ Redux • According to Kosslyn there have been 3 stages in the debate over the nature of mental images: • The role of images in learning and memory (Paivio’s Dual Code theory). Influential at the time but now abandoned except for a few recidivists like Barsalou. • Spatial properties of images and their role in dynamic processes, as assessed by reaction time measures (Kosslyn’s research on ‘metric properties of images’) • Discovery of brain mechanisms underlying visual imagining, which led to ‘the resolution of the imagery debate’.

49. Explaining mental scanning, mental rotation and image size effects in terms of “functional space” • When people are faced with the natural conclusion that the “iconic” position entails space (as in scanning and size effects) they appeal to “functional space” • A Matrix in a computer are often cited as an example • Consider a functional space account of scanning or of mental rotation: • Why does it take longer to scan a greater distance in a functional space? • Why does it take longer to rotate a mental image a greater angle?

50. But there are examples of solving geometry problems easily with imagery • There are many problems that you can solve much more easily when you imagine a layout than when you do not. • In fact many instances of solving problems by imagining a layout that seem very similar to how would solve them if one had pencil-and-paper. • The question of how pictures, graphs, diagrams, etc help in reasoning is very closely related to the question of how imagined layouts function in reasoning. That is not in question. What is in question is what happens in either the visual or imagined cases and how images can benefit from this processes even though there is no real diagram.