Is perception continuous with cognition?

1. Is perception continuous with cognition? The Cognitive Impenetrability of Vision Read Seeing & Visualizing Chapter 2 or the BBS article on my web site: ruccs.rutgers.edu/faculty/pylyshyn.html

2. The accepted answer goes along with intellectual (and political) fashions The zeitgeist in the second half of the 20th century was one of populist values which emphasized egalitarianism and the limitless possibility of the human mind. In keeping with this spirit, many scholars mistakenly repudiated innateness and, emphasizing plasticity, embraced learning as the defining character of human nature. It was in this era that the belief arose that everything we perceived or thought was dependent on our cultural or linguistic context. Hence the popularity of Worf and Sapir, of Bruner’s New Look in perception as well as deconstruction movements in European (and, of course, Californian) thought.

3. Through the last half of 1900s both the general public and the social science community assumed that perception and cognition were continuous – that you could not distinguish between the two. Bruner’s New Look in Perception: Perception modeled on Science – hypothesis formation, verification & modification Bruner, J. S. (1957). On perceptual readiness. Psychological Review, 64, 123-152. Effect on Philosophy of Science: Feyerabend, Hanson, Kuhn (These philosophers assumed that there is no “innocent eye” so all observations are “theory-laden”. Quine’s “The myth of the given”) The cultural ethos gave strong support to the view that the mind is highly plastic and at the mercy of the environment

4. What are some reasons for thinking that vision is cognitively penetrable (and not modular) • Expectation and the perception of patterns • Perception in noise of words, sentences, and probable sequences • Assimilation of perception to the norms (Postman) • Visual recognition as involving the framing hypotheses (Potter) • Perceptual learning and expertise (bird watchers, wine tasters…) • Apparent motion and “problem solving” in vision (Rock) • The effect of hints and foreknowledge on fragmented figures or stereograms • Neuroscience evidence for top-down effects • The experience in computer vision (Heterarchy) • Why should we doubt the continuity thesis? • Illusions…. • Methodological concerns (signal detection theory)

5. What are some reasons for thinking that vision is cognitively penetrable (and not modular) • Expectancy and the perception of patterns • Perception in noise of words vsnonwords, sentences, statistical properties of sequences • Assimilation of perception to the norms (Postman) • Explanation in terms of framing hypotheses (Potter) • Perceptual learning (bird watchers, wine tasters…) • Apparent motion and “problem solving” in vision (Rock) • The effect of hints and foreknowledge on closure figures or stereograms • Neuroscience evidence for top-down effects • The experience in computer vision (Heterarchy –Shirai) • Why should we doubt the continuity thesis? • Illusions…. • Methodological concerns (signal detection theory)

6. Some reasons to assume that vision is cognitively penetrable • Informal • Magic tricks • The proofreader’s problem • Speech perception • Phonetic monitoring • Cross modal effect on hearing (listening while reading) • Phoneme restoration effect

7. Most people believe that we when we read we make predictions about the word next which is whywhy we often substitute the wrong word for an unexpected word and is also why the predictability index (the ClozeScore) is a useful measure of readability, widely adopted in the past by newspapers and language teachers.

8. Is a mental image something that is seen (with the mind’s eye)? If you think that mental imagery involves (uses) visual processes, it is important to know which aspects or stages or visual functions it uses. If it involves reasoning and inferences based on what is sees then it’s not an interesting thesis since mental imagery is assumed to be a modality of thought. In that case the thesis turns out to be the claim that imagery shares cortical resources with thought which nobody doubted. If, on the other hand, visual perception is used to interpret mental images then this is a strong thesis since it suggests that images themselves are (or can be) seen.

9. Seeing Mental Images • The central tenet of the Picture Theory of mental imagery is that mental images constitute the same kind of spatial representations as are found on, say, the retinal or primary visual cortex. • If, on the other hand, a mental image is an already-interpreted form of representation, this would support the thesis I am putting forward; that images are only phenomenologically picture-like while in their function (i.e., in their causal role) they are conceptual and therefore symbolic descriptions. • This not only makes a big difference in one’s view about the mind but also has repercussions for the treatment of reports of conscious experience in scientific theories.

10. Is vision like science itself? • Does vision involve hypothesis formation and testing, as was once believed to be the method used in science? • Potter and Bruner’s hypothesis testing experiment • If the answer is YES then the question whether mental imagery uses vision becomes circular (or empty) since clearly vision serves thought by providing new information. • So the question whether there are general top-down hypotheses-proposing or hypothesis-tendering) effects becomes central.

11. Familiarity and the reconstruction of partially hidden patterns

12. Familiarity and the reconstruction of partially hidden sounds • Signal detection theory helps to isolate stages in information processing • Signal detection theory helps to isolate slages in information processing VERNALIT INTERVAL TRLAVNEI

13. Meaning and perception of phonemesThe ‘phoneme restoration effect’ • The pitcher’s thoughts about the dangerous batt▒ made him nervous • The soldier’s thoughts about the dangerous batt▒ made him nervous

14. How to get someone to see fragmented figures?

15. Autostereogram or Magic Eye® figures

16. Apparent motion is a function of early vision, yet is subject to various “intelligent” interpretations

17. In the “Ternus Configuration” short time delays result in “single element motion” (the middle object persists as the “same object” so it does not appear to move)

18. Long time delays result in “group motion” because the middle object does not persist but is perceived as a new object each time it reappears

19. But long delays, when the disappearance appears to be due to occlusion by an opaque surface, maintain objecthood, and therefore behave like short delays

20. But long delays, when the disappearance appears to be due to occlusion by an opaque surface, maintain objecthood, and therefore behave like short delays

21. Apparent “problem solving” in vision

22. Does the circle appear to move or are there two circles being covered?

23. Does the circle appear to move or are there two circles being covered?

24. Apparent “problem solving” in vision (Rock)

25. Another example from Irving RockBecause the rectangle is seen as transparent, the circle is seen to move in apparent motion

26. Here the rectangle is seen as opaque so it ‘explains’ the circle’s visible-invisible cycle. Therefore the circle is not seen to move.

27. Many reasons for thinking that vision is cognitively penetrable • Expectancy and the perception of patterns • Perception in noise of words vs nonwords, sentences, statistical properties of sequences • Assimilation of perception to the norms • Explanation in terms of readiness: Seeing as… • Perceptual learning (bird watchers, wine tasters…) • The effect of hints and foreknowledge on closure figures or stereograms • Neuroscience evidence for top-down effects • The experience in computer vision • Why should we doubt the continuity thesis? • Illusions…. • Methodological concerns (signal detection theory)

28. Centrifugal neural pathways • There are almost as many outward (efferent) nerve fibers as inward fibers • There is evidence of top-down control of sensors and top-down effects on percepts (e.g., filling-in effect for blind spot and other scotomas) • Early attentional gating of a cat’s auditory signals Hernandez-Péon, R., Scherrer, R. H., & Jouvet, M. (1956). Modification of electrical activity in the cochlear nucleus during "attention" in unanesthetized cats. Science, 123, 331-332.

29. Many reasons for thinking that vision is cognitively penetrable • Expectancy and the perception of patterns • Perception in noise of words vs nonwords, sentences, statistical properties of sequences • Assimilation of perception to the norms • Explanation in terms of readiness: Seeing as… • Perceptual learning (bird watchers, wine tasters…) • The effect of hints and foreknowledge on closure figures or stereograms • Neuroscience evidence for top-down effects • The experience in computer vision • Why should we doubt the continuity thesis? • Illusions…. • Methodological concerns (signal detection theory)

30. Early experience in computational perception (and AI) suggested that knowledge-based perception leads to better performance Minsky & Papert’s: “Heterarchy, not hierarchy” or the knowledge-based approach to perception, Shirai’s success in building an edge detector and other model-based vision systems. Riseman & Hanson (1987): “It appears that human vision is fundamentally organized to exploit the use of contextual knowledge and expectations in the organization of spatial primitives… Thus the inclusion of knowledge-driven processes at some level in the image interpretation task, where there is still a great degree of ambiguity in the organization of the visual primitives, appears inevitable (286).”

31. The Blackboard Architecture used in many AI applications is highly non-modular because all parts can communicate with one another. e.g. ‘Hearsay’ speech recognition system

32. Pandemonium An early architecture, similar to the blackboard architecture, was proposed by Selfridge in 1959. This idea continues to be at the heart of many psycho-logical models, including ones implemented as neural net (or connectionist) models.

34. On the other hand …. • From a function perspective it makes sense that the earliest stage of vision should be built to be fast and very often (though not necessarily always) veridical. • The parable of the blind clockmaker and Simon’s “partially decomposable systems” • The influence of David Marr’s Principle of least commitment: Do not do something that you may later wish to undo (e.g., depth first search)

35. The beginnings of a modular view in computer vision David Marr (1982): “The principle of least commitment… requires not doing something that may later have to be undone, and I believe that it applies to all situations in which performance is fluent. It states that algorithms that are constructed according to a hypothesize-and-test strategy should be avoided because there is probably a better method.”

36. What evidence is there that vision is a modular process? • There is a lot of prima facie evidence that vision works independently of what we believe and what we expect. • In order to explain why that seems to be so we need to distinguish between the part of vision that is unique to vision and the part that is shared by all intellectual processes. The unique part is called Early Vision.

37. Irvin Rock produced a lot of the evidence that is cited in support of the view that perception involves “Inference” and “Problem Solving” in order to account for the visual input, but he also says: “The major difference between perception and thought is that perception is based on a rather narrow range of internalized knowledge, as far as inference and problem solving are concerned… Perception must rigidly adhere to the appropriate internalized rules, so that it often seems unintelligent and inflexible in its imperviousness to other forms of knowledge (p 340)”. From: Rock, I. (1983). The Logic of Perception. Cambridge, Mass.: MIT Press.

38. Illusions don’t depend on what you believe!

39. Do hints speed up closure of fragmented figures?According to Reynolds (1985) the only thing that makes a difference to ease of closure is knowing that there is a sensible reading of the fragmented figure. Knowing the name of the figure does not help.

40. Meaning and difficult percepts What helps us see them? • Fragmented figures; autostereograms (“magiceye”); Random dot stereograms • Category hints? • Description? • Model of what you should see? • Knowing where to look/attend?  • Knowing they are ambiguous?  • Having seen them once before?  Saye, A., & Frisby, J. P. (1975). The role of monocularly conspicuous features in facilitating stereopsis from random-dot stereograms. Perception, 4(2), 159-171. Frisby, J. P., & Clatworthy, J. L. (1975). Learning to see complex random-dot stereograms. Perception, 4(2), 173-178.

41. Amodal completion is automatic and non-inferential (not rational) What would the figures look like if the black square occluders were removed? Try to see through them.

42. Independence of feature detection and object (pattern) perception Evidence from brain-damage: visual agnosic with spared nonvisual pattern-recognition[Humphreys & Riddoch, 1987. To see but not to see: a case study of visual agnosia] • This patient could had severe agnosia and could not visually recognize familiar things (including his wife’s face) or discriminate shapes. • But he had normal eye movements and sensory abilities (including stereo and motion detection) • He could see local features and, with enough time and effort, could often infer the identity of the object (just as the New Look suggests) • He could describe and draw objects from memory and could recognize objects by touch, so his pattern memory was normal • It “supports the view that the perceptual representation used in this matching process can be ‘driven’ solely by stimulus information, so that it is unaffected by contextual knowledge.” H&R, p104)

43. Signal Detection Paradigm If correct detection improves without increase in false positives it’s an increase in sensitivity. If it improves but so does the false positive rate it suggests an increase in bias towards acceptance.

44. Examples from the language module • Phoneme restoration effect appears to be a response bias effect • Lexical ambiguity appears to be resolved after a period of time, before which all options are available.

45. Meaning and perception of phonemes • The pitcher’s thoughts about the dangerous batter made him nervous • The pitcher’s thoughts about the dangerous battle made him nervous • The pitcher’s thoughts about the dangerous batt▒ made him nervous • The soldier’s thoughts about the dangerous batter made him nervous • The soldier’s thoughts about the dangerous battle made him nervous • The soldier’s thoughts about the dangerous batt▒ made him nervous • Signal detection analysis of responses shows that the effect is connected to the response selection stage* *Samuel, A. G. (1981). Phonemic restoration: Insights from a new methodology. Journal of Experimental Psychology: General, 110(4), 474-494.

46. Maybe cognition has a post-perceptual selection function? Swinney study of resolution of lexical ambiguity Visual reading task at ▲: Ant (related), Spy (inappropriate), bag (unrelated)

47. Swinney ambiguous-word priming experiment • No context:Rumor had it that for years the government building had been plagued with problems. The man was not surprised when he found several bugs▲ in the corner of his room. • Context biased to insects:Rumor had it that for years the government building had been plagued with problems. The man was not surprised when he found several spiders, roaches, and other bugs▲ in the corner of his room. • Context biased to spying:Rumor had it that for years the government building had been plagued with problems. The man was not surprised when he found several microphones, recorders, cameras, and other bugs▲ in the corner of his room. At points marked with ▲ a word was presented visually and subjects had to decide as fast as possible whether it was a word (a lexical decision task, where half of the time it was not a word). Examples of these words are ant, spy or sew. Nonwords were formed from the same letters: tna, ysp, swe.

48. Swinney ambiguous priming experiment • Swinney found that both senses of the ambiguous word (eg “bug”) primed the decision task – so both spy and ant were primed relative to the neutral word (sew) • The priming effect for the inappropriate sense of the word disappeared after about 0.7 to 1.0 seconds. After this only words related to the appropriate sense were primed. Swinney, D. A. (1979). Lexical access during sentence comprehension: (Re)consideration of context effects. Journal of Verbal Learning & Verbal Behavior, 18(6), 645-659.

49. Visual Detection of Anomalous objects Biederman

50. Alteration of perception with practice: The case of expert perceivers • Visual expertise often arises from learning what to attend to(pre-visual) as well as which patterns are diagnostic (post-visual). • Shiffrar & Biederman study of expert chicken sexers • Perception & recall of board positions by chess masters • Studies of athletes’ perception