Here is Smith at sea on a foggy evening, and as lost as ever he can be. Suddenly the skies clear and Smith sees the Pol

1. What you see when you see a thing depends upon what you see the thing as…But what you see the thing as depends upon what you know about what you are seeing…

2. Here is Smith at sea on a foggy evening, and as lost as ever he can be. Suddenly the skies clear and Smith sees the Pole Star. What happens next? In particular, what are the consequences of what Smith perceives for what he comes to believe and do? That depends upon what he sees the Pole star as... If for example he sees the star that is the Celestial North Pole then Smith will know, to that extent, where he is -- and we may confidently expect the he will utter “saved!” and make for port. Whereas, if he sees the Pole Star, but takes it to be a firefly, then seeing the Pole Star may have no particular consequences for his behavior or his further cognitive states. Smith will be just as lost after he sees it as he was before.

3. COGNITIVE ASPECTS OF VISUAL PERCEPTION ANDPRODUCTION Marjorie Zielke October 6, 2004

4. Course Research Topics • Visual Acquisition of Knowledge • The Efficiency of the Visual • Cognitive Science as one framework

5. COGNITIVE ASPECTS OF VISUAL PERCEPTION ANDPRODUCTION • What is Cognitive Science? • Visual Processing vs. Cognitive Perception • How the Brain Processes Visual Images • Models of Thought – Cognitive Perception • Vision and the other Senses • How the blind “see” and the implications..Oliver Sacks, New Yorker Magazine, July 28, 2003 • Interesting, related research at UTD; School of Brain Sciences • Future Implications • Why have an interest in cognitive science as interactive developers? • How can we use knowledge of the brain to design projects? • Question and Comments

6. What is Cognitive Science? • Multidisciplinary area which combines fields of neuroscience, psychology, philosophy, anthropology, computer science • Definition: The study of intelligence and intelligent systems, with particular reference to intelligent behavior as computation. (Simon & Kaplan) • Also referred to as the study of the mind. • “Applied” Fields of study: Learning and Conceptual Organization, Computational Linguistics, Psycholinguistics, Cognitive Neuroscience, Human Computer Interaction in Aviation • Vicarious discipline…understand more about computer science/interactive development from studying the brain and understand more about the brain from studying computer science/interactive development. • Personal understanding is a work in progress

7. Vision vs. Perception

8. Vision

9. Optic nerve Optic chiasm Optic tract Lateral geniculate nucleus Optic radiation Primary visual cortex

14. Perception

15. Another approach to understanding visual perception is to develop models of how the fluctuating patterns of light reaching the eyes are processed to yield information about the surrounding world, without necessarily referring to any physiological mechanisms.

16. Ames Room

18. Cognitive Perception…Models of Thought • Bottom up, Top Down • Behaviorists: Observation • Empiricists: Perception requires the use of knowledge of the world to interpret the retinal image. • Gestalt • Marr • Constructionist • Connectionist

21. Gestalt • Gestalt – a number of principles of perceptual organization to describe how certain perceptions are more likely to occur than others. • Proximity: Things that are close together are grouped together • Similarity: Things that look similar are grouped together • Common Fate: Things that appear to move together are grouped together –flock of birds, school of fish • Good Continuation: perceive smooth curves rather than irregular forms • Closure: one will see a closed rather than an open figure.

22. Marr—He was going to die… • How do we transform a pattern of light on the retina into awareness of the visible world. • First: spatial array of values of light intensity and hue • Second: A symbolic specification of the positions, motions and identities of surround objects • How can the first representation be processed to obtain the second • Marr argued that visual process is modular, a number of separate subprocesses. • Assume that each processing stage draws on the representation created at the one before so that there is a one way flow of information. • Seems at odds with empiricists tradition, which holds that perception requires the use of knowledge of the world to interpret the retinal image. • But, Marr was not opposed to a role for knowledge or hypotheses in visual processing, and accepted a “top-down” or “conceptually driven” component. • However, his aim was to establish the limits of what could be achieved through a purely bottom-up or data driven analysis of visual input. • In this way, he argued, it would be possible to specify exactly the circumstances in which knowledge of the world is needed to resolve the ambiguities in visual processing, and so to avoid falling back on it as a general-purpose, poorly specified solution to problems in vision.

23. Marr used the term “computational theory” to describe this aspect of his approach to visual perception. The term does not mean a theory that is just something to do w/computers. Instead it expresses the specific and very powerful idea that the first stage in understanding perception is to identify the information that a perceiver needs from the world, and the regular properties of the world that can be incorporated into processes for obtaining that information. In other words, we need to know what computations a visual system needs to perform, before attempting to understand how to carry them out.

24. Connectionists • Neural Networks representations of the world are expressed in terms of activities that are based on the properties of neurons. • Many modeling techniques of this kind have been devised, using different networks and different “learning rules” governing the changes in the strengths of connections between them.

26. How the Blind SeeThree MemoirsFrom “The Mind’s Eye” Oliver Sacks, New Yorker Magazine, July 28, 2003 • John Hull • Zoltan Torey • Sabriye Tenberken

27. Synesthesia syn-es-the-sian. Physiol.Sensation produced at a point other thanor remote from the point of stimulation, as of a color from hearing acertain sound (fr. Gk, syn = together + aisthesis = to perceive). Synesthesia is an involuntary joiningin which the real information of one sense is accompanied by a perception in another sense. In addition to being involuntary, this additional perception is regarded by the synesthete as real, often outside the body, instead of imagined in the mind's eye. It also has some other interesting features that clearly separate it from artistic fancy or purple prose. Its reality and vividness are what make synesthesia so interesting in its violation of conventional perception. Synesthesia is also fascinating because logically it should not be a product of the human brain, where the evolutionary trend has been for increasing separation of function anatomically. R. Cytowic, "Synesthesia: A Union of the Senses" Springer-Verlag, NY (p.1)

28. The world of the blind…of the blinded…it seems, can be especially rich in such in-between states – the intersensory, the metamodal—states for which we have no common language.

29. Famous Study of Creativity: French Mathematician Jacques Hadamard asked many scientists and mathematicians, including Einstein about their thought process…

30. Einstein replied…”The physical entities which seem to serve as elements in thought are…more or less clear images which can be “voluntarily” reproduced and combined…”

31. “…Some are of visual and some of muscular type. Conventional words or other signs have to be sought for laboriously only in a secondary stage…”

32. “Nor was Einstein unique in this respect…found that almost all scientists work this way…

33. “…visualizing the brain as a “perceptual juggling act of interacting routines…”

34. Vision and the Other Senses

35. “There is increasing evidence from neuroscience for the extraordinarily rich interconnectedness and interactions of the sensory areas of the brain, and the difficulty, therefore, of saying that anything is purely visual or purely auditory, or purely anything.”

36. “What You See is What You Hear” • Vision is believed to dominate our multisensory perception of the world • However, auditory information can qualitatively alter the perception of an unambigous visual stimulus to create a striking visual illusion. • Findings indicate that visual perception can be manipulated by other sensory modalities

37. “What You See is What You Hear” • When a single visual flash is accompanied by multiple auditory beeps, the single flash is incorrectly perceived as multiple flashes. • Results obtained by flashing a uniform white disk for a variable number of times on a black background. • Observers were asked to judge how many visual flashes were presented on each trial.

38. “What You See is What You Hear” • Surprisingly, observers consistently and incorrectly reported seeing multiple flashes whenever a single flash was accompanied by more than one beep. • Figure 1b shows that observers performance was the same, irrespective of whether a single flash was accompanied by two beeps, or two flashes by one or no beeps, suggesting that the illusory double flash is perceptually equivalent to the physical double flash.

39. “What You See is What You Hear;” “Nature” Magazine;” Dec. 14, 2000

40. Applications

41. Recognizing moving faces:a psychological and neural synthesis • Alice J. O’Toole, Dana A. Roark, and Herve Abdi

44. Facial Motion…What is it and What is it for? • Rigid head motion can be interpreted as social interaction signals. • To begin or end a conversation, we turn our heads to look at, or away from, someone. • We redirect the attention of others with a head-turn and we nod to indicate agreement. • Rigid head movements provide the observer with a moving stimulus and with more views of the head than would be encountered from a static observer and subject.

45. Facial Motion…What is it and What is it for? • Non-rigid head movements can be grouped into speech production movements, facial expression movements and eye gaze changes. • The visual cues provided by the face during speech function to boost the intelligibility of speech. • Facial expression’s can convey a person’s mood. • Changes in the direction of eye gaze provide information about the object of attention. • Dynamic information contributes more in poor viewing conditions. Because facial structure is a more reliable cue to recognition than the dynamic identity signature, motion information is most beneficial when viewing conditions are not optimal for extracting the facial structure. • Face familiarity mediates the role of dynamic information in recognition. Because characteristic motions and gestures occur only intermittently, they are learned more slowly than static facial features. • The relative importance of motion information to recognition will increase, therefore, with a viewer’s experience with a face.

47. Facial Motion…What is it and What is it for? • Subjects can discriminate individuals on the basis of facial motion information alone.

48. Future Implications

49. “Some philosophers—professional artificial-intelligence critic Hubert Dreyfus for one—maintain that achieving human-level intelligence is impossible without a body.”The Age of Spiritual Machines,Ray Kurzweil, Penguin Books, 1999

50. All the Senses are Important in Creating Art…Particularly the Tactile Sense