Brain related semantics

1. Brain related semantics CSCTR – Session 8 Dana Retová

2. Grounded cognition • Rejects the standard view that amodal symbols represent knowledge in semantic memory • Cognition shares the same mechanisms with perception, action and introspection • Simulation • A core form of computation in the brain • Reenactment of perceptual, motor and introspective states acquired during experience • As experience occurs, the brain captures the states across modalities and integrates them with a multimodal representation stored in memory

3. Mental imagery vs. amodal symbols • Modal representation and imagery representing knowledge • Epicurus, Kant, Reid • Behaviorists • Imagery not sufficiently scientific • Cognitivists • Amodal representation (feature lists, semantic networks, frames) • Elegant and powerful formalisms for representing knowledge • Could be implemented in AI

4. Problem with traditional AI • No evidence supports the presence of amodal symbols in cognition • Grounding problem • Traditional theories fail to explain how cognition interfaces with perception and action • Problem where the brain stores amodal symbols • How is it consistent with neural principles of computation?

5. Grounded cognition theories • Simulation • Situated action • Bodily states • Modal representations are central to knowledge

6. Theories of Grounded Cognition • Cognitive Linguistics Theories • Lakoff & Johnson (1980, 1999) • Abstract concepts are grounded metaphorically in embodied and situated knowledge • Theories of situated action • Gibson (1979) • Role of environment in shaping cognitive mechanisms • Coupling of perception and action during goal achievement • Social interaction • Research in robotics • Dynamic systems as preferred architecture • Fixed representations do not exist

7. Cognitive Simulation Theories • Memory theories • Glenberg (1997) • Memory is not just passive storage of information • Perception of relevant objects triggers affordances for action stored in memory • Reasoning about future actions relies on remembering affordances while suppressing perception of the environment • Social simulation theories • How we represent the mental states of other people • We use simulations of our own minds • To feel someone else’s pain we simulate our own pain • Mirror neurons • Empathy, imitation, social coordination

8. Cognitive Simulation Theories • Perceptual Symbol Systems • Synthetic approach • Implements standard symbolic functions • Type-token binding, inference, productivity, recursion, propositions • A single multimodal representation system in the brain supports diverse forms of simulation across different cognitive processes • High-level perception • Working memory • long-term memory • conceptual knowledge • Convergence zone architecture (Damasio 1989, Simmons & Barsalou 2003) • Single representation system controlled by multiple simulation mechanisms

9. Convergence zone • ensemble of neurons within which many feedforward/feedback loops make contact. • It 1) receives forward projections from cortical regions located in the connectional level immediately below • 2) Sends reciprocal backward projections to the originating cortices • 3) Sends forward projections to cortical regions in the next connectional level; and • 4) Receives projections from heterarchically placed cortices and from subcortical nuclei in thalamus, basal forebrain, and brainstem.

10. Empirical Evidence for Grounded Theories • Perceptual Inference • Perception-action coordination • Perception of space • Memory • Implicit memory • Explicit memory • Working memory • Conceptual processing

11. Perceptual inference • Vision and motion • Goldstone (1995) • Association between shape and color • Hansen et al. (2006) • Object’s natural color distort achromatic perception of the object toward the opponent color • Motion (Freyd 1987, Shiffrar & Freyd 1990,1993) • Subjects simulate the visual trajectory beyond its actual trajectory • Also during apparent motion, simulation of possible action shapes the perception of motion • Speech (Warren 1970) : http://www.youtube.com/watch?v=UlJs24j3i8E • Lexical knowledge produces simulation in speech perception – missing phoneme simulation

12. Perception-action coordination • Simulations of potential actions • Viewing an object grasped with a precision or power grip (grape vs. hammer) produces a simulation of the appropriate action (Tucker & Ellis 1998) • This is affected by • object’s orientation (Symes et al. 2007) • Size (Glover et al. 2004) • Simulations of both grasping and functional actions (Bub et al 2007) • Also name triggers simulation (Tucker & Ellis 2004) • Hearing a word activates the articulatory action associated with producing it (Pulvermuller 2006) • Perceived effort affects visual perception (Proffitt 2006) • Being tired from a run makes a hill look steeper • Carrying a heavy pack makes a path look longer

13. Perception-action coordination • Motor simulations • Motor system constructs a feed-forward simulation of the action to guide and correct it (Grush 2004, Wolpert et al. 1999) • Generating visual inferences about the anticipated actions of perceived agents (Wilson & Knoblich 2005)

14. Perception of space • The perception of space is shaped by the body and it’s relation to the environment • Locating objects has various difficulty along different axes • Vertical • easiest • Front-back • Left-right • Most difficult – bodily cues are lacking • Perception of near space extends with arm length (Longo & Laurenco 2007)

15. Implicit memory • Results form simulation of perceptual memories • Repetition priming is strongest when the modalities of the memory and stimulus match (e.g. auditory) (Kirsner et al., 1989) • Repetition priming is strongest when perceptual details of the memory and stimulus match (e.g. orientation, size,…) (Jacoby&Hayman, 1987) • Imagining produces repetition priming similar to actual perception (Roediger&Blaxton, 1987)

16. Explicit memory • Multimodal simulations of previous episodes • Important for constructing future events • The retrieval of a word stimulates the modal operations performed at encoding (Wheeler et al. 2000) • Visual areas become active during retrieval following visual study while auditory areas become active following auditory study • Greater activation in modal areas when remembering something that really occurred than false memories (Slotnick & Schacter 2004)

17. Working memory • Absent stimulus is stored in working memory (Levy & GoldmanRakic 2000) • To maintain working memory, neurons in the frontal lobes maintain a simulation of the absent stimulus in the modal system that processed it originally. • Some frontal regions maintain working memories of objects, other spatial locations, motion, textures, etc. • They are highly selective for the specific features • Visual imagery in working memory simulates visual processing (Finke 1989, Kosslyn 1980,…) • Analogously, motor imagery, auditory imagery, etc. • Mental rotation of visual objects -> motor simulations of making them turn (Richter et al. 2000)

18. Conceptual processing • Behavioral evidence • When asked whether an property belongs to an objects subjects simulate properties to verify them (Solomon & Barsalou 2004) • Lesion evidence • Lesions in one modality – losing categories that rely on it for processing (Damasio 1994, …) • E.g. damage to visual areas – losing of ability to categorize animals (visual processing is dominant) • Damage to motor areas – categorization of tools • Neuroimaging evidence (Martin 2001, 2007) • When processing conceptual knowledge, brain areas representing properties are active

19. Language comprehension • Perceptual simulation • Motor simulation • Affective simulation

20. Language comprehension • Situation models • Evidence of modal representations in language comprehension • Spatial representation (Bower & Morrow 1990) • People confused pictures with text (Intraub & Hoffman 1992) • Replacing words with pictures did not disrupt sentence processing (Potter 1986)

21. Perceptual simulation • Subjects read a sentence and then processed a picture that either matched or mismatched something implied by the sentence • “The ranger saw the eagle in the sky” • Picture of an eagle – wings outstretched or folded • Visual irrelevant information interferes with spatial inferences (Fincher-Keifer 2001)

22. Motor simulation • Verbs for head, arm and leg actions produce head, arm and leg simulation in the respective areas of the motor system (Pulvermuller 2005) • When action to make a response is consistent with text meaning, the response is quicker (Glenberg & Kaschak 2003) • Subjects simulate corresponding motion through space (Richardson et al. 2003) • Positive/negative valence (Meier & Robinson) • High/low power (Schubert 2005)

23. Affective simulation • Subjects’ faces configured according to sentences with emotional content (Havas 2007) • When facial emotion matched the content comprehension was better • Gesture • Producing gestures helps speakers retrieve words whose meaning are related to the gestures (Krauss 1998) • Also help listeners comprehend what speaker says • Children can gesture before speaking

24. Thought • Physical reasoning • Gear, pulleys • Driven by spatial simulation • Sketchy, not holistic and detailed • Abstract reasoning • Content effects • Reasoning about time using space domain - metaphors

25. Social Cognition • Embodiment effects • Activating elderly stereotype causes people to walk slowly and to perform lexical decision slowly (Dijksterhuis & Bargh 2001) • Engaging the smiling musculature produces positive affect (Strack et al. 1988) • Social mirroring • Individual differences in the ability to simulate other people’s mental states correlate with rated empathy (Jackson et al. 2005) • Development • Mirroring, object permanence…

26. Questions?