The Brain and Multimodal Communication

1. The Brain and Multimodal Communication Elisabeth Ahlsén University of Gothenburg SSKKII Interdisciplinary Center & Department of Linguistics Nationella Forskarskolan i Kognitionsvetenskap ht 2008

2. Contents • Communication on different levels of awareness and control - corresponding brain functions • What models cover - view of cognition and communication • Multimodal communication - some controversies • Examples of current research trends 5. Brain damage - cognition and communication

3. 1. Communication on different levels of awareness and control - corresponding brain functions Communicative Body Movements: 1) Facial gestures (nose, eyebrows, cheek, forehead, chin etc) 2) Head movements 3) Gaze direction, mutual gaze 4) Lip movement 5) Hand and arm movements 6) Leg and foot movements 7) Body posture 8) Distance btw communicators 9) Spatial orientation 10) Touch 11) Non-linguistic sounds

5. Homunculus

6. Types of representation 3 ways of carrying, conveying and sharing information • Index • Icon • Symbol Variation in awareness and intentionality Bodily communication often more spontaneous, unaware, unintended (index, icon) than words (symbols).

7. Three levels of intentionality: index icon symbol indicate x display x signal x

8. indicate -> display -> signal conventional signal -> automatic

9. Dimensions of content in communication Words Body Factual information Emotions and Attitudes Physiological states Identity Social relations Communication Management Interactive communication management Own communication management

10. Interactive view on embodied communication Two communicators form one dynamic system by establishing stabilizing communication links Feedback employs all of these levels, too. Feedback Max Feedback + Max Outlook

11. Communication - Cognition Memory (WM, LTM) Attention (sustained attention) Central executive …

12. Right hemisphere lesions • Left neglect, spatial disorder • Prosody disorders • Lexical-semantic disorders • Problems with emotion information • Discourse disorder (complexity, ToM, metaphor, humor, irony, etc)

13. Primitive brain - Limbic system - Neocortex

14. James-Lange vs Cannon-Bard Formation of emotions

16. Amygdala Mediation and control of major affective activities like friendship, love and affection, on the expression of mood and, mainly, on fear, rage and aggression The center for identification of danger. Fundamental for self preservation. Humans with marked lesions of the amygdala, loose the affective meaning of the perception of an outside information, like the sight of a well known person. The subject knows, exactly, who the person is, but is not capable to decide whether he likes or dislikes him (or her).

17. Hippocampus Formation of long-term memory. When both hippocampi (right and left) are destroyed, nothing can be retained in the memory. The subject quickly forgets any recently received message. The intact hippocampus allows the animal to compare the conditions of a present threat with similar past experiences, thus enabling it to choose the best option.

18. Fornix & parahippocampal gyrus Connecting pathways

19. Thalamus Lesion or stimulation of the medial dorsal and anterior nuclei of the thalamus are associated with changes in emotional reactivity. The medial dorsal nucleus makes connections with cortical zones of the pre-frontal area and with the hypothalamus. The anterior nuclei connect with the mamillary bodies, and through them, via fornix, with the hippocampus and the cingulate gyrus.

20. Hypothalamus Lesions of the hypothalamic nuclei interfere with several vegetative functions and some of the so-called motivated behaviors. Specifically, its lateral parts seem to be involved with pleasure and rage, while the median part is like to be involved with aversion, displeasure and a tendency to uncontrollable and loud laughing. However, in general terms, the hypothalamus has more to do with the expression (symptomatic manifestations) of emotions

21. Cingulate gyrus(gyrus cinguli) Its frontal part coordinates smells and sights with pleasant memories of previous emotions. This region also participates in the emotional reaction to pain and in the regulation of aggressive behaviour

22. Brain stem Responsible for the "emotional reactions", (indeed, they are just reflex answers) of inferior vertebrates Remains active, not only as alerting mechanisms, vital for survival, but in the maintenance of the sleep-awake cycle.

23. Septum Pleasant sensations, mainly those related to sexual experiences Anterior to the thalamus

24. Prefrontal area Its intense bi-directional connections with thalamus, amygdala and other subcortical structures, account for the important role it plays in the genesis and, specially, in the expression of affective states. When the pre-frontal cortex suffers a lesion, the subject looses his sense of social responsibility as well as the capacity for concentration and abstraction. In some cases, although consciousness and some cognitive functions, like speech, remain intact, the subject can no longer solve problems, even the most elementary ones.

26. 2. What models cover - view of cognition and communication A. Localist-associationist ”classical” model Language - cortical centers + subcortical white fibres -Broca’s area - production -Wernicke’s area - perception-comprehension -Connections -Concept center Geschwind

27. B. Dynamic localization of function • R. Luria Functional systems of many subfunctions in different areas, not always language specific, (e.g. rapid alternations, inertia) Dynamic systems, can be restored-changed 3 blocks: brain stem, posterior PTO, anterior

28. C. Hierarchical / evolution based models (”holistic”) J.H. Jackson- J. Brown: Microgenesis ”from deep to surface” in phylogenesis, ontogenesis and microgenesis” Focus on role of subcortical, older structures, emotion etc. Automatic vs. Controlled behavioron Levels of function for language, emotion, perception

29. 3. Multimodal communication - brain and cognition - some controversies • Localism - Associationism - Holism • Modularity - Interaction • Symbol manipulation - Distributed activation /representation • Linear models - Interactive Activation Models (IAM,ANN) • Language/speech and gesture intertwined or separate/compementary - related how? • Definition/delineation of language / communication system • Different types of body movements - how are they related and how do they relate to spoken language?

30. 4. Examples of current research trends • Embodiment - many aspects, gesture one of them, role of body for development of language another • ”Growth point” (McNeill) • Cultural influence vs universality of gesture + relation to speech • Building embodied artificial agents • Studying coordination-communication on lower levels of awareness and control • Neuroimaging? • Action-Perception link - localization

31. 5. Brain damage & multimodal communication From language/words sentences to communication - multimodal Widening of the concept of aphasia (cf. language) - controversy From left hemisphere cortex to any language-communication disorder, including, RH, subcortical-cerebellar, distributed, e.g. Alzheimer etc., TBI New approaches needed and substantial research New models needed

32. Next episode… On evolution, mirror neurons, the role of imitation and pantomime and possible revisions of theories about communication disorders…

33. Assignment Choose one of the controversies mentioned above and write one page argumenting for each of the two sides or suggesting two different views and approaches to answering one of the questions.

34. Literature: Arbib M (2005). From monkey-like action recognition to human language: an evolutionary framework for neurolinguistics. Brain and Behavioral Sciences, 28, 105–24. Ahlsén, E. 2008. Neurological disorders of embodied communication. In.I. Wachsmuth, G. Knoblich & M. Lenzen (eds.) Embodied Communication in Humans and Machines. Oxford: Oxford University. Press.