EUROPEAN SOCIETY FOR THE STUDY OF COGNITIVE SYSTEMS MAASTRICHT 18-20 MAY 2001

1. EUROPEAN SOCIETY FOR THE STUDY OF COGNITIVE SYSTEMS MAASTRICHT 18-20 MAY 2001

3. THE NATURAL ORIGIN OF LANGUAGE The Structural Inter-relation of Language, Visual Perception and Action 2001

4. PRECURSORS PLATO LUCRETIUS WILKINS HERDER HUMBOLDT CHARLES DE BROSSES

5. PLATO in the CRATYLUS "Cratylus says that everything has a right name of its own, which comes by nature and that a name is not whatever people call a thing by agreement, just a piece of their own voice applied to the thing, but that there is a kind of inherent correctness (orthoteta tina ton onomaton pephukenai) which is the same for all men, both Greeks and foreigners" whereas Hermogenes was not persuaded that "there is any correctness of names other than convention and agreement". Socrates reached the conclusion: "A name then it appears is a vocal imitation (mimema phonei) of that which is imitated... It will seem ridiculous, no doubt, that things are made manifest through imitation in letters and syllables; nevertheless it cannot be otherwise. For there is no better theory upon which we can base the truth of the earliest names for things (unless we have recourse to dei ex machina)".

6. LUCRETIUS De Rerum Natura 5: 1040-1045 "putare aliquem tum nomina distribuisse rebus et inde homines didicisse vocabula prima, desiperest. nam cur hic posset cuncta notare vocibus et varios sonitus emittere linguae tempore eodem alii id non quiesse putentur." To propose that someone allocated names to things and that it was from him that men learnt their first language is foolish. Why should he have been able to designate each thing with a word and utter spoken sounds at a time when nobody else could do this?

7. JOHN WILKINS Bishop of Chester, Warden of Wadham College, Oxford Founder of the Royal Society Essay Towards a Real Character and a Philosophical Language 1668 "It were exceedingly desirable that the names of things might consist of such sounds as should bear in them some analogy to their natures; and the figure and character of their names should bear some proper resemblance to those sounds that men might easily guess at the sense or meaning of any name or word, upon the first hearing or sight of it. But how this can be done in all the particular species of things I understand not".

8. JOHANN GOTTFRIED HERDER Essay on the Origin of Language 1772 Whence comes to man the art of changing into sound what is not sound? What has a color, what has roundness in common with the name that might evolve from it? The protagonists of the supernatural origin of language have their answer ready-made: "Arbitrary! Who can search and understand God's reason for why green is called green and not blue?” I trust no one will blame me if in this case I cannot understand the meaning of the word arbitrary. To invent a language out of one's brain, arbitrarily and without any basis of choice, is - at least for a human soul that wants to have a reason, some reason for everything - is no less of a torture than it is for a body to be crushed to death. An arbitrarily thought-out language is in all senses contrary to the entire analogy of man's spiritual forces.

9. WILHELM VON HUMBOLDT Uber die Kawisprache auf der insel Java 1836 Linguistic variability and intellectual development The sound is not “a directly imitative sign but indicates a quality which the sign and the object have in common. . . . sounds which partly independently and partly in comparison with others produce an impression which to the ear is similar to that which the object makes upon the mind.” “This kind of sign process which is based upon the particular meaning of each individual letter and whole groups of letters has undoubtedly exercised a prevailing, perhaps even exclusive, influence on primitive word formation.”

10. CHARLES DE BROSSES Charles de Brosses. President of the Parlement de Dijon, was the first author to propose the origin of language as a product of the physiological organisation of the human being. In his Traité de la Formation Méchanique des Langues et des Principes Physiques de l'Etymologie (Paris 1765) he said: "Toutes les observations çi-dessus prouvent qu'il y a des figures des mots, des caractéristiques de sons, liés à l'existence des sensations intérieures, qu'il y en a de liés à l'existence des objets extérieurs ou du moins à l'effet qu'elles produisent sur le sensorium “All these observations prove that there are word-forms, word-sound features, which are linked to the existence of internal sensations, and at the same time are also linked to external objects or at least to the effect that these objects have on the central nervous system.”

11. ROMAN JAKOBSON The Sound Shape of Language 1987 Sound iconism: an iconic (similarity) relation between sound and meaning There is a latent tendency for the sounds of given words to be congruent with their meanings . . . very often built on the phenomenal interconnection between the different senses - on synesthesia The magical power which is associated with sound per se

12. MOTOR EQUIVALENCE The central element of the motor theory of language

13. MOTOR EQUIVALENCE (with the underlying motor basis of perception) To convert: an action into equivalent action word into action action into word word into gesture gesture into word word into sound sound into word visual percept into word visual percept into gesture

14. WORD SOUND/MEANING CATEGORIES The sound-structure of a word can represent the meaning of the word in a number of ways: • Most straightforwardly the word can sound like its meaning. Examples: hiss, wail, sigh, cuckoo, tick-tock, ding-dong. • A word can generate indirectly a sound representing what the word means. The names of many animals are in this category with other words referring to things which produce sounds. Examples: cat, dog, wolf, wasp, bell, whistle, klaxon, thunder, wind. • A word can directly reproduce the action to which it refers. Examples: spit, suck, chew, yawn, sneer.

15. A word can generate a deictic gesture, that is, a gesture which involves pointing to what the word refers to. Words for many body parts and some pronouns are in this category. Examples: head, ear, eye, you, he. • A word can generate an action which represents what the word refers to. This is similar to deictic words. Many function words fall in this category as well as many simple action verbs. Examples: this, that, at, forward, back, hit, throw, take, push, point, pluck, pick, sew, heavy, light. • A word may generate a gesture which outlines what the word refers to. Examples: arch, edge, heap. • A word may generate an action picturing the use of an object referred to. Example: needle,

16. A word may generate a picturing action plus an associated sound. Examples: whip, air, fire. • A word may generate a gesture which amounts to the 'showing' of what the word refers to. This is particularly the case for body parts. Examples: hand, arm, elbow, wrist. • A word may generate a deictic gesture plus an action gesture. Examples: hat, flower. • A word may generate a complex containing visual, sound, deictic or other elements. Examples: snake, rain. • A word may generate an internal feeling or action. Words for emotions seem to come in this category. Examples: bitter, sweet, sour, sad, angry, cheerful.

17. A word may generate an action or an internal feeling for an abstract, non-concrete meaning. Examples: remember, understand, know. The associated actions may metaphorically represent the meaning e.g. elevate, separate, grasp, spontaneous. • A word may generate a movement or position specifically referring to an aspect of time. Examples: now, then • There is an Illustrative List of words where the sound/meaning relation falls in one or other of the above categories at http://www.demon.co.uk/verified.htm

18. The neural basis of the word sound/meaning equivalences is explained in terms of the Motor Theory of Language and of the integration of Perception and the Motor System.

19. MOTOR THEORY OF LANGUAGE

20. MOTOR THEORY OF LANGUAGE The Motor Theory is a theory of the origin and functioning of language. The theory is that the structures of language (phonological, lexical and syntactic) were derived from and modelled on the pre-existing complex neural systems which had evolved for the control of body movement. Motor control at the neural level requires pre-set elementary units of action which can be integrated into more extended patterns of bodily action -- neural motor programs. Speech is essentially a motor activity (a stream of articulatory gestures). Language made use of the elementary pre-set units of motor action to produce equivalent phonological units (phonemic categories). The neural programs for individual words were constructed from the elementary units in the same way as motor programs for bodily action. The syntactic processes and structures of language were modelled on the motor ‘syntax’.

21. PROPOSITIONS 1. There are basic (innate) elementary neural motor programs from which all bodily movements are constructed 2. These elementary motor programs control all the precise ballistic and targeted movements of the hand and arm 3. The elementary programs when redirected to the articulatory organs produce an equivalent set of elementary speech sounds (elementary articulatory gestures) 4. Every articulatory program can be redirected (through motor equivalence) to produce an equivalent movement of the hand and arm 5. Gestures of the hand and arm are structured by the contours of perceived objects or of larger bodily actions 6 Every gesture structured by a perceived object or action can be redirected to produce an equivalent articulatory action

22. PROPOSITION 1 1. There are basic (innate) elementary neural motor programs from which all bodily movements are constructed "Both the effects of simplifying the dynamics computation and the limitations of feedback control in biological arms ... strongly suggest that there must exist substantially correct preprograms in order for humans to make accurate fast arm movements."(Hollerbach 1985: 140) "A Vocabulary of Motor Acts. ... We propose that in inferior area 6 there is a vocabulary of elementary motor acts coded at the single neuron level. This vocabulary is essentially related to arm-mouth movements." (Rizzolatti and Gentilucci 1988: 281) "Movement plans may be complex in the sense of being composed of separable component tasks. These components may be coordinated at some level by the voluntary motor system, in order to combine tasks into appropriate actions ." (Haggard 1991: 153) “these methods have established that all natural movements are organized in discrete segments. . . . So movement is organized on the basis of a repertoire of synergies which go to form possible actions. . . . A library of movements ready to be executed” “ces méthodes ont permis de constater que tous les mouvements naturels sont organisés en segments discrets. . . . Le mouvement est donc organisé à partir d'un répertoire de synergies qui compose autant d'actes possibles. ... une bibliothèque de mouvements facilement déclenchables” (Berthoz 1997: 152, 176)

23. PROPOSITION 3 3. The elementary programs when redirected to the articulatory organs produce an equivalent set of elementary speech sounds (elementary articulatory gestures) Articulatory phonology takes seriously the view that the units of speech production are actions, and therefore that they are dynamic, not static. (Haskins Laboratories) “Utterances are modeled as organized patterns ... of gestures, in which gestural units may overlap in time. The phonological structures defined in this way provide a set of articulatorily based natural classes” (Browman and Goldstein 1992: 155) “Such gestures not only can characterize the movements of the speech articulators but also can act as phonological primitives” (Browman and Goldstein 1990: 313)

24. PROPOSITION 4 4. Every articulatory program can be redirected (through motor equivalence) to produce an equivalent movement of the hand and arm MOTOREQUIVALENCE “ ‘Motor equivalence’ is the term for a simple and remarkable property of the brain which allows one to perform the same bodily movement with very different effector systems. For example, I can write the letter A with my hand, with my foot, or even with my mouth; I could even make an A by walking on the beach” (Berthoz 1997: 246) "comparing findings on the motor organization of speech with the organization of voluntary movements about the elbow ...We have found that the kinematic patterns for movements of the tongue dorsum were similar to those of voluntary flexion-extension movements about the elbow" (Ostry and Cooke 1987: 223). "the task dynamic model we are using for speech was exactly the model used for controlling arm movements, with the articulators of the vocal tract simply substituted for those of the arm." (Browman and Goldstein 1991: 314)

25. PROPOSITION 5 5. Gestures of the hand and arm are structured by the contours of perceived objects or of larger bodily actions "While people talk, they also use their hands. 'illustrative gestures' are used to indicate shapes, sizes, directions and to point, for example to describe a spiral staircase. .. Where illustrative gestures are similar in form to their reference, emblems [gestures with arbitrary meanings] usually are not" (Argyle 1987:63) A gesture may be an indication. This is perhaps not so much resemblance as a variant of the action-gesture. The most rudimentary gesture is to point to the object referred to or more particularly to the feature of the body referred to. A gesture for the ear is to point to or touch the ear - and so on. "We respond to gestures with an extreme alertness and, one might almost say, in accordance with an elaborate and secret code that is written nowhere, known by none and understood by all." (Sapir quoted by Plutchik 1980: 269) “Iconic gestures appear to be images of concepts and imply the existence of schemas which produce them” (McNeill 1981: 203)

26. PROPOSITION 6 6 Every gesture structured by a perceived object or action can be redirected to produce an equivalent articulatory action GESTURE AND SPEECH AS LINKED PROCESSES "speech and gesture arise as interacting elements of a single system” (McNeill 1987: 503) "The central thesis is that the visual system and the motor system are functionally inseparable ... they are components of a unified perceptuo-motor system, which is itself a component of the organism-environment system." (Lee 1980: 281) “Visually directed action implies continuous transformation of incoming visual stimuli into motor commands.” (Jeannerod 1986: 41)

27. IMPLEMENTATION OF THE MOTOR THEORY EQUIVALENCE OF SPEECH SOUNDS AND MOTOR ELEMENTS

28. MOTOR/SPEECH-SOUND GROUPS

29. SYSTEM TABLE

30. VOWEL LINE Use back key to reverse movement

31. SYSTEM II

32. MAIN CONSONANTS Use back key to reverse movement

33. PROJECTIVE [Ballistic] Use back key to reverse movement

34. SYSTEM III

35. VOWEL + CONSONANT Use back key to reverse movement

36. SYSTEM V

37. LATERAL MOVEMENT Use back key to reverse movement

38. GENERATING MOTOR EQUIVALENT GESTURES AND SOUNDS FROM WORD STRUCTURES

39. ANIMATIONS FUNCTION WORDS NOUNS VERBS Click to see animations - but this will involve restarting the Presentation. Or wait to access the animations after the Presentation is completed.

40. MOTOR EQUIVALENCE OF ANIMAL AND BIRD NAMES AND SOUNDS

41. ANIMAL AND BIRD NAMES catcow dog • English catcow dog • French chat vachechien • Japanese nekoushi inu hyenalionpanther • English hyenalionpanther • French hyènelionpanthère • Various fisisingachirutapuli crow duck wasp • Englishcrow duckwasp • French corbeau canard guepe • Japanesekarasuahiru suzumebachi

43. PERCEPTION AND THE MOTOR SYSTEM

44. PERCEPTION AND THE MOTOR SYSTEM LOTZE WILLIAM JAMES KARL LASHLEY MERLEAU-PONTY

45. RUDOLF HERMANN LOTZE Mikrokosmus: Ideen zur Naturgeschichte und Geschichte der Menschheit 1854 “As soon as the image of a definite movement arises in our consciousness, combined with the wish that it should take place, we have the internal state to which is attached as a necessary result the appearance of that definite movement and when this preliminary condition of its occurrence is present, it takes place forthwith.”

46. WILLIAM JAMES The Principles of Psychology 1890 Ideomotor action: Wherever movement follows unhesitatingly and immediately the notion of it in the mind, we have ideomotor action. We think the act, and it is done; and that is all that introspection tells us of the matter... it is no curiosity, but simply the normal process stripped of disguise.

47. KARL LASHLEY The Problem of Serial Order in Behavior 1951 Dynamic processes in perception 1954 "The perceptual processes in vision may be far more dependent upon integration with the postural-kinaesthetic system than we ordinarily assume. It is not impossible that all the spatial characteristics of vision are .. dependent upon integration with the postural system. It may be that we shall have to seek the source of visual percepts in the integration of these two systems."

48. MAURICE MERLEAU-PONTY Phenomenology of Perception 1962 “A motor presence of the word. The word is first of all an event which grips my body. When I read the word 'warm' my body prepares itself for heat and so to speak, roughs out its outline.” ”Language originates from the mixture of the world and ourselves which precedes all reflection; for the child the name is the essence of the object and the child does not name the object but recognizes it. Language is not composed of conventional signs but is a form of 'psychic gesticulation’

49. NOTON AND STARK Eye Movements and Visual Perception 1971 EYE MOVEMENTS of subject viewing photograph of a bust of Queen Nefertiti [from Yarbus]

50. ACTION MIRRORING MELTZOFF DECETY RIZZOLATTI AND GALLESE