How Gestures Embody Meaning

1. How Gestures Embody Meaning David McNeill Departments of Psychology & Linguistics University of Chicago Presented by Jim Goss Midwest Faculty Seminar April 24-6, 2008

2. What is the reason for looking at speech-synchronized gestures? • It provides us with a way to examine how language, thought, and communication interface • Speech-synch’d gestures are spontaneously created by the speaker as she/he speaks. • The shape and space of gestures reflect speaker’s thought processes. • Gestures are timed with speech to create thought processes that are integrated with speech • Gesture/speech synchrony provides a window onto language and mind.

3. The Gesture Continuum(categories from Kendon) Spontaneous Gesticulation (Mode 1) —> Language-slotted (Mode 2) —> Pantomime (Mode 2) —> Emblems (Mode 2) —> Signs Mode 1=unwitting gestures, Mode 2=gestures intended as symbols (due to S. Duncan) Gestures differ in timing and convention. We focus on gesticulation. As one goes from left to right on the continuum we find that: • The obligatory presence of speech declines • Language like properties increase • Individually defined (idiosyncratic) form-meaning pairs are replaced by socially regulated signs

4. Types of Gestures • Beats • Deictic • Iconic • Metaphoric

5. Opposing Modes of Creating Meaning • Properties of gesture • Imagistic • Synthetic • Any gesture can be a combination of the four types of gestures • Global/Holistic • The parts only make sense in reference to the whole • Instantaneous • Largely unconscious • Gestures can convey complementary or supplementary information to speech

6. Opposing Modes of Creating Meaning • Properties of formal language • Analytic • English is a analytic language that depends a great deal on word order, but all languages are analytic to some degree • Combinatoric- • rules for combination (grammar) can produce an infinite variety of utterances • Linear • Based on socially-conventionalized rules and forms • Form/meaning duality

7. The cartoon stimulus • Two scenes from “Canary Row”, a WB Tweety and Sylvester cartoon from about 1950. • In each scene, Sylvester tries to reach Tweety by means of a drainpipe (Tweety is perched in a window some floors up from the street) - Sylvester climbs on the outside of the pipe first, then on the inside. Things don’t work out so well for Sylvester.

8. Gesture Phases - Regular and Slow Motion “and Tweety runs and gets a bowling prep hold stroke hold ba[ll and dropsit down retraction the drain pipe]” Viv. drops it down prestr. poststr. regular vivrunslow.mpg slow

9. Meanings of Gesture Phases in“gets a bowling b[all and dropsit down the drain pipe]” • Preparation - onset of idea (during “ball”) • Prestroke hold - targeting the stroke • Stroke - the core content of the utterance in which two representational modes co-exist(“it do-”) • Poststroke hold - completes stroke content (“-wn”) • Holds ensure the integrity of the idea and maintain the synchrony of co-expressive speech and gesture

10. Timing • The key phase is the stroke - the meaningful part of the gesture. • Strokes coincide with the co-expressive linguistic segment(s) more than 90% of the time. • The remainder of strokes anticipate these segments, often because of speech disfluency. • The preparation is the onset of the gesture and routinely leads the co-expressive speech.

11. Thus, at the Instant of the Stroke, Two Simultaneous Forms of Thinking With the combination of speech and gesture, thought is both: • Instantaneous - distributed • Global - analytic • Ephemeral - unchanging Also, they are co-expressive which is essential The semiotic differences for the same idea fuel dynamic thinking-for-speaking through instability- the dialectic need to work itself out In this dialectic, properties of gesture images combine with simultaneous linguistic properties; this is a ‘Growth Point’.

12. SPEECH AND GESTURE - Unbreakable bond The tight combination of gesture and speech is a psycholinguistic fact of some importance. It creates the conditions of an imagery-language dialectic. It is shown in a variety of phenomena: • DAF (Delayed Auditory Feedback) • Memory • Stuttering • Gestures of the blind • Deaffrentation- loss of proprioception

13. Gestures and Speech Behave as Single Tightly Coupled Entity Non-DAF: this speaker doesn’t gesture at all, possibly because of hyper-elaboration of speech - repeated embeddings: • Then he tries, then he proceeds-->to clamber up-->inside a drainspout--> which eventually-->will lead up to DAF (delayed auditory feedback): suddenly we find gestures and speech and gesture in synch despite the interference Non DAF DAF

14. This Speech/Gesture Mismatch is Preserved Between Interlocuters • Memory doesn’t differentiate between speech and gesture Experimenter Subject Says “bounces,” gestures path only Says “rolls” but gestures “bounce”

15. Speech/Gesture Mismatches • Gesture blends during deception (Amy Franklin) • Speech/gesture mismatches when solving problems (Susan Goldin-Meadow)

16. Gestures and Speech are Tightly Coupled -Gestures Inoculate for Clinical Stuttering(R. Mayberry) Stroke interrupted if underway No stuttering at stroke onset Gestures are held (interrupted) during stuttering, but synchrony is preserved

17. Gestures and Speech are Tightly Coupled - The Blind(Iverson & Goldin-Meadow) • Congenitally blind speakers also produce gestures with speech • Have never seen gestures. • Produce gestures regardless of whether listener can see or not.

18. IW • Lost his proprioception at age 19 • He must plan out every movement ahead of time and monitor it visually • When he cannot see his hands his instrumental and co-speech gestures • Maintain morphokinetic accuracy • Poor topokinetic control • Without vision speech/gesture synchrony and co-expressiveness are preserved

19. Language as Prosthesis Gestures Embodying Meaning

20. Closing the Loop • Mead’s Loop “Gestures become significant symbols when they implicitly arouse in the individual making them the same response which they explicitly arouse in other individuals” - G.H. Mead • Thought-Language-Hand Link • With mirror neurons Mead’s loop co-opts areas of the brain where actions are orchestrated- BA 44, BA 45 • Significances other than the significances of the actions themselves co-opt the system

21. The combination of speech and gesture favors the formation of gesture-linguistic units • The word ‘unit’ is used in Vygotsky’s sense: • “By a unit we mean a product of analysis which, in distinction from elements, possesses all the basic properties of a whole. Further, these properties must be a living portion of the unified whole which cannot be broken down further...” • We will consider a theory - the ‘growth point’ theory - of what comprises a unit in this field that retains the basic properties of a whole with “living properties”

22. The Growth Point Hypothesis • The growth point is proposed as the minimal unit of the dialectic between imagery and language. • A growth point is a package that has both linguistic categorial and imagistic components. • Growth points are inferred from the totality of communicative events with special focus on speech-gesture synchrony and co-expressivity. • By focusing on these properties we bring out the duality of cognition

23. Why it is called a “growth point” • A GP is meant to be the initial idea unit out of which speech-gesture organization emerges • The GP is a theoretical unit in which utterance formulation is viewed as a process of mental change over very short time spans • The GP addresses the concept that there is a definite starting point for an idea unit

24. To identify a GP • We must combine three kinds of information: • Co-expressive speech and gesture • Synchronous speech and gesture • The field of oppositions that makes this co-expressive, synchronous combination the point of newsworthy differentiation in context • Only by combining all three can we hypothesize the existence of a GP

25. Case Study of a GP • Earlier “it down” example: • “and Tweety runs and gets a bowling bal[l and dropsit down the drain] pipe” • Thanks to the holds, the stroke coincides exaclty with “it down”, which was co-expressive with it. • But this is odd from a grammatical vantage point, because “it down” is not a constituent of the sentence. • Yet it was the core idea unit, the GP. Viv. drops it down regular slow Vivianslow.mpg

26. GP Properties • The inferred GP is the image of downward movement plus the linguistic content of the “it” (i.e., the bowling ball) and the path particle “down”. • In the GP, the downward thrusting image is categorized as a bowling ball on a downward path. • Does not include the verb “drops” - preparation phase continued through it and ended with a prestroke hold until the verb was safely over.

27. GP Properties • Both image and linguistic categorial contentare important • Imagery is important: • Grounds the linguistic categories in a specific visuospatial context. • Provides the GP with the property of ‘chunking’- speaking is a chunk of linguistic output organized around the presentation of an image. This chunking is realized in a phonologically centered, gestured ‘pulse’.

28. GP Properties • Linguistic categorization is also important: • Brings the imageinto the system of categories of the language • GP is an image with a foot in the door of language - visuospatial thinking (instantaneous, global, imagistic) is tied to the categorial content of language (inherently segmented with combinatoric potential)

29. How the GP is Formed -Differentiation • The GP is formed by differentiation • It is an emergent point or “figure” differentiated from a context or (back)ground. • The GP is the point of contextual weight and newsworthiness, the significant departure in the immediate context of speaking. • It corresponds to the ‘psychological predicate’ in Vygotsky and is the point of maximum ‘communicative dynamism’

30. The Psychological Predicate - Key to explaining differentiation and context • Psychological predicate - not necessarily a grammatical predicate. • Marks a significant departure from the immediate context. Vygotsky examples: • Implies this context as background. • The speaker shapes the background in a certain way, in order to make possible the intended significant contrast within it.

31. Timing Reflects DifferentiationTWO ‘CLIMBS UP’ EXAMPLES - S. Duncan • Shows that timing is differentiated relative to context • Climbs 1 “[he climbs upthe…]” Climbs 2 “climbs [up in through the]”

32. ‘Context’ = ‘Fields Of Opposition’ • Background is, in part, a mental construction – the speaker's effort to construct a meaning • This is a model of meaningfulness – the joint product of context and GP is a new meaning which exists only in relation to a background. • Use the “field of oppositions” and “significant (‘newsworthy’) contrast” to refer to this constructed meaning.

33. ‘It down’ Plus Image is a Psychological Predicate • Pre- and poststroke holds show that “it down” was targeted by the gesture- not an accidental combination: • The continued preparation and then prestroke hold ‘drops it d..’ shows that the stroke was withheld until “it” • The poststroke hold ‘…it down’ shows that the stroke was actively preserved through “down” even though movement had ceased. • What was the field of oppositions? This question can be approached empirically via the concept of a catchment

34. The Catchment - Gestural Access to Field of Oppositions • The catchment gives us a way to observe the effective context (field of oppositions). • A catchment is a thematic segment in gesture form. • Catchments suggest the current fields of oppositions during ongoing discourse. • A possible communicative role of gesture at the catchment level = guiding the listener to the current field of oppositions

35. Definition • A catchment is recognized from recurrences of gesture form features over a stretch of discourse • It’s a kind of thread of consistent dynamic visuospatial imagery running through the discourse • The logic is that discourse themes produce gestures with recurring features; these recurrences give rise to the catchment. • Thus, working backwards, the catchment offers clues to the discourse themes in the text with which it co-occurs.

36. Viv.’s Battle Plan Catchments Identified from hand use: • C1 - 1 handed = Sylvester as a solo force • C2- 2 similar handed = the bowling ball as an antagonistic force • C3 - 2 different handed = the relative spatial positions of the bowling ball and Sylvester inside the pipe

37. Context in the Case Study Viv’s Battle Plan (full scene) (1) he tries going [up the insid][e of the drainpipe and] (2)Tweety Bird runs and gets a bowling ba[ll and dropsit down the drai]npipe (3) [and as he's coming up] (4) [and the bowling ball's coming d][own (5) he ssswallows it] (6) [and he comes out the bottom of the drai][npipe (7) and he's got this big bowling ball inside h][im (8) [and he rolls on down] [into a bowling all][ey (9) and then you hear a stri]ke

38. C1One-handed gestures- items (1)and(6) - ties together references to Sylvester as a solo force. C2Two-handed symmetrical gestures- items (2), (7), (8) and (9) - groups descriptions where the bowling ball is the antagonist, the dominant force. The 2-handed symmetric gesture form highlights the shape of the bowling ball. C3Two-handed asymmetrical gestures - items (3), (4) and (5) - groups items in which the bowling ball and Sylvester are equals differing only in their direction of motion. (1) he tries going [[up the insid][e of the drainpipe and]]- (6) [and he comes out the bottom of the drai][npipe (2) Tweety Bird runs and gets a bowling ba[ll and dropsit down the drai]npipe (7) and he's got this big bowling ball inside h][im (8) [[and he rolls on down] [into a bowling all]][ey (9) and then you hear a sstri]ke (3) [[and as he's coming up] (4) [[and the bowling ball's coming d]][own (5) he ssswallows it] Catchment Interpretations- 1

39. Catchment Interpretations -2 • The verb “drops”, therefore, was excluded from the GP. This despite the fact that the gesture showed ‘dropping.’ • We can explain this as follows. The verb describes what Tweety did, not what the bowling ball did (it went down). “Drops” was not part of significant contrast involving the bowling ball - the core idea at (2) was the bowling ball and its action, not Tweety and his. • The detailed synchrony of speech and gesture thus incorporated the context at the moment of speaking.

40. Unpacking - Rest of the Story • Unpacking is resolution of an imagery-language dialectic with further mental growth over short intervals. • The term describes extracting the implications of the GP through further meaning generation. • Unpacking is not just mechanical translation into language. • It requires its own generation of meanings possibly with its own context - beyond the GP’s context.

41. Some Implications • The sentence didn’t start from the verb - “drops” emerged in the unpacking as a result of new meaning - a caused-motion construction. • The verb is not necessarily the core of the dynamic process of evolving meaning and utterance. • The original GP and the unpacking were both guided by contextual contrasts - each by its own. • Meaning develops during unpacking - the opposite concept from an ‘input’ - keep going until a well-formed stopping point is reached.

42. Morals • An utterance, even though seemingly self-contained, contains content from outside of its own structure. • This other content ties the utterance to the context at the level of thinking. • It is this fact - by no means unique or peculiar - that conflicts with the axioms of modularity yet fits the schema outlined in the GP. • That multiple contexts collaborate to form one grammatical structure implies that a sense of grammatical form enters into utterances in piecemeal and oblique ways.

43. ‘Living properties’ from coordinative structures • The question is: does the idea of the bowling ball as an antagonistic force moving downwards automatically take care of features such as size (largish), placement (upper), direction (down), and motive force (agenthood)? • The key: ideas or significances can be attractors of coordinative structures; coordinative structures zero in on meaning. Features of gesture arise during the action itself. Once a gesture has been created usually we can identify features of form that carry meanings, but these are the outcomes of the gesture, not sources.

44. General conclusions • Language is inseparable from imagery. • It has a dual reality, as Wundt (and Saussure, about the same time) saw - both instantaneous (here: image) and successive (here: linguistic and social) - combined in a dialectic that fuels change. • This observation has a number of implications: • Speaking is in part imagery • Imagery and language jointly comprise the growth point • This GP itself isn’t speakable - it is an idea unit - unpacked by finding further meanings that are true to the point of high communicative dynamism and context, and situate the GP in a construction • Across many languages, imagery and catchments differ, but GPs and unpacking–embodiments–seem universal.

45. Thank You! Clockwise: Werner, Husserl, Merleau-Ponty, Bakhtin, Jakobson, IW, Saussure, Vygotsky, and Wundt – in different ways, the GP hypothesis has drawn on all these individuals (and more).