University of Sao Paulo Workshop: Gesture Theory

1. University of Sao PauloWorkshop: Gesture Theory Applications to Second Language Development Aug. 9, 2012 Steve McCafferty, University of Nevada, Las Vegas, Dept. of Educational Psychology and Higher Education

2. Clark, P.B. (1986). Mandelbrot, Comment j'aidecouvert les fractales, La Recherche, 420-424. http://www.gap-system.org/~history/Biographies/Mandelbrot.html. Retrieved 3/10/ 2010 About the Discovery of Fractals by Mandelbrot: “What he saw was that the overwhelming smoothness paradigm with which mathematical physics had attempted to describe Nature was radically flawed and incomplete. Fractals and pre-fractals once noticed were everywhere.”

3. Gesture and the origins of language • Gesture first: Some researchers suggest that gesture was how language began, that is, before speech there was gesture (after bipedalism), language developing later • Gesture and speech co-development: Other researchers (notably McNeill, 2012) argue that gesture developed only in modern humans together with language, about 150-200 thousand years ago, after the expansion of the pre-frontal lobes was complete (Deacon, 1997) (Growth Point Hypothesis)

4. Gesture and the origins of language • In any case, Streeck (2010), Streeck and LeBaron (2000) and Müller (1998) have proposed that gestures derive from practical, world-directed actions. However, as McNeill (2012) points out, gestures can be multidimensional with regard to meaning-making, encompassing abstract as well as concrete information at the same time

5. Embodiment: Philosophy and Neurology • Spinoza: The mind body connection (opposite of de Carte) • Vygotsky: no separation of cognition/affect or social/psychological in development • The World on Paper (2004) by David Olsen (literacy) • Luria, Sacks, Ramachandran, Damasio (neurology)

6. Embodiment: Philosophy and Neurology • Mirror neurons (when we watch others move our mirror neurons fire in the brain in the same way as the person doing the activity) (see Arbib, 2012) • McNeill: The Growth Point Hypothesis (Brodmann’s area 44 and Broca’s area 45 in the brain are adjacent • Goldman & Gallese (1998), on the basis of mirror neurons, put forth their Simulation Theory – that we “read” the minds of others if we have been in similar states (Mead’s Loop suggests that people become aware of their own actions [not others] this way)

7. Signs: Ecological perspectives • Von Uexkull (1902) was the first to study sign-mediated relations with the environment and these were non-linguistic (social/biological) • Vygotsky (1930s) recognized the importance of ecosocial, environmental contexts on development • Gibson’s (1979) theory of affordances is grounded in body and mind functioning as a whole for the purposes of the organism

8. Signs: Ecological perspectives • Thibault (2004) takes an ecosocial semiotic view of language in contexts • All of these researchers note that the relationship to the environment is one in which the organisms that live there both are shaped by it as well as shape it in a dynamic, on-going process

9. Embodiment: Linguistics • (1926)Saussure: lange vs. parole • (1933) Bloomfield: AM structualism • (1950s) Birdwhistell (kinesics), Hall (proxemics), Ruesch and Kees (NVC) • (1950s) Chomsky: Competence not performance (Structural model undermined by the generative approach [Scollon & Scollon, 2009, p. 175])

10. Embodiment: Linguistics • Other modes of communication were not found to function like language (Scollon & Scollon, 2009) • “Language and linguistic analysis may be an important source of concepts, but it is fatal to the [multimodality] research endeavor to simply transport linguistic analysis over into the analysis of other modes” (Scollon & Scollon, 2009, p. 177) • Harris (1998; 2003) at Oxford Univ. is one linguist who does not isolate language from communicative and situational contexts (Integrated Linguistics)

11. Gesture Studies: Language and Linguistics • McNeill’s (1992; 2005; 2012) Growth Point Hypothesis: Speaking is more than uttering speech sounds with meaning; more deeply it is also “inhabiting” language (a term from Merleau-Ponty 1962). • The GP can be regarded as a hypothesis of how this multilayered inhabitance takes place, using vocal, manual and body movements, all orchestrated by imagery and incorporating context, purpose and thought • How speech and gesture organize is a dynamic process depending on contexts

12. Gesture Studies: Language and Linguistics • Language and gesture operate differently in the making of meaning • Language is a linear, hierarchical, sequential, abstract contribution to thought and communication • Gesture is a holistic, imagistic, concrete contribution to thought and communication • There is a dialectical synthesis of these two opposite modes of semiotic packaging in relation to contexts in order to make meaningspontaneously

13. Gesture Studies : Language and Linguistics • Krauss, Chen, and Gottesman (2000) view gesture as auxiliary to speech in their Lexical Retrieval Hypothesis • Kita’s (2000) Information Packaging Hypothesis: Spatio-motoric thinking “organizes information with action schemas” and is typical of people’s “interaction with an object, locomotion, and imitating somebody else’s action” (p. 164)

14. Gesture Studies : Language and Linguistics • De Ruiter (2007) sees gesture as driven by communicative intent in a parallel but unconnected coordination with speech (the Sketch Model) • Alibali and Heath’s (2003) Semantic Information Hypothesis argues that The visibility between speaker and listener should influence speakers’ production of gestures that convey semantic information

15. Gesture Studies : Language and Linguistics • Kendon (2004) views gesture as primarily communicative and as a function of language • Goldin-Meadow (2008) in part, views gesture as reducing the cognitive load for both thinking and communicating. She also views gesture mismatches as indicative of cognitive development in children. • Roth (2002): highschool science students first manipulated materials while conducting physics experiments, then explained the experiment using gestures, eventually being able to verbalize them

16. Vygotsky and development • “Vygotsky denied the existence of human beings without culture – after all, he considered culture and culturally mediated thought/action to be the hallmark of the emergence of human beings as a distinct species” (Cole & Gajdamaschko, 2007, p. 201) • Towards the end of his life Vygotsky moved from a focus on word to a focus on meaning, taking into account the larger realm of semiosis

17. Vygotsky and development • However, culture vs. nature, culture vs. biology are dualisms not accepted by Vygotsky, Mead, or James • Regardless, at some point Western psychology decided that the material world is the domain of scientists and that psychologists deal with the internal, non-material world (Costall, 2007, p. 119); in some current iterations psychology is entirely biological

18. Elaboration of Vygotsky and development • Timescales intermix, so that a trajectory though time-space involves encountering new aspects of things experienced earlier that become linked in dynamic and interdependent ways (Lemke, 2000). True for second language development as well.

19. Gesture Studies: Social, Cultural Signs • “Ape gestural signals work in ways more similar to human communication than do ape vocal signals , particularly in terms of flexible production and attention to the attentional states of the partner” (Tomasello & Call, 2007, p. 222) • We mirror the gestures and other nonverbal behaviors of others as an aspect of showing social solidarity (Tannen, 1990)

20. Gesture Studies: Social, Cultural Signs • “[Nonverbal] imitation/conformity/ solidarity/affiliation … has important consequences for the evolution of human cooperative communication” (Tomasello, 2008, p. 152)

21. Gesture Studies: Social, Cultural Signs • Cultural forms of nonverbal behavior constitute “the primary public dimension that defines our personal identity, and in it, style and tradition matter, to the degree that these things establish who we are, who our friends are, and where we stand in society” (Donald, 2001, p. 240)

22. Gesture Studies: Social, Cultural Signs • Cultural forms of nonverbal behavior constitute “the primary public dimension that defines our personal identity, and in it, style and tradition matter, to the degree that these things establish who we are, who our friends are, and where we stand in society” (Donald, 2001, p. 240)

23. Gesture Studies: Social, Cultural Signs • “Indeed, when a young child pretends to drink from an empty cup and looks playfully to the adult’s face, one could say that in addition to pretense for the self this is also an iconic gesture to share this representation with the adult, communicatively” (Tomasello, 2008, p. 152)

24. An Ecosocial, Semiotic Approach to Meaning-Making • How people shape their environment semiotically and how this in turn shapes them as part of the complex, dynamic, and nonlinear process “in which the world and its meanings are actively produced by us” (Thibault 2004, p. 184)

25. An Ecosocial, Semiotic Approach to Meaning-Making • In turn, this conceptualization suggests, as Thibault (2004, p. 234) further argues, that “[m]eaning is not … something which is isolable in just one part of the system. Rather it is a result of the relation between the agent-observer and the overall system of relations on all levels.”

26. An Ecosocial, Semiotic Approach to Meaning-Making • As such, it is further argued by Thibault (2004, p. 176), that “… meaning is stored, not at the level of the individual per se, but at the level of … contextual configurations … which integrate individuals to their ecosocial environment and therefore, to the systems of interpretance that are embedded in these.”

27. An Ecosocial, Semiotic Approach to Meaning-Making • Newcomers are subjected to mastering the “use of objects and symbols, pragmatically, semiotically, and semantically, something that can only be done by participating in socio-cultural practice” (Rosa, 2007, p. 305) • Newcomers construct a sense of self “as an object among others, as an agent and as an actor” (ibid, p. 308)

28. An Ecosocial, Semiotic Approach to Meaning-Making • Language is grounded upon forms of life, not something that can be considered in isolation, independent of the multiple functions it fulfils in the lives of its users (Wittgenstein, 1953/1958) • “Speaking is more than uttering speech sounds with meaning; more deeply it is also ‘inhabiting’ language (a term from Merleau-Ponty, 1962 as found in McNeill, 2012)

29. An Ecosocial, Semiotic Approach to Meaning-Making • Figured worlds are socially and culturally constructed or interpretive realms where people come together to construct joint meanings and activities (Boaler & Greeno, 2000) • Performing is the act of children as they dress up to pretend to be their parents (ZPD – the act of becoming) (Newman and Holzman, 1993; Holzman, 2011). This is dramaturgical as well. • Teachers prolept students beyond what is known in the present to what they will know (van Lier, 2004)

30. An Ecosocial, Semiotic Approach to Meaning-Making • Bakhtin (1934/5, as found in Rosa, 2007, p. 307) suggests how a word becomes populated with sense and reaches a fully conventional stage: “The word of language – is half alien. It becomes ‘one’s own’ when the speaker inhabits it with his intention, his accent, masters the word, brings it to bear upon his meaningful and expressive strivings. Until that moment of appropriation … the word exists on the lips of others, in alien contexts, in service of other’s intentions.”

31. An Ecosocial, Semiotic Approach to Meaning-Making • “Consciousness became … a result of the internalization of (social) communication with semiotic materials (cultural), accumulated along the (historical) past of the cultural group” (Valsiner & Rosa, 2007, p. 8) • As an aspect of inner speech, L2 learners may also develop an L2 inner voice, which includes gestures as well as gesturing (Shigematsu, unpublished dissertation, Univ. of Nevada)

32. Actuation • How organisms come to act on affordances in the environment is the process of actuation • Mediational Actuations of Communication: Three stages: (Rosa, 2007, p. 303). • 1) exposure to an aspect of communication that is the object to be understood • 2) engaging in the act itself • 3) conventional use of object in social settings

33. Actuation •Newcomers are subjected to mastering the “use of objects and symbols, pragmatically, semiotically, and semantically, something that can only be done by participating in socio-cultural practice” (Rosa, 2007, p. 305) • Newcomers construct a sense of self “as an object among others, as an agent and as an actor” (ibid, p. 308)

34. L2 Gestures, Gesturing (naturalistic contexts) • A comparative study on the gestures of immigrant and assimilated Southern Italians (Sicilians and Neapolitans) and Eastern European Jews (Polish and Lithuanian) in New York City found that the gestures produced by the assimilated groups differed from those produced by the immigrant groups. The gestures of the assimilated groups had become both similar to one another and generally more American (Effron, 1941/1972)

35. L2 Gestures, Gesturing (naturalistic contexts) • L1 Anglophone children deploy L2 Francophone emblematic gestures when speaking to age-matched L1 French speakers (Von Raffler-Engler, 1976) • Mohen &Helmer (1988): L2 children acquire L2 emblematic gestures at very young ages • Jungheim (2006) studied the actuation of emblems in naturalistic contexts • McCafferty (1998; 2008), McCafferty & Ahmed (2000): acquisition of L2 gestures and gesturing

36. L2 Gestures, Gesturing (naturalistic contexts) • Nicoladis (2007) suggests that with bilinguals the weaker language includes the use of more deictic gestures than with the stronger language, although she offers no real explanation for this

37. L2 Gestures, Gesturing (classroom contexts) • Jungheim (1991) found that comprehension of selected emblems was increased if students were explicitly taught through watching the teacher perform the gestures in the L2 classroom (adult Japanese L2 learners of English in Japan) • Allen (1995), although researching lexical retention, found that when coupled with emblems (teacher gesturing), students remembered the items tested significantly better than without such coupling

38. L2 Gestures, Gesturing (classroom contexts) • Tellier (2008) found that when students enacted gestures as portrayed by the teacher along with new vocabulary they retained the vocabulary better than when they did not

39. L2 Gestures, Gesturing (classroom contexts) • Haught and McCafferty (2008) focused on L2 university students in the U.S., finding that taking a direct dramaturgical approach – verbally rehearsing short scenes of a highly cultural nature – led students to imitate gestures first inadvertently produced by the instructor, a former actor, and then to use them on their own (the scenes were practiced recursively). Only his gestures were copied

40. L2 Gestures, Gesturing (mixed contexts) • Nardotto Peltier & McCafferty (2010) found that university instructors of Italian in the U.S. who had spent time in Italy engaged in an almost constant use of Italian emblematic gestures as well as Italian ways of gesturing (growth point acquisition as suggested by McNeill, personal communication, October, 2011)

41. L2 Gestures, Gesturing (mixed contexts) • In follow-up interviews, all of the instructors noted that communicating in Italian goes beyond just speaking, emphasizing the embodied nature of this practice. Students in the study were found to mirror their instructors’ Italian gestures and to gesture in an Italian fashion. Also, instructors had not been video recorded previously and all were surprised at the extent of their Italian gestures.

42. L2 Gestures, Gesturing (classroom contexts) • Zhao (2007) found that instructors in university ESL classes in the U.S. used a number of verbal metaphors and metaphoric gestures as a pedagogical element in the teaching of writing for academic purposes • Students used these same gestures when interacting with each other, for example, drawing a virtual vertical line in space from top to bottom to illustrate a sequence of connected ideas as formulated in a logical, hierarchical, linear order (internalization)

43. L2 gesture and motion events • A number of researchers have investigated the possible acquisition of motion events from one language typology to another in which gesture plays a significant role for verb frame as opposed to satellite frame languages, for example Spanish and English. Results to date are inconclusive.

44. Foreigner talk and gesture • It appears that some native speakers, when they feel that the L2 interlocutor is of beginning or intermediate proficiency in the L2 will resort to concretizing their gestures to provide explicit reference (sense) for the language learner (McCafferty & Rosborough, 2007; Stam & Tellier, forthcoming)

45. Generalizations for L2 gesture • Overall, it appears that there is greater frequency of gesturing for speakers in the L2 than in the L1, but that this diminishes with increasing levels of L2 proficiency

46. Generalizations for L2 gesture • There has been a concern that L2 gestures operate as “compensation” for L2 linguistic deficiency. However, this may not be a valid concern in the sense that people gesture as an aspect of meaning-making, a process which may require greater work in the gestural channel in an L2 (see McCafferty 2008, a & b), but at the same time, is a feature of L1 gesture too, as dependent on contexts.

47. Generalizations for L2 gesture • Compensation does not consider the use of gestures of identity in the case of naturalistic learners as well, that is, the effort to accommodate the L2 surrounds both intra-personally and inter-personally