Embodied Coding through Perspective

1. Embodied Coding through Perspective • Brian MacWhinney, CMU

2. Embodiment • The problem of the mental code. Something from nothing. • Mentalese vs. bodyese • The perception-motor cycle • Seeing things through our actions on them • The brain connects to the body • Even true on the level of the saccade • The mirror system connects the body to others

3. What does the code do? • It shapes memory and retrieval. • It permits resonance. • It structures mental models • Long-term • Current conversation • It gives meaning through the homunculus. • So, meaning is based in the end on the body.

4. Bodyese and Perspective • When we act, we have • A stance (balanced, sitting, stretched) • A spatial position (in front, behind) • A temporal flow (repeat, single, end) • Goals and plans (push, open, turn, fall) • A social position (giver, brother, trust) • All of these actions situate us in a perspective

5. Processing • Incrementalism • In comprehension, we build competing perspectives and select • Picture-sentence matching, sometimes with mental pictures • In production, we enact a walk-through of our past experiences • Tip of the iceberg theory • Enactive-depictive • Speech-gesture linkage theory

6. MacWhinney & Pleh 1987 • # switches • SS: The dog that chased the cat kicked the horse. 0 • OS: The dog chased the cat that kicked the horse. 1- • OO: The dog chased the cat the horse kicked. 1+ • SO: The dog the cat chased kicked the horse. 2 • SS > OS = OO > SO • The dog the cat the boy liked chased snarled. 4+ • (dog -> cat -> boy -> cat -> dog)

7. Counting Shifts • Advantage of first mention - Gernsbacher • Reflexive shift • Mary combed herself. • The desert trains the young to be tough. • Complex NP shift -- The communist farmers hated died. • Shift to object, implicit causer • Bill chased the cat. • Coming towards him, John saw five bicycles. • Bill criticized Mary, because she/he • Bill apologized to Mary, because she/he

8. Principles • Pronouns must be referential • First mention maintains unless … • Missing mentions involve costs • Objects prepare shifts • Verbs code perspectives

9. Perspective across Cognition unifiedembodied image perspective perspective perspective perspective space/time deixis direct experience social plans language as a functional neural circuit

10. How language links to levels • Words - abandon, beloved • Linkers - deixis, anaphora • Phrases - (my (little doll)) • Clauses - grammar • Discourse - characters on stage

11. Words • Banana －affordances，actions • Navajo table and chair • Miller and Johnson-Laird’s “table” • Verbs as links to motor actions -- give, walk, talk, push, run • Huge schemas -- promise, gravity, a certiorari

12. Coding and Resonance • Local organization • Somatotopic • Coding through the whole system • Even crazy resonances facilitate learning • Is there hard evidence?

13. Clauses: An example • “The cat licks itself”

14. Depictive Imagery

15. Enactive Imagery • take stance of cat • find area to lick • enact licking • shift attention to area being licked • perceive touch through licking

16. Two pathways enactive depictive

17. Compatible Images • He hammered the nail into the wall. • He hammered the nail into the floor. • Nail pointing up. • Nail pointing across. • Tasks: verify, name • Zwaan, Schwartz, Tversky, Klatzky, Glenberg ….

18. Mirror Neurons - Rizzolatti E grasps M grasps E with pliers M grasps

19. Monkey grasps in dark

20. Ungrounding • Identification • Imagery • Dreaming

21. Identification • Imitation and identification require “mapping” of body image • This requires an accessible body image that can be dynamically remapped • Once mapped, we can follow another’s motions • Johnson (1998), Meltzoff and Moore, Csibra et al. • Eyes and movement trigger identification • Identification is basic, not cue-driven • Mapping to other • Birds “have no hands” (Menn) • Child looks for its tail. (Mervis) • Child opens mouth for scissors (Fincham)

22. Three spatial perspectives • Ego-centered -- egocentric • Here, go up, go left • Object-centered -- allocentric • On the table, left of the tower • Come, go, arrive • Ego takes object perspective • Frame-centered -- geocentric • Toward the Bay, north

23. Three temporal perspectives • Speech time (ego centered) • Now, today • Reference time (object centered) • Thursday, graduation, when he arrived … • Event time against reference • Before he arrived, John had lost his wallet • After he arrived, John lost his wallet • We take the perspective of the reference time

24. Aspect • Aspects are contours of actions or events • Terminative: I stop chopping. • Inceptive: I began chopping. • Iterative: I kept on chopping. • The whole action is the background and the aspect involves approaching the action with a particular involvement

25. Causal Roles • Predicates are directly grounded • Case is routinized perspective • Grammar arises to encode perspective shifts • Constructions serve to express perspective modulation

26. Case is routinized perspective • Arg1 – The initial perspective or starting point • Arg2 – Recipient of shifted perspective • Arg3 – External perspective • Three roles are enough for an item-based grammar (Construction Grammar, Word Grammar)

27. Item-based Sockets • Phrasal constructions provide sockets for extension and pushy polysemy (MacWhinney 1989) • Give me another sand. • A wiser Reagan returned from Rejkjavik. • Verbs provide sockets for social expectations • criticize vs. amazed

28. Constructions • Passive Adverbialization • Double Object Attachment • Relativization Dislocation • Appositive Clefting • Fictive Agent Topicalization • Conflation Possession • Comparison Ellipsis • Complementation Coordination ….

29. Ambiguity • Tim saw the Grand Canyon flying to New York. • Visiting relatives can be a nuisance. • The women discussed the dogs on the beach. • I ordered her pancakes. • Although John always jogs, a mile seems long. • The horse raced past the barn fell.

30. Scope • Many arrows didn’t hit the target. • The target wasn’t hit by many arrows. • Three students read two books. • Two books were read by three students. • Every man who owns a Porsche loves it.

31. C-Command

32. C-command as Perspective • Starting points must be referentially committed (MacWhinney, 1977) • Pronouns immediately seek their referents • McDonald and MacWhinney (1996) • Mary criticized Billi, because hei ... • Maryi amazed Bill, because shei … • Perspective flows to referents online

33. Centrality forces commitment • S > DO > IO > Poss • * Hei says Billi came. • * John told himi that Billi was crazy. • ? John said to himi that Billi was crazy. • John told hisi mother that Billi was crazy. • C-Command blocking depends on how you draw the tree

34. Backgrounding • Releasing commitment: • After hei came, Johni drank three beers. • Near himi, Johni saw a snake. • Yes, we can backtrack, but it is marked: • *Near Johni, hei saw a snake. • ?Near John’si computer desk, hei placed a printer. • ?After Johni came, hei drank three beers. • ?Hei came and then Johni drank three beers.

35. Reflexives maintain Perspective • Igori hid the book behind himi/himselfi. • Johni kicked *himi/himselfi. • Johni ignored the oil on himi/*himselfi. • Johni heard a story about himi/himselfi. • Maxi told a story about *himi/himselfi.

36. Perspective interruption • *Jessiei stole a photo of heri out of the archives. • Jessiei stole me a photo of heri out of the archives.

37. Shifts • Jessie stole a photo of *her/herself out of the archives. • Jessie stole a silly photo of her/herself out of the archives. • Anna hid a snapshot of *her/herself under the linoleum. • Anna hid the snapshot of *her/herself under the linoleum. • Lucie talked about the operation on *her/herself. • Lucie talked about the operation on her/herself that Dr. Edward performed.

38. Clitic Assimilation • Why do you want to go? • Why do you wanna go? • Who(m) do you want to go? • * Who(m) do you wanna go? • I get ta go. (Privilege) • I got ta go. (Privilege) • I gotta go. (Obligation)

39. An example • A cyclone hammered the Bangladesh coast Monday with the force of "hundreds of demons" leveling entire villages of mud and thatch huts, flooding crops, and killing at least six people. • Three men and two children were crushed under collapsed buildings or hit by flying pieces of tin roofs in the southern port of Chittagong. One man died in Teknaf, about 110 miles down the coast, when he was blown off his roof, while trying to secure it. • The storm roared in from the Bay of Bengal with wind gusts of 125 mph, forcing a half-million people to flee their huts and huddle in concrete shelters. Many power and telephone lines were down, so a full account of casualties and damage was not available.

40. Mental Spaces • A contemporary philosopher, leading a seminar (From Fauconnier and Turner) • I claim that reason is a self-developing capacity. Kant disagrees with me on this point. He says it’s innate, but I answer that that’s begging the question, to which he counters, in Critique of Pure Reason, that only innate ideas have power. But I say to that, what about neuronal group selection? And he gives no answer. • One space for the teacher, one for Kant, and a comparison space where the debate occurs.

41. Put the bottom block below the apple

42. Linguistic Devices • Your beloved cat -- Your damned cat. • John criticized Bill, because he .. • John apologized to Bill, because he … • Japanese speaker’s territory • Evidentials, deontic-evidential

43. Perspectival Domains • The bicyclist appears to have escaped injury • Did the bicyclist appear to have escaped injury? • The reporter said that the bicyclist appeared to have escaped injury. • The reporter asked if the bicylcist appeared to have escaped injury. • John said that Marco hates Sarah’s beloved cousin. • The adults in the picture are facing away from us, with the children hidden behind them. (Cantrall)

44. Logophoricity • John said, “The bicycle is here.” • John said that the bicycle is here. • Unfortunately, John said that the bicycle is behind the damn wall. • Did John say that the bicycle is evidently behind the damn wall?

45. Frontal support • Inferior frontal cortex supports code generation through perceptual-motor images for memory lists • Linkage of frontal areas to temporoparietal areas supports reenactment (simulation) • Interconnections of frontal motor and attentional areas • Linkage of prefrontal to subcortical and cingulate gyrus

46. Evolution • Apes can shift perspective (Call & Tomasello) • Gestural communication required further perspective taking • Creativity revolution: language or symbol? • Charismatic leaders and priests could use language to motivate the people

47. Classic Issues • Formalism vs. Functionalism in Grammar • Language and Consciousness • Grounding in Language Development • Symbol Grounding • Robotic Grounding • Language as a Special Gift

48. Assumptions that don’t change • Basic activation/inhibition framework • Power Law • Neural Network modelling • MacWhinney-Bates Competition Model • Cue validity, cue strength • Lexical grammar • Incrementalism • Barsalou makes mental models safe for AI • Dual code theory still works

49. Conclusions • Cognition is grounded on perceptual and motoric simulations, triggered by language and gesture • The forms of language are shaped by the process of perspective-taking • The brain provides mechanisms to implement simulation and perspective