Embodiment and Computation: Convergent Constraints on Language Use

1. Embodiment and Computation:Convergent Constraints on Language Use Nancy Chang nchang@icsi.berkeley.edu UC Berkeley / International Computer Science Institute

2. “Harry walked to the cafe.” “Harry walked into the cafe.” CAFE CAFE What does language do? A sentence can evoke an imagined scene and resulting inferences: • Goal of action = atcafe • Source = awayfrom cafe • cafe = point-like location • Goal of action = insidecafe • Source = outsidecafe • cafe = containing location

3. The scientist walked into the wall. WALL Bonk!! Embodied inferences The hobo drifted into the house. Thesmoke drifted into the house.

4. Metaphorical inference • France fell into recession. Germany pulled it out. • The economy is moving along at the pace of a Clinton jog. • The Indian Government is stumbling in implementing its liberalization plan.

5. Embodied knowledge needed • What things can serve as containers? • rooms but not walls (usually) • How do different entities interact? • how people and gases interact with houses. • How are different actions/states related? • stumbling / walking, falling / containment • How can actions vary? • rate, direction, degree of force, etc. … that is, more than predicate-argument structure! WALK(x), FALL(y), HIT(x,y), etc.

6. Embodiment in language • Perceptual and motor systems play a central role in language production and comprehension • Theoretical proposals • Linguistics: Lakoff, Langacker, Talmy • Neuroscience: Damasio, Edelman • Cognitive psychology: Barsalou, Gibbs, Glenberg, MacWhinney • Computer science: Steels, Feldman

7. Goal: computationally precise theories of language Theory of Language Structure Theory of Language Acquisition Theory of Language Use

8. Theory of Language Structure Theory of Language Acquisition Theory of Language Use

9. Theory of Language Structure Theory of Language Acquisition Theory of Language Use

10. Simulation hypothesis We understand utterances by mentally simulating their content. • Simulation exploits some of the same neural structures activated during performance, perception, imagining, memory… • Linguistic structure parametrizes the simulation. • Language gives us enough information to simulate

11. Utterance “Harry walked to the cafe.” Linguisticknowledge Analysis Process General Knowledge Simulation Specification Schema Trajector Goal walk Harry cafe Belief State Cafe Simulation Language understanding as simulative inference

12. Embodiment and Simulation “What is an idea? It is an image that paints itself in my brain.” — Voltaire

13. Computational efficacy • Embodied representations the norm in robotics! • Computational representations for lexical semantics: have been developed for: • Spatial relations (Regier 1996) • Actions (Bailey 1997, Narayanan 1997) • Objects / attributes (Roy 1998) • Metaphor understanding system based on simulation (Narayanan 1997)

14. Metaphor system architecture Target domain Metaphor maps Source domain (Narayanan 1997)

15. Indian Government stumbling in implementing liberalization plan Metaphor understanding system

16. Missing link: grammar! • Metaphor understanding system demonstrates that embodied inferences for difficult case are feasible. • BUT: system has no grammar! • How do we bridge the gap? • Need a grammatical theory/formalism that can served as an interface between linguistic units and embodied, dynamic, encyclopedic, context-based information (i.e., that can support simulation).

17. 2. Embodied Construction Grammar ”It is not enough to say that the mind is embodied; one must say how.”— Damasio

18. “Syntactic investigation of a given language has as its goal the construction of a grammar that can be viewed as a device of some sort for producing the sentences of the language under analysis.” (Chomsky 1957) Inadequate notion of grammar Meaning-free: syntax separate from meaning, function and processing; unanalyzable symbolic units Inflexible: strict word order, strictly hierarchical, strictly compositional What passes as grammar?

19. Who’s up to the task? • Most theories of language are not explicitly and systematically tied to action and perception • Promising exceptions • Cognitive Grammar / cognitive linguistics • Construction Grammar • Typically criticized for being informal / vague • We borrow liberally from both and formalize.

20. Cognitive Linguistics “Language is an integral part of cognition which reflects the interaction of cultural, psychological, communicative, and functional considerations, and which can only be understood in the context of a realistic view of conceptualization and mental processing.” International Cognitive Linguistics Association website(http://www.cognitivelinguistics.org/aims.shtml)

21. Key borrowed ideas • Conceptual structures are embodied. • Meaning is conceptualization (part of larger cognitive system). • Concepts are grounded in human experience as physical, psychological and social beings in the world. • Basic symbolic unit at all levels is a form-meaning pair, or construction. • Syntax is not independent of semantics. • Phrasal/clausal constructions can contribute meaning independently of constituents. (Lakoff 1987, 1985; Langacker 1991, 1987) (Fillmore 1988, Kay & Fillmore 1999, Lakoff 1987, Goldberg 1995)

22. Traditional levels of analysis UTTERANCE Pragmatics Semantics Syntax Morphology Phonology Phonetics

23. Form phonological cues word order intonation inflection Meaning event structure sensorimotor control attention/perspective social goals... Cafe Form-meaning mappings for language Linguistic knowledge consists of form-meaning mappings:

24. Trajector Source Goal Path Construction Grammar A construction is a form-meaning pair whose properties may not be strictly predictable from other constructions. (Construction Grammar, Goldberg 1995) Form Meaning block walk to

25. Constructions as maps between relations Complex constructions are mappings between relations in form and relations in meaning. Form Meaning • Mover + Motion • before(Mover, Motion) MotionEventmover(Motion, Mover) “is” + Action+ “ing”before(“is”, Action) suffix(Action, “ing”) ProgressiveActionaspect(Action, ongoing) DirectedMotionEventdirection(Motion, Direction) mover(Motion, Mover) Mover+Motion+Directionbefore(Motion, Direction) before(Mover, Motion)

26. More on Construction Grammar (Goldberg 1995) • Clause-level patterns correspond to basic events transitive: Agent Action Patient ditransitive (dative): Giver Action Recipient Gift • Economical: no explosion of senses Hepushedthe ball. He pushed her the ball. • Novel uses handled more robustly Mary pushedthe tissue off the table. ?Mary sneezedthe tissue off the table. *Mary sleptthe tissue off the table.

27. Embodied Construction Grammar(Bergen and Chang 2002) • Embodied representations • active perceptual and motor schemas (image schemas, x-schemas, frames, etc.) • situational and discourse context • Construction Grammar • Linguistic units relate form and meaning. • Both constituency and (lexical) dependencies allowed. • Constraint-based • based on feature structure unification (as in HPSG) • Diverse factors can flexibly interact.

28. ECG Structures • Schemas • image schemas, force-dynamic schemas, executing schemas, frames… • Constructions • lexical, grammatical, morphological, gestural… • Maps • metaphor, metonymy, mental space maps… • Spaces • discourse, hypothetical, counterfactual…

29. boundary bounded region Image schemas • Trajector / Landmark (asymmetric) • The bike is near the house • ? The house is near the bike • Boundary / Bounded Region • a bounded region has a closed boundary • Topological Relations • Separation, Contact, Overlap, Inclusion, Surround • Orientation • Vertical (up/down), Horizontal (left/right, front/back) • Absolute (E, S, W, N) LM TR

30. Embodied schemas schema name schemaSource-Path-Goal roles source path goal trajector schemaContainer roles interior exterior portal boundary role name Boundary Interior Trajector Portal Source Goal Path Exterior These are abstractions over sensorimotor experiences.

31. CAFE Embodied constructions ECG Notation Form Meaning constructionHARRY form : /hEriy/ meaning : Harry Harry constructionCAFE form : /khaefej/ meaning : Cafe cafe Constructions have form and meaning poles that are subject to type constraints.

32. Representing constructions: TO constructionTO form selff.phon /thuw/ meaning evokes Trajector-Landmark as tl Source-Path-Goal as spg constraints: tl.trajector«spg.trajector tl.landmark«spg.goal local alias identification constraint The meaning pole may evoke schemas (e.g., image schemas) with a local alias. The meaning pole may include constraints on the schemas (e.g., identification constraints «).

33. The INTO construction constructionINTO form selff.phon /Inthuw/ meaning evokes Trajector-Landmark as tl Source-Path-Goal as spg Container as cont constraints: tl.trajector«spg.trajector tl.landmark«cont cont.interior«spg.goal cont.exterior«spg.source • TO vs. INTO: • INTO adds a Container schema and appropriate bindings.

34. Constructions with constituents:The SPATIAL-PHRASEconstruction constructionSPATIAL-PHRASE constructional constituents sp : Trajector-Landmark lm : Thing form spfbefore lmf meaning spm.landmark « lmm local alias order constraint identification constraint • Constructions may alsospecify constructional constituents and impose form and meaning constraints on them: • order constraints • identification constraints

35. An argument structure construction constructionDIRECTED-MOTION subcase of Pred-Expr constructional constituents a : Ref-Exp m: Pred-Exp p : Spatial-Phrase form afbefore mf mfbefore pf meaning evokes Directed-Motion as dm selfm.scene « dm dm.agent « am dm.motion «mm dm.path «pm schemaDirected-Motion roles agent : Entity motion : Motion path : SPG

36. The CAUSED-MOTIONconstruction constructionCAUSED-MOTION subcase of Pred-Expr constructional constituents agent : Entity action: Action patient: Entity path : SPG form agentfbefore actionf actionfbefore patientf actionfbefore pathf meaning evokes Caused-Motion as cm selfm.scene « cm cm.agent « agentm cm.action «actionm cm.patient«patientm cm.path «pathm

37. Language Understanding Process • An utterance is perceived • This activates the form pole of some constructions • The Analysis process assembles the constructions, using construal where necessary, and binds together their forms and their meanings • The product is a constructional analysis • This yields a semspec -- parameterized schemas linked together in specified ways • The semspec is input into the Simulation process, where the understander imagines the content • Resulting inferences are propagated through the conceptual system.

38. constructionWALKED form selff.phon [wakt] meaning : Walk-Action constraints selfm.time before Context.speech-time selfm..aspect  encapsulated “Harry walked into the cafe.” Utterance Analysis Process Constructions General Knowledge Semantic Specification Belief State Simulation CAFE Simulation-based language understanding

39. Simulation specification • A simulation specification consists of: • schemas evoked by constructions • bindings between schemas

40. Language Understanding Process

41. Constructional analysis

42. Semantic Specification

43. Basic Feature Structure A new rule for “I” The corresponding fstruct Pronoun  I number SG person  1st -The top part of the rule is the old CFG rule. -This data structure is attached to the nonterminal during parsing so that the parser can use the information. -The next two lines set the agreement features. -The feature is on the lhs of the colon And the value is rhs of the colon. -The  denotes assignment to the feature listed on the lhs.

44. Feature Structure Unification • To check the compatibility of two fstructs • Two feature structures are compatible if they have the same value for every feature they have in common (or if one or both leave the value unspecified). • This process of checking compatibility is called unification. • Unification • Is a recursive process that takes two feature structures and either returns the combined feature structure if they are compatible or it returns failure. • Base case: Two values unify if they are the same string. • Recursive Case: Two feature structures unify if for each feature they have in common, those values unify. • The resulting feature structure just adds the features they don’t have in common to the resulting structure.

45. Language Analysis and Embodied Construction Grammar John Bryant jbryant@icsi.berkeley.edu

46. Getting From the Utterance to the SemSpec • Need a grammar formalism • Embodied Construction Grammar (Bergen & Chang 2002) • Need new models for language analysis • Traditional methods too limited • Traditional methods also don’t get enough leverage out of the semantics.

47. Embodied Construction Grammar • Semantic Freedom • Designed to be symbiotic with cognitive approaches to meaning • More expressive semantic operators than traditional grammar formalisms • Form Freedom • Free word order, over-lapping constituency • Precise enough to be implemented

48. Traditional Parsing Methods Fall Short • PSG parsers too strict • Constructions not allowed to leave constituent order unspecified • Traditional way of dealing with incomplete analyses is ad-hoc • Making sense of incomplete analyses is important when an application must deal with “ill-formed” input • Traditional unification grammar can’t handle ECG’s deep semantic operators.

49. Recognizer Example Mary kicked the ball into the net. This is the initial Constituent Graph for caused-motion. Patient Agent Action Path

50. Recognizer Example Construct: Caused-Motion Constituent: Agent Constituent: Action Constituent: Patient Constituent: Path The initial constructional tree for the instance of Caused-Motion that we are trying to create.