What does language do?

1. “Harry walked to the cafe.” “Harry walked into the cafe.” CAFE CAFE What does language do? A sentence can evoke an imagined scene and resulting inferences: • Goal of action = at cafe • Source = away from cafe • cafe = point-like location • Goal of action = inside cafe • Source = outside cafe • cafe = containing location

2. Language understanding (Utterance, Situation) Conceptual knowledge Linguistic knowledge Analysis Interpretation

3. Cafe Language understanding: analysis & simulation “Harry walked to the cafe.” Utterance Lexicon Constructicon Analysis Process General Knowledge Semantic Specification Schema Trajector Goal walk Harry cafe Belief State Simulation

4. Interpretation: x-schema simulation walker at goal Constructions can • specify which schemas and entities are involved in an event, and how they are related • profile particular stages of an event • set parameters of an event energy goal=home walker=Harry Harryiswalkinghome.

5. Traditional Levels of Analysis Pragmatics Semantics Syntax Morphology Phonetics

6. “Harry walked into the cafe.” Pragmatics Semantics Utterance Syntax Morphology Phonetics

7. Trajector Source Goal Path Construction Grammar A construction is a form-meaning pair whose properties may not be strictly predictable from other constructions. (Construction Grammar, Goldberg 1995) Form Meaning block walk to

8. Form phonological cues word order intonation inflection Meaning event structure sensorimotor control attention/perspective social goals... Cafe Form-meaning mappings for language Linguistic knowledge consists of form-meaning mappings:

9. Constructions as maps between relations Complex constructions are mappings between relations in form and relations in meaning. Form Meaning • Mover + Motion • before(Mover, Motion) MotionEventmover(Motion, Mover) “is” + Action+ “ing”before(“is”, Action) suffix(Action, “ing”) ProgressiveActionaspect(Action, ongoing) DirectedMotionEventdirection(Motion, Direction) mover(Motion, Mover) Mover+Motion+Directionbefore(Motion, Direction) before(Mover, Motion)

10. Embodied Construction Grammar • Embodied representations • active perceptual and motor schemas • situational and discourse context • Construction Grammar • Linguistic units relate form and meaning/function. • Both constituency and (lexical) dependencies allowed. • Constraint-based (Unification) • based on feature structures (as in HPSG) • Diverse factors can flexibly interact.

11. Representing image schemas schema name schemaSource-Path-Goal roles source path goal trajector schemaContainer roles interior exterior portal boundary role name Boundary Interior Trajector Portal Source Goal Path Exterior These are abstractions over sensorimotor experiences.

12. Hypothesis: Linguistic input is converted into a mental simulation based on bodily-grounded structures. Components: Semantic schemas image schemas and executing schemas are abstractions over neurally grounded perceptual and motor representations Linguistic units lexical and phrasal construction representations invoke schemas, in part through metaphor Inferencelinks these structures and provides parameters for a simulation engine Inference and Conceptual Schemas

13. Embodied Construction GrammarECG(Formalizing Cognitive Linguisitcs) • Linguistic Analysis • Computational Implementation • Test Grammars • Applied Projects – Question Answering • Map to Connectionist Models, Brain • Models of Grammar Acquisition

14. ECG Structures • Schemas • image schemas, force-dynamic schemas, executing schemas, frames… • Constructions • lexical, grammatical, morphological, gestural… • Maps • metaphor, metonymy, mental space maps… • Spaces • discourse, hypothetical, counterfactual…

15. schema <name> subcase of <schema> evokes <schema> as <local name> roles < local role >: <role restriction> constraints <role> ↔ <role> <role>  <value> <predicate> schema Hypotenuse subcase of Line-Segment evokes Right-Tri as rt roles {lower-left: Point} {upper-right: Point} constraints self ↔ rt.long-side ECG Schemas

16. schema SPG subcase of TrajLandmark roles source: Place path: Directed–Curve goal: Place {trajector: Entity} {landmark: Bounded- Region} schema Container roles interior: Bounded-Region boundary: Curve portal: Bounded-Region Source-Path-Goal; Container

17. schema RD roles category gender count specificty resolved Ref modifications schema RD5 // Eve roles HumanSchema Female one Known Eve Sweetser none Referent Descriptor Schemas

18. construction <name> subcaseof <construction> constituents <name>:<construction> form constraints <name> before/meets <name> meaning: constraints // same as for schemas construction SpatialPP constituents prep: SpatialPreposition lm: NP form constraints prep meets lm meaning: TrajectorLandmark constraints selfm ↔ prep landmark ↔ lm.category ECG Constructions

19. construction Into subcase of SpatialPreposition form: WordForm constraints orth  "into" meaning: SPG evokes Container as c constraints landmark ↔ c goal ↔ c.interior construction The subcase of Determiner form:WordForm constraints orth  "the" meaning evokes RD as rd constraints rd.specificity  “known” Into and The CXNs

20. construction DetNoun subcase of NP constituents d:Determiner n:Noun formconstraints d before n meaning constraints selfm ↔ d.rd category ↔ n construction NPVP subcase of S constituents subj: NP vp: VP formconstraints subj before vp meaningconstraints profiled-participant ↔ subj Two Grammatical CXNs

21. construction ActiveSelfMotionPath subcase of ActiveMotionPath constituents {v: verb} {pp:SpatialPP} formconstraints {v before pp} meaning:SelfMotionPathEvent constraints{spg ↔ pp} {profiled-participant ↔ mover}{profiled-process ↔ motion} {profiled-process ↔ v} ConstructionWalkedVerb subcase of PastPerfectiveVerb formconstraints orth"walked" meaning:WalkAction 

22. Combined score determines best-fit Syntactic Fit: Constituency relations Combine with preferences on non-local elements Conditioned on syntactic context Antecedent Fit: Ability to find referents in the context Conditioned on syntax match, feature agreement Semantic Fit: Semantic bindings for frame roles Frame roles’ fillers are scored

23. Constructs -------------- NPVP[0] (0,5) Eve[3] (0,1) ActiveSelfMotionPath [2] (1,5) WalkedVerb[57] (1,2) SpatialPP[56] (2,5) Into[174] (2,3) DetNoun[173] (3,5) The[204] (3,4) House[205] (4,5) Schema Instances ------------------- SelfMotionPathEvent[1] HouseSchema[66] WalkAction[60] Person[4] SPG[58] RD[177] ~ house RD[5]~ Eve 0Eve1walked2into3the4house5

24. SelfMotionPathEvent[1].mover SPG[58].trajector WalkAction[60].walker RD[5].resolved-ref RD[5].category Filler: Person4 SpatialPP[56].m Into[174].m SelfMotionPathEvent[1].spg Filler: SPG58 SelfMotionPathEvent[1] .landmark House[205].m RD[177].category SPG[58].landmark Filler:HouseSchema66 WalkedVerb[57].m WalkAction[60].routine WalkAction[60].gait SelfMotionPathEvent[1] .motion Filler:WalkAction60 Unification chains and their fillers

25. Summary: ECG • Linguistic constructions are tied to a model of simulated action and perception • Embedded in a theory of language processing • Constrains theory to be usable • Frees structures to be just structures, used in processing • Precise, computationally usable formalism • Practical computational applications, like MT and NLU • Testing of functionality, e.g. language learning • A shared theory and formalism for different cognitive mechanisms • Constructions, metaphor, mental spaces, etc.

26. Embodied Compositional Semantics after Ellen Dodge edodge@berkeley.edu

27. Questions • What is the nature of compositionality in the Neural Theory of Language? • How can it be best represented using Embodied Construction Grammar?

28. Examples • He bit the apple • He was bitten (by a toddler) • He bit into the apple • His white teeth bit into the apple. • He shattered the window • The window was shattered • The window shattered

29. Compositionality • Put the parts together to create the meaning of the whole. • Questions: • what is the nature of the parts? • How and why do they combine with one another? • What meaning is associated with this composition?

30. Short answers • Parts = constructions, schemas • Combination = binding, unification • Meaning of the whole = simulation of unified parts

31. Simulation parameters • Constructions unify to create semantic specification that supports a simulation • Two types of simulation parameters for event descriptions: • Event content • Event construal

32. Summary • Parts = constructions, schemas • Combination = binding, unification • Meaning of the whole = simulation of the combined parts

33. First example • He bit the apple.

34. Schemas schema MotorControl subcase of Process roles Actor↔ Protagonist Effector Effort Routine constraints Actor ← animate

35. schema Contact subcase of SpatialRelation roles Entity1: entity Entity2: entity schema MotorControl subcase of Process roles Actor↔ Protagonist Effector Effort Routine constraints Actor ← animate schema ForceTransfer evokes Conact as C roles Supplier ↔ C.entity1 Recipient ↔ C.entity2 Force schemaForceApplication subcase ofMotorControl evokesForceTransfer as FT roles Actor ↔ FT.Supplier ↔ Protagonist Acted Upon↔ FT.Recipient Effector Routine Effort ↔ FT.Force.amount

36. Schema networks Contact MotorControl ForceTransfer Motion Effector Motion SelfMotion ForceApplication CauseEffect MotionPath Effector MotionPath SelfMotion Path SPG Agentive Impact SpatiallyDirectedAction Contact

37. Verb Constructions Construction BITE1 subcase of Verb form: bite meaning: ForceApplication constraints: Effector ← teeth Routine ← bite // close mouth schemaForceApplication subcase ofMotorControl evokesForceTransfer as FT roles Actor↔ FT.Supplier ↔ Protagonist Acted Upon ↔ FT.Recipient Effector Routine Effort↔ FT.Force.amount

38. Verb Constructions cxn BITE meaning: ForceApplication schema MotorControl cxn GRASP meaning: ForceApplication schema ForceApplication subcase of MotorControl cxn PUSH meaning: ForceApplication cxn SLAP meaning: AgentiveImpact schema Agentive Impact subcase of ForceApplication cxn KICK meaning: AgentiveImpact cxn HIT meaning: AgentiveImpact

39. Argument Structure Construction construction ActiveTransitiveAction2 subcase of VP constituents: V : verb NP: NP form constraints: VF before NPF meaning: CauseEffect evokes; EventDescriptor as ED; ForceApplication as FA constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Causer ↔ ED.ProfiledParticipant FA ↔ Vm Causer ↔ FA.Actor Affected ↔ FA.ActedUpon Affected ↔ NPm

40. Argument Structure Construction construction ActiveTransitiveAction2 subcase of VP constituents: V : verb NP: NP form constraints: VF before NPF meaning:CauseEffect evokes; EventDescriptor as ED; ForceApplication as FA constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Causer ↔ ED.ProfiledParticipant FA ↔ Vm Causer ↔ FA.Actor Affected ↔ FA.ActedUpon Affected ↔ NPm

41. CauseEffect schema schemaCauseEffect subcase of ForceApplication; Process roles Causer ↔ Actor Affected ↔ ActedUpon ↔ Process.Protagonist Instrument ↔ Effector

42. Schema Network Contact MotorControl ForceTransfer Process Motion Effector Motion SelfMotion ForceApplication CauseEffect MotionPath Effector MotionPath SelfMotion Path SPG Agentive Impact SpatiallyDirectedAction Contact

43. Argument Structure Construction construction ActiveTransitiveAction2 subcase of VP constituents: V : verb NP: NP form constraints: VF before NPF meaning:CauseEffect evokes:EventDescriptor as ED; ForceApplication as FA constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Causer ↔ ED.ProfiledParticipant FA ↔ Vm Causer ↔ FA.Actor Affected ↔ FA.ActedUpon Affected↔ NPm

44. Schema Network Contact MotorControl ForceTransfer Process Motion Effector Motion SelfMotion ForceApplication CauseEffect MotionPath Effector MotionPath SelfMotion Path SPG Agentive Impact SpatiallyDirectedAction Contact

45. Important points • Compositionality does not require that each component contain different information. • Shared semantic structure is not viewed as an undesirable redundancy

46. Argument Structure Construction construction ActiveTransitiveAction2 subcase of VP constituents: V : verb NP: NP form constraints: VF before NPF meaning: CauseEffect evokes; EventDescriptor as ED; ForceApplication as FA constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Causer ↔ ED.ProfiledParticipant FA ↔ Vm Causer ↔ FA.Actor Affected ↔ FA.ActedUpon Affected ↔ NPm

47. Event Descriptor schema schema EventDescriptor roles EventType: Process ProfiledProcess: Process ProfiledParticipant: Entity ProfiledState(s): State SpatialSetting TemporalSetting

48. Argument Structure Construction Construction ActiveTransitiveAction2 subcase of VP constituents: V : verb NP: NP form constraints: VF before NPF meaning: CauseEffect evokes; EventDescriptor as ED; ForceApplication as FA constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Causer ↔ ED.ProfiledParticipant FA ↔ Vm Causer ↔ FA.Actor Affected ↔ FA.ActedUpon Affected ↔ NPm

49. Bindings with other cxns construction NPVP1 constituents: Subj: NP VP : VP form Constraints Subj f before VPf meaning: EventDescriptor ProfiledParticipant ↔ Subjm construction ActiveTransitiveAction2 subcase of VP constituents: V ; NP form: VF before NPF meaning: CauseEffect evokes; EventDescriptor as ED constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Causer ↔ ED.ProfiledParticipant Affected ↔ NPm

50. Bindings with other cxns Construction NPVP1 constituents: Subj: NP VP : VP form constraints Subj f before VPf meaning: EventDescriptor ProfiledParticipant ↔ Subjm construction ActiveTransitiveAction2 subcase of VP constituents: V ; NP form: VF before NPF meaning: CauseEffect evokes; EventDescriptor as ED constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Causer ↔ ED.ProfiledParticipant Affected ↔ NPm schema EventDescriptor roles EventType ProfiledProcess ProfiledParticipant ProfiledState(s) SpatialSetting TemporalSetting