Lectures I. Overview 2. Simulation Semantics 3. ECG and Best-fit Analysis 4. Compositionality 5. Simulation, Count

1. Analyzer: Discourse & Situational Context Lectures I. Overview 2. Simulation Semantics 3. ECG and Best-fit Analysis 4. Compositionality 5. Simulation, Counterfactuals, and Inference Constructions Utterance incremental, competition-based, psychologically plausibleA Semantic Specification: image schemas, bindings, action schemas Simulation

2. Introduction: NTL • NTL’s main tenets • direct neural realization, and • continuity of thought and language • both of which entail a commitment to parallel processing and spreading activation • existence of language communities • conventional beliefs, grammars • simulation semantics • language understanding involves some of the brain circuitry involved in perception, motion, and emotion • best-fit process • underlying learning, understanding, and production of language

3. Levels in a Neural Theory of Language The Neural Observation Level: Discoveries made via experimental neuroscience. The Neural Computation Level: A hypothesized (connectionist) account of what “Neural Computation” is and how the brain uses it to function. The Formal Level: The use of a single formal notation linking the Neural Computational and Cognitive Linguistics levels. In Embodied Construction Grammar (ECG), the notation is used in standard forms of computation, both to model the functionality of various aspects of the brain and for use in automatic language analysis. The Cognitive Linguistics Level: The analysis of language and thought using ideas that fit empirical results from the cognitive and brain sciences. The Cognitive and Linguistic Observation Level: Empirical observations about language and thought.

4. Introduction: ECG • Embodied Construction Grammar • part of the Construction Grammar tradition (Croft 2001, Fillmore 1998, Fried & Boas 2005) • adds embodied semantics • Designed as a toolto formally explore the NTL principles • in a tractable, expressive way • not the only way to formalize NTL; cannot directly describe some of its aspects (e.g., spreading activation)

5. Embodied Construction GrammarECG(Formalizing Cognitive Linguistics) • Community Grammar and Core Concepts • Deep Grammatical Analysis • Computational Implementation • Test Grammars • Applied Projects – Question Answering • Map to Connectionist Models, Brain • Models of Grammar Acquisition

6. ECG for linguistic analysis • ECG unifies insights from construction grammars and cognitive linguistics • ECG is not just about representation: • A computationally precise model makes it possible to build systems for linguistic analysis and interpretation • Some history: • Jurafsky (1996) first used construction grammar in a model of interpretation • Bryant (2003): robust child-language interpretation • Steels and de Beule (2006): language learning over populations • Ball (2007): psychologically plausible language interpretation

7. ECG for linguistic analysis • Constructional Analyzer • fits into the unified cognitive science (Feldman 2006) • and builds on • cognitive linguistics • construction grammar • psycholinguistics • simulation-based language inference (Narayanan 1997) • Natural Language Processing techniques

8. ECG for linguistic analysis • Constructional Analyzer (Bryant 2008) • Input: • Grammar • Utterance • Context Model • Output • Semantic Specification, or SemSpec

9. Simplifying grammar by exploiting the understanding process Mok and Bryant, BLS 2006 • Omission of arguments in Mandarin Chinese • Construction grammar framework • Model of language understanding • Our best-fit approach

10. Productive Argument Omission (in Mandarin) 1 • Mother (I) give you this (a toy). 2 • You give auntie [the peach]. 3 • Oh (go on)! You give [auntie] [that]. 4 • [I] give [you] [some peach]. CHILDES Beijing Corpus (Tardiff, 1993; Tardiff, 1996)

11. Arguments are omitted with different probabilities All arguments omitted: 30.6% No arguments omitted: 6.1%

12. Problem: Proliferation of constructions …

13. If the analysis process is smart, then... • The grammar needs only state one construction • Omission of constituents is flexibly allowed • The analysis process figures out what was omitted

14. Constrained Best Fit in Nature inanimate animate framing, compromise society, politics

15. Competition-based analyzer finds the best analysis • An analysis is made up of: • A constructional tree • A set of resolutions • A semantic specification The best fit has the highest combined score

16. Combined score that determines best-fit • Syntactic Fit: • Constituency relations • Combine with preferences on non-local elements • Conditioned on syntactic context • Antecedent Fit: • Ability to find referents in the context • Conditioned on syntactic information, feature agreement • Semantic Fit: • Semantic bindings for frame roles • Frame roles’ fillers are scored

17. Analyzing ni3 gei3 yi2 (You give auntie) Two of the competing analyses: • Syntactic Fit: • P(Theme omitted | ditransitive cxn) = 0.65 • P(Recipient omitted | ditransitive cxn) = 0.42 (1-0.78)*(1-0.42)*0.65 = 0.08 (1-0.78)*(1-0.65)*0.42 = 0.03

18. Using frame and lexical information to restrict type of reference

19. Discourse & Situational Context • child mother • peach auntie • table Can the omitted arg be recovered from context? • Antecedent Fit: ?

20. How good of a theme is a peach? How about an aunt? • Semantic Fit:

21. The argument omission patterns shown earlier can be covered with ONE construction • Each construction is annotated with probabilities of omission • Language-specific default probability can be learned P(omitted|cxn): 0.78 0.42 0.65

22. Leverage processing to simplify representation • The processing model is complementary to the theory of grammar • By using a competition-based analysis process, we can: • Find the best-fit analysis with respect to constituency structure, context, and semantics • Eliminate the need to enumerate allowable patterns of argument omission in grammar • This is currently being applied in models of language understanding and grammar learning.

23. ECG for linguistic analysis • Workbench byLuca Gilardi • wraps the Constructional Analyzer • two different uses • simplifies creation and revising of grammars • helps testing grammars

24. ECG for linguistic analysis • ECG: the notation • two basic primitives: • schemas • constructions • organized in subcase lattices • i.e., hierarchical inheritance structures with (possibly) multiple parents • Ex.: • SlidePast is a subcase of Verb, • which is a subcase of Word, • which in turn is a subcase of RootType (not shown)

25. ECG for linguistic analysis • Workbench • single window • simple! • lattices on the left • editing area in center • grammar file view on the right • top, center: input utterance

26. ECG for linguistic analysis • Workbench • one adds new schemas and constructions in the central pane • they are shown automatically in the lattice representation

27. ECG for linguistic analysis • ECG: the notation • we’ll see what’s needed for analyzing a simple sentence • he slid • we need some notation first • keyword are in bold • ECG is a Construction Grammar • two poles: form and meaning • constructions: • pair form and meaning • schemas • represent the meaning constraint of a construction • subcase of • introduces an inheritance relation in a construction or a schema • other features: • role: introduces a part (or feature) in the structure • evokes: an associated structure that’s neither a part nor a subcase • bindings: • ECG is also a unification grammar • specified by double arrows: <-->

28. ECG for linguistic analysis • ECG: the notation • the semantics of he slid • TrajectorLandmark, SPG • conventional image schemas • related by inheritance • SPG inherits all TL’s roles: • trajector, landmark, profiledArea • MotionAlongAPath • actions involving a protagonist • the path is represented by the evoked SPG • evokes introduces a new role (spg in this case) • the mover is bound to the trajector of the evoked SPG schema TrajectorLandmark roles trajector landmark profiledArea schema SPG subcase of TrajectorLandmark roles source path goal schemaMotionAlongAPath subcaseof Motion evokesSPGasspg constraints mover ↔ spg.trajector

29. ECG for linguistic analysis • ECG: the notation • the semantics of he slid • Motion • asubcase ofProcess • the mover and the protagonist are bound together by the double arrows • i.e., the mover is the primary participant in a Motion action • the x-net role is typed (via the “:”) to be of the x-schematic type motion • @process is in external ontology • x-schemas • fine-grained process structure representations • e.g. walking, pushing, sliding can all be represented as x-schematic structures (Narayanan 1997) schema Process roles protagonist x-net: @process schemaMotion subcaseofProcess roles mover: @entity speed // scale heading // place x-net: @motion// modified constraints mover ↔ protagonist

30. Schema Lattice Contact MotorControl ForceTransfer Motion Effector Motion SelfMotion ForceApplication CauseEffect MotionPath Effector MotionPath SelfMotion Path SPG Agentive Impact SpatiallyDirectedAction Contact

31. Verb Constructions cxn BITE meaning: ForceApplication schema MotorControl cxn GRASP meaning: ForceApplication schema ForceApplication subcase of MotorControl cxn PUSH meaning: ForceApplication cxn SLAP meaning: AgentiveImpact schema Agentive Impact subcase of ForceApplication cxn KICK meaning: AgentiveImpact cxn HIT meaning: AgentiveImpact

32. ECG for linguistic analysis • ECG: the notation • the semantics of he slid • Just two more schemas • EventDescriptor (or ED) • the meaning of an entire scene • the verbal argument structure is typically bound to the eventType role • the verb’s meaning is usually bound to profiledProcess • ReferentDescriptor (or RD) • typically represents constraints associated with referents of nominal and pronominal constructions schemaEventDescriptor roles eventType: Process profiledProcess: Process profiledParticipant profiledState spatialSetting temporalSetting schema RD roles ontological-category givenness referent number

33. ECG for linguistic analysis • ECG: the notation • the analysis of he slid • Now for the constructions • pair form and meaning • cname.frefers to the form pole of the construction cname • cname.mrefers to its meaning pole • Verb • Word • gives a Verb an orthographic form • HasVerbFeatures • verbal agreement features (number and person) • its meaning is a Process • SlidePast • a Verb with an orthographic form • and an x-schematic motor program • its meaning is MotionAlongAPath generalconstruction Verb subcaseof Word, HasVerbFeatures meaning: Process constructionSlidePast subcaseofVerb form constraints self.f.orth ← "slid" meaning: MotionAlongAPath constraints self.m.x-net ← @slide

34. ECG for linguistic analysis • ECG: the notation • the analysis of he slid • Clause-level construction • Declarative: brings together • a subject (an NP constituent), • the construction for He is a subcase of NP • and a finite verb phrase, fin, of type VerbPlusArguments • IntransitiveArgumentStructure is a subcase of this (green marks the inherited structure) construction Declarative subcaseofS-With-Subj constructional constituents subj: NP fin: VerbPlusArguments form constraints subj.f before fin.f meaning constraints subj.m.referent↔self.m.profiledParticipant self.m↔fin.ed self.m.speechAct← "Declarative”

35. ECG for linguistic analysis • ECG: the notation • the analysis of he slid • NP • construction He is one of its subcases • NominalFeatures: agreement features of nominals (number, case, gender, ...) • meaning: a Referent Descriptor generalconstruction NP subcaseofRootType constructional: NominalFeatures meaning: RD

36. ECG for linguistic analysis • ECG: the notation • the analysis of he slid • VerbPlusArguments • an ancestor of IntransitiveArgumentStrucure • also a subcase ofArgumentStructure • meaning: a Process (in green the inherited structure) • generalconstructionArgumentStructure • subcaseofHasVerbFeatures • meaning: Process • evokesEventDescriptorased • constraints • self.m ↔ ed.eventType generalconstructionVerbPlusArguments subcaseofArgumentStructure constructional constituents v: Verb constraints self.features ↔ v.features meaning: Process constraints v.m↔ ed.profiledProcess evokesEventDescriptorased self.m ↔ ed.eventType

37. ECG for linguistic analysis • ECG: the notation • the analysis of he slid • SemSpec synthesis • after the best-fit process has terminated • the VerbPlusArgument construction • binds the Verb’s meaning pole with the profiledProcess role of the ED • bind its own meaning pole with the ED’s eventType role • the Declarative cxn • binds that same ED to its meaning pole • constrains the subject’s referent to be the same as its meaning pole’s profiledParticipant • in the form block, simply constrains the subject to appear before the verb

38. ECG for linguistic analysis • ECG: the notation • the analysis of he slid • SemSpec synthesis • after the best-fit process has terminated construction Declarative subcaseofS-With-Subj constructional constituents subj: NP fin: VerbPlusArguments form constraints subj.f before fin.f meaning constraints subj.m.referent↔self.m.profiledParticipant self.m↔fin.ed self.m.speechAct← "Declarative” generalconstructionVerbPlusArguments subcaseofArgumentStructure constructional constituents v: Verb constraints self.features ↔ v.features meaning: Process constraints v.m↔ ed.profiledProcess evokesEventDescriptorased self.m ↔ ed.eventType

39. ECG for linguistic analysis • ECG: the notation • the analysis of he slid • SemSpec synthesis • last piece of analysis: the argument structure chosen by the best-fit process • IAS binds its meaning pole with the Verb’s • constrains the protagonist of the action to be the same as the evoked ED’s profiledParticipant • together with the constraint described above for VerbPlusArguments, implies that the event described by the intransitive argument structure is the same as the one described by its verb constituent. • Goldberg (1995) describes for cases in which the meaning of the verb and argument structure constructions do not unify. (inherited structure in green) constructionIntransitiveArgumentStructure subcaseofVerbPlusArguments constructional constituents v: Verb constraints self.features ↔ v.features self.features.verbform ← FiniteOrGerund meaning: Process constraints • evokes EventDescriptoras ed • self.m ↔ ed.eventType self.m.protagonist ↔ ed.profiledParticipant self.m ↔ v.m

42. ECG for psycholinguistic modeling • The best-fit process in the Analyzer • inspired by cognitive science, psychology, computer science • algorithm is cognitively plausible • scans and incorporates in an interpretation one word at a time • can only entertain a limited number of interpretations • approximates spreading activation with probabilities • combines syntactic and semantic evidence to rank competing interpretations • such process is what we call the best-fit heuristic

43. ECG for psycholinguistic modeling • The best-fit process in the Analyzer • best fit heuristic: cognitive motivation • psychology and psycholinguistics • constraint-based (or interactionist) paradigm • [...] constraint-based models assume that multiple syntactic alternatives are evaluated using both linguistic and non-linguistic sources of constraint. The comprehension system continuously integrates all the relevant and available information in order to compute the interpretation that best satisfies those constraints. (McRae, Spivey-Knowlton, & Tannenhaus, 1998) • models that fit the constraint-based paradigm • Narayanan & Jurafsky (1998) • McRae et al. (1998) • Pado (2007)

44. ECG for psycholinguistic modeling • The best-fit process in the Analyzer • best fit heuristic: cognitive motivation • Connectionist models • best-fit models that use • spreading activation to combine multiple domains • competition between the connectionist model’s units to model competing hypotheses • Examples: • Lane & Henderson (1998): connectionist network for syntactic parsing • Feldman (2006): reduction of language interpretation to connectionist models

45. ECG for psycholinguistic modeling • The best-fit process in the Analyzer • best fit heuristic: cognitive motivation • Construction grammars • defines grammaticality in terms of formal properties (syntax) and function (semantic and pragmatic constraints)

46. ECG for psycholinguistic modeling • The best-fit process in the Analyzer • best fit heuristic: cognitive motivation • Natural Language Processing (CS) • joint models of lexicalized PCFGs can be seen as best-fit models • they use lexical dependency as a proxy for direct semantic information

47. ECG for psycholinguistic modeling • Analyzer: modeling reading times • the best-fit machinery has been tested with real psycholinguistic data • McRae, Spivey, Tannenhaus (McRae at el., 1998) • self-paced reading paradigm with pairs of reduced relative sentences: • The cop arrested by the detective was guilty • The crook arrested by the detective was guilty • Sentences differed on whether the subject was a good agent of the p.p. (cop) or a good patient (crook) • sentence 1 is initially easier at the p.p. • harder at the prepositional phrase and main verb

48. ECG for psycholinguistic modeling • Analyzer: modeling reading times • words presented two at a time • semantic fit affects reading time • explanation: • consequence of violation of semantic expectations The cop arrested by the detective was guilty • the cop arrested is biased towards the cop doing the arresting • by the detective violates such expectation

49. ECG for psycholinguistic modeling • Analyzer: modeling reading times • data from Penn TreeBank, Propbank, original data from McRea et al. to approximate constituent filler probabilities • simple grammar • 40 reduces samples from McRea et al. • 40 unreduced samples as baseline

50. ECG for psycholinguistic modeling • Analyzer: modeling reading times • some discrepancies due to best-fit heuristic chosen • results qualitatively accurate nonetheless