Language Understanding and Unified Cognitive Science

1. Language Understanding and Unified Cognitive Science Jerome Feldman International Computer Science Institute U. California at Berkeley Berkeley, CA jfeldman@icsi.berkeley.edu

2. Unified Cognitive Science Neurobiology Psychology Computer Science Linguistics Philosophy Social Sciences Experience Take all the Findings and Constraints Seriously

3. Functionalism In fact, the belief that neurophysiology is even relevant to the functioning of the mind is just a hypothesis. Who knows if we’re looking at the right aspects of the brain at all. Maybe there are other aspects of the brain that nobody has even dreamt of looking at yet. That’s often happened in the history of science. When people say that the mental is just the neurophysiological at a higher level, they’re being radically unscientific. We know a lot about the mental from a scientific point of view. We have explanatory theories that account for a lot of things. The belief that neurophysiology is implicated in these things could be true, but we have very little evidence for it. So, it’s just a kind of hope; look around and you see neurons: maybe they’re implicated. Noam Chomsky 1993, p.85

4. Embodiment Of all of these fields, the learning of languages would be the most impressive, since it is the most human of these activities. This field, however, seems to depend rather too much on the sense organs and locomotion to be feasible. Alan Turing (Intelligent Machines,1948) Continuity Principle of the American Pragmatists

5. Analyzer: Discourse & Situational Context Lectures I. Overview 2. Simulation Semantics 3. ECG and Best-fit Analysis 4. Compositionality 5. Simulation, Counterfactuals, and Inference Constructions Utterance incremental, competition-based, psychologically plausible Semantic Specification: image schemas, bindings, action schemas Simulation

6. Psycholinguistic evidence • Embodied language impairs action/perception • Sentences with visual components to their meaning can interfere with performance of visual tasks (Richardson et al. 2003) • Sentences describing motion can interfere with performance of incompatible motor actions (Glenberg and Kashak 2002) • Sentences describing incompatible visual imagery impedes decision task (Zwaan et al. 2002) • Simulation effects from fictive motion sentences • Fictive motion sentences describing paths that require longer time, span a greater distance, or involve more obstacles impede decision task (Matlock 2000, Matlock et al. 2003)

7. Neural evidence: Mirror neurons • Gallese et al. (1996) found “mirror” neurons in the monkey motor cortex, activated when • an action was carried out • the same action (or a similar one) was seen. • Mirror neuron circuits found in humans (Porro et al. 1996) • Mirror neurons activated when someone: • imagines an action being carried out (Wheeler et al. 2000) • watches an action being carried out (with or without object) (Buccino et al. 2000)

8. Foot actions Hand actions Mouth actions The Mirror System Buccino et al., 2001 The mirror system, like the motor system, is somatotopically organized. • humans watching videos of actions without objects • humans watching same actions with objects foot hand mouth

9. Fast Brain ~ Slow Neurons Mental Connections are Active Neural Connections There is No Erasing in the Brain

10. Movement vs. ActionsPulvermueller Lab

11. 1000 operations/sec 100,000,000,000 units 10,000 connections/ graded, stochastic embodied fault tolerant evolves learns 1,000,000,000 ops/sec 1-100 processors ~ 4 connections binary, deterministic abstract, disembodied crashes frequently explicitly designed is programmed Brains ~ Computers

12. Learning early constructions (Chang, Mok) The ICSI/BerkeleyNeural Theory of Language Project ECG

13. walker at goal energy walker=Harry goal=home Active representations • Many inferences about actions derive from what we know about executing them • Representation based on stochastic Petri nets captures dynamic, parameterized nature of actions • Walking: • bound to a specific walker with a direction or goal • consumes resources (e.g., energy) • may have termination condition(e.g., walker at goal) • ongoing, iterative action

14. Learning Verb MeaningsDavid Bailey A model of children learning their first verbs. Assumes parent labels child’s actions. Child knows parameters of action, associates with word Program learns well enough to: 1) Label novel actions correctly 2) Obey commands using new words (simulation) System works across languages Mechanisms are neurally plausible.

15. System Overview

16. Learning Two Senses of PUSH Model merging based on Bayesian MDL

17. Learning early constructions (Chang, Mok) The ICSI/BerkeleyNeural Theory of Language Project ECG

18. The Binding Problem • Massively Parallel Brain • Unitary Conscious Experience • Many Variations and Proposals • Our focus: The Variable Binding Problem

19. SHRUTI • SHRUTI does inference by connections between simple computation nodes • Nodes are small groups of neurons • Nodes firing in sync reference the same object

22. Proposed Alternative Solution • Indirect references • Pass short signatures, “fluents” • Functionally similar to SHRUTI's time slices • Central “binder” maps fluents to objects • In SHRUTI, the objects fired in that time slice • Connections need to be more complicated than in SHRUTI • Fluents are passed through at least 3 bits • But temporal synchrony is not required

23. Analyzer: Discourse & Situational Context Lectures I. Overview 2. Simulation Semantics 3. ECG and Best-fit Analysis 4. Compositionality 5. Simulation, Counterfactuals, and Inference Constructions Utterance incremental, competition-based, psychologically plausible Semantic Specification: image schemas, bindings, action schemas Simulation

24. Ideas from Cognitive Linguistics • Embodied Semantics (Lakoff, Johnson, Sweetser, Talmy • Radial categories (Rosch 1973, 1978; Lakoff 1985) • mother: birth / adoptive / surrogate / genetic, … • Profiling (Langacker 1989, 1991; cf. Fillmore XX) • hypotenuse, buy/sell (Commercial Event frame) • Metaphor and metonymy (Lakoff & Johnson 1980, …) • ARGUMENT IS WAR, MORE IS UP • The ham sandwich wants his check. • Mental spaces (Fauconnier 1994) • The girl with blue eyes in the painting really has green eyes. • Conceptual blending (Fauconnier & Turner 2002, inter alia) • workaholic, information highway, fake guns • “Does the name Pavlov ring a bell?” (from a talk on ‘dognition’!)

25. boundary bounded region Image schemas LM • Trajector / Landmark (asymmetric) • The bike is near the house • ? The house is near the bike • Boundary / Bounded Region • a bounded region has a closed boundary • Topological Relations • Separation, Contact, Overlap, Inclusion, Surround • Orientation • Vertical (up/down), Horizontal (left/right, front/back) • Absolute (E, S, W, N) TR

26. Schema Formalism SCHEMA <name> SUBCASE OF <schema> EVOKES <schema> AS <local name> ROLES < self role name>: <role restriction> < self role name> <-> <role name> CONSTRAINTS <role name> <- <value> <role name> <-> <role name>

27. A Simple Example SCHEMA hypotenuse SUBCASE OF line-segment EVOKES right-triangle AS rt ROLES Comment inherited from line-segment CONSTRAINTS SELF <-> rt.long-side

28. CAFE Language understanding: analysis & simulation constructionWALKED form selff.phon [wakt] meaning : Walk-Action constraints selfm.time before Context.speech-time selfm..aspect  encapsulated “Harry walked into the cafe.” Utterance Analysis Process Constructions General Knowledge Semantic Specification Belief State Simulation

29. Semantic specification • The analysis process produces a semantic specification that • includes image-schematic, motor control and conceptual structures • provides parameters for a mental simulation

30. Task: Interpret simple discourse fragments/ blurbs France fell into recession. Pulled out by Germany US Economy on the verge of falling back into recession after moving forward on an anemic recovery. Indian Government stumbling in implementing Liberalization plan. Moving forward on all fronts, we are going to be ongoing and relentless as we tighten the net of justice. The Government is taking bold new steps. We are loosening the stranglehold on business, slashing tariffs and removing obstacles to international trade.

32. Results • Model was implemented and tested on discourse fragments from a database of 50 newspaper stories in international economics from standard sources such as WSJ, NYT, and the Economist. • Results show that motion terms are often the most effective method to provide the following types of information about abstract plans and actions. • Information about uncertain events and dynamic changes in goals and resources. (sluggish, fall, off-track, no steam) • Information about evaluations of policies and economic actors and communicative intent (strangle-hold, bleed). • Communicating complex, context-sensitive and dynamic economic scenarios (stumble, slide, slippery slope). • Commincating complex event structure and aspectual information (on the verge of, sidestep, giant leap, small steps, ready, set out, back on track). • ALL THESE BINDINGS RESULT FROM REFLEX, AUTOMATICINFERENCES PROVIDED BY X-SCHEMA BASED INFERENCES.

33. Embodied Construction Grammar • Embodied representations • active perceptual and motor schemas • situational and discourse context • Construction Grammar • Linguistic units relate form and meaning/function. • Both constituency and (lexical) dependencies allowed. • Constraint-based (Unification) • based on feature structures (as in HPSG) • Diverse factors can flexibly interact.

34. Embodied Construction GrammarECG(Formalizing Cognitive Linguistics) • Community Grammar and Core Concepts • Deep Grammatical Analysis • Computational Implementation • Test Grammars • Applied Projects – Question Answering • Map to Connectionist Models, Brain • Models of Grammar Acquisition

35. Verb Constructions Construction BITE1 subcase of Verb form: bite meaning: ForceApplication constraints: Effector ← teeth Routine ← bite // close mouth schemaForceApplication subcase ofMotorControl evokesForceTransfer as FT roles Actor↔ FT.Supplier ↔ Protagonist Acted Upon ↔ FT.Recipient Effector Routine Effort↔ FT.Force.amount

36. Semantic SpecificationHe bit the apple EventDescriptor eventtype ProfiledProcess ProfiledParticipant CauseEffect causer affected ForceApplication actor actedupon routine  bite effector  teeth RD27 category Person RD55 category Apple

37. Modeling context for language understanding and learning • Linguistic structure reflects experiential structure • Discourse participants and entities • Embodied schemas: • action, perception, emotion, attention, perspective • Semantic and pragmatic relations: • spatial, social, ontological, causal • ‘Contextual bootstrapping’ for grammar learning

38. Constrained Best Fit in Nature inanimate animate

39. Computational models Grammatical induction language identification context-free grammars, unification-based grammars statistical NLP Word learning models semantic representations logical forms discrete representations continuous representations statistical models Developmental evidence Prior knowledge concepts event-based knowledge social cognition lexical items Data-driven learning basic scenes lexically specific patterns usage-based learning Two perspectives on language learning

40. Language Acquisition • Opulence of the substrate • Prelinguistic children already have rich sensorimotor representations and sophisticated social knowledge • intention inference, reference resolution • language-specific event conceptualizations (Bloom 2000, Tomasello 1995, Bowerman & Choi, Slobin, et al.) • Children are sensitive to statistical information • Phonological transitional probabilities • Most frequent items in adult input learned earliest (Saffran et al. 1998, Tomasello 2000)

41. Form: Participants : Mother, Naomi, Ball Scene : Discourse : text : throw the ball intonation : falling Throw thrower : Naomi throwee : Ball speaker :Mother addressee Naomi speech act : imperative activity : play joint attention : Ball Experiment: learning verb islands • Question: • Can the proposed construction learning model acquire English item-based motion constructions? (Tomasello 1992) • Given: initial lexicon and ontology • Data: child-directed language annotated with contextual information

42. before before The intuition behind learning a new form-meaning pairing from context Put-Action put-agent put-theme location construction Put Coat construction Coat Sofa construction Here

43. The learner learns a new lexically-specific construction from the form-meaning pair construction Put-Coat-Here constituents v: Put o: Coat p: Here form vf before of before pf meaning: Caused-Motion-Scene selfm.means  vm selfm.mover  om selfm.path  pm

44. Experiment: learning verb islands Subset of the CHILDES database of parent-child interactions (MacWhinney 1991; Slobin ) • coded by developmental psychologists for • form: particles, deictics, pronouns, locative phrases, etc. • meaning: temporality, person, pragmatic function,type of motion (self-movement vs. caused movement; animate being vs. inanimate object, etc.) • crosslinguistic (English, French, Italian, Spanish) • English motion utterances: 829 parent, 690 child utterances • English all utterances: 3160 adult, 5408 child • age span is 1;2 to 2;6

45. A quantitative measure: coverage • Goal: incrementally improving comprehension • At each stage in testing, use current grammar to analyze test set • Coverage = % role bindings analyzed • Example: • Grammar: throw-ball, throw-block, you-throw • Test sentence: throw the ball. • Bindings: scene=Throw, thrower=Nomi, throwee=ball • Parsed bindings: scene=Throw, throwee=ball • Score test grammar on sentence: 2/3 = 66.7%

46. Learning to comprehend

47. world knowledge utterance comm. intent constructicon reinforcement (usage) reinforcement (usage) analyze & resolve generate hypothesize constructions & reorganize discourse & situational context analysis utterance simulation response reinforcement(correction) reinformcent (correction) Usage-based learning,comprehension, and production

48. Unified Cognitive Science Neurobiology Psychology Computer Science Linguistics Philosophy Social Sciences Experience Take all the Findings and Constraints Seriously

49. The ICSI/BerkeleyNeural Theory of Language Project • Alumni • Terry Regier (UCB Ling, CogSci) • Johno Bryant (Ask) • David Bailey (Google) • Leon Barrett (Google) • Nancy Chang (Sony Paris) • Joe Makin (UCSF) • Eva Mok (U. Chicago) • Andreas Stolcke (ICSI, SRI) • Dan Jurafsky (Stanford Ling) • Olya Gurevich (Powerset) • Benjamin Bergen (UCSD) • Carter Wendelken (UCB) • Srini Narayanan (ICSI, UCB) • Steve Sinha (US Govt.) • Gloria Yang (UCSF) • Luca Gilardi (ICSI) • Principal investigators • Jerome Feldman (UCB,ICSI) • George Lakoff (UCB Ling) • Srini Narayanan (UCB,ICSI) • Lokendra Shastri (now India) • Affiliated faculty • Chuck Fillmore (ICSI) • Eve Sweetser (UCB Ling) • Rich Ivry (UCB Psych) • Lisa Aziz-Zadeh (USC) • Graduate Students • *Ellen Dodge (Ling) • Michael Ellsworth (Ling) • Joshua Marker (Ling) • Shweta Narayan (Ling)