Simulation Semantics

1. Simulation Semantics Ben Bergen Tuesday Seminar October 7, 2003

2. Thanks • Jerome Feldman • Susanne Gahl • Art Glenberg • Teenie Matlock • Shweta Narayan • Srini Naryanan • Terry Regier • Amy Schafer

3. The basic question • What is meaning?

4. The logical solution • One popular answer is that meaning is like a formal logic system • Words map onto pieces of logical statements • Verbs map onto predicates • Nouns identify arguments of those predicates

5. The logical solution • LanguageJohn climbed the tree • Logical Form climbed(John,tree)

6. The logical solution • Problem 1: We’re terrible at logic • Problem 2: We understand language before logic • Problem 3: What does “Hi” mean, logically? • Problem 4: No empirical evidence language users perform logical operations during language use • Problem 5: Even if meaning involves logic, this just pushes back the question of meaning - what does it mean for climbed(John,tree) to be meaningful?

7. Situating meaning • One partial solution to the problem of what the logic means is to say that it’s about the world • Once you have a logical representation of the meaning of a sentence, you can interpret it in terms of the world • John climbed the tree is true if and only if some particular John climbed some particular tree

8. Situated meaning • LanguageJohn climbed the tree • Logical Form climbed(John,tree) • World climbed5(John10,tree47)

9. Situating meaning • Problem 1: We have incomplete knowledge of the world • Problem 2: Some of our world knowledge is created through language, and thus cannot presuppose it. (What’s a Jabberwocky?) • Problem 3: Many to most of the semantic distinctions we can make have to do with our interpretation of the world, not the world itself • The dresser runs along the wall vs. The dresser leans against the wall. • John isn’t stingy, he’s thrifty. • Problem 4: When we say that meaning is about the world, what does that mean is going on inside a language user’s head?

10. How about embodiment? • One candidate solution is the idea of embodiment (Lakoff 1987, Langacker 1991, Talmy 2000) • Language gains meaning through embodied experiences that the language user has had, that the language relates to • So language is meaningful by evoking recollection of past experiences, often combined in new ways

11. Simulation • Understanding language requires the language user to simulate, that is, mentally imagine, the content of the utterance • If I say “The pink elephants danced across the stage”, you have to imagine the scene • If I say “John climbed the tree”, you imagine a human male climbing a tree and experience motor and perceptual content of performing and/or observing the action • That’s what it means for that sentence to be meaningful - communication is mind control

12. “I don’t buy it” • Brain areas that control specific motor actions are used to perceive, imagine, and process verbs and sentences about those very same actions • Brain areas responsible for specific parts of the visual field are also used to process sentences that describe events occurring in those same regions • Motor and perceptual imagery are components of meaningful language understanding

13. Simulated meaning • LanguageJohn climbed the tree • ? • Simulation • World

14. Mirror neurons • Parts of the motor systems are used for things other than action (mirror neurons) • Pre-motor and parietal cortex are used in action perception (Gallese et al. 1996, Rizzolatti et al. 1996, Boccino 2002) • Imagination of motor action (Jeannerod 1996, Lotze et al 1999) • Recall of motor action (Nyberg et al 2001)

15. Effector-specific motor areas are used in action and perception Foot Action Hand Action Mouth Action

16. Mirror neurons • Processing motor action language • Differential motor area activation to mouth, leg verbs in lexical decision (Pulvermüller et al 2001) • Differential motor area activation in passive listening to hand, mouth, leg sentences (Tettamanti et al ms)

17. Mirror neurons in understanding

18. Mirror neurons • In other words, understanding the meaning of words such as walk or grasp involves mental simulation, executed by same functional clusters used in action and perception • In other other words, in understanding action language like walk or grasp, one imagines oneself walking or grasping an object, for example

19. Study 1 - A behavioral study • Are the same structures used for perceiving motor actions also used for understanding motor language? • If so, processing of action images and action language should interfere with each other

20. Study 1 - Method • An image-verb matching task • A visual image (1s) representing an action using one of three effectors - mouth, hand, or foot. • A mask, for 450 msec (50 msec blank) • A written verb, either a good descriptor of the image or not • Ss decided quickly if verb described image well

21. Study 1 - Method • Conditions • Matching: verb matched image (1/2 trials) • Non-matching, same effector (1/4 trials) • Non-matching, different effector (1/4 trials)

22. Study 1 - Method • The perception and verbal meaning sub-tasks might use the same (mirror) circuitry • If so, non-matching images and verbs should activate different circuits • In general, the more similar representations are (e.g. if they share an effector) the more mutual inhibition they should exhibit • Hypothesis: Interference when verb and image don’t match but use same effector, thus slower reactions than with different effectors

23. Matching Condition

26. trip

27. Non-Matching, Same Effector Condition

30. punch

31. Non-Matching, Different Effector Condition

34. reach

35. Study 1 - Method • 48 images presented twice to 39 Ss - in matching and one non-matching condition • Verbs were associated with images in a pretest; subjects named verbs best describing images • Matching verb got the most responses • Non-matching same effector verb: no subjects listed as a description of the image, and didn’t seem confusable • Non-matching diff. effector randomly selected

36. Study 1 - Results • p<0.01 in subject and item analyses

37. Study 1 - Interpretation • Explanation 1: overall similarity between the two actions, not shared effector • Ss might take longer to reject verbs using the same effector because those verbs had meanings that were globally more similar to the action depicted

38. Study 1 - Interpretation • Measured semantic similarity between non-matching and matching verbs for each image • LSA (Landauer et al 1998) • Method for extracting & representing similarity among texts, based on the contexts they appear in • Two words or texts will be rated as more similar the more alike their distributions are • LSA performs like humans in a range of behaviors, e.g. synonym and multiple-choice tasks • The pairwise comparison function produces a similarity rating from -1 to 1 for any pair of texts

39. Study 1 - Interpretation

40. Study 1 - Interpretation • Explanation 2: images in the non-matching, same effector condition might be more interpretable as the non-matching verb • Indication: non-matching same effector verbs selected from among those that no subjects in the pretest offered as descriptors of the image • Follow-up experiment using verbal stimuli yielded the same effect (but stronger!)

41. Study 1 - Discussion • Mirror circuits used in executing and recognizing actions are specific (Gallese et al. 1996), e.g. to gesture type, like precision grip • Neural structures encoding similar actions are more likely to become co-active (in action or perception), leading to confusion • Therefore, the more similar two actions are, the more strongly neural structures encoding them must mutually inhibit eachother

42. Study 1 - Discussion • The matching task asked subjects to determine whether two inputs were the same • In non-matching conditions, this could be strong and stable activation of two mirror circuits • When the two mirror structures share an effector, they will strongly inhibit each other • It will thus take longer for two distinct mirror structures to become active, and for a subject to arrive at two distinct action perceptions • Unlike actions have less lateral inhibition, so two mirror structures become co-active more quickly

43. Study 1 - Conclusions • Understanding motion verbs seems to use resources overlapping with those used in perceiving and executing actions • This supports an embodied view of human semantics, where linguistic meaning is tightly linked to detailed motor knowledge

44. Study 2 - Introduction • Motor imagery seems to be activated for language understanding. How about perceptual imagery? • For example, when processing motion sentences involving upwards or downwards motion, language users might internally imagine an object moving upwards or downwards

45. Study 2 - Background • Richardson, Spivey, Barsalou, and McRae (2003) tested whether processing sentences with up or down meanings interferes with visual processing • Visual imagery interferes with visual perception (the “Perky effect”), so visual imagery when understanding a sentence could impede object categorization • (1) a spoken sentence, then (2) a visual categorization task - a circle or square appeared up, down, right or left, and (3) an interleaved comprehension task • Four types of sentence (determined by a rating test) • Concrete Horizontal: The miner pushes the cart. • Concrete Vertical: The plane bombs the city. • Abstract Horizontal: The husband argues with the wife. • Abstract Vertical: The storeowner increases the price.

46. Study 2 - Background • Richardson et al’s findings • Interference when sentence and object in same condition • Abstract difference was significant; concrete difference was not - this is counter-intuitive

47. Study 2 - Background • Problems with the experiment • No way to tell whether it’s the sentences in their entirety or some component of them (e.g. nouns, verbs, clausal constructions) that yields the effect • E.g. The balloonfloatsthrough the cloud • Conflates up and down, left and right • The results may be the product of combined facilitory and inhibitory effects • Conflates sentences with very different semantics • The ship sinks in the ocean & The strongman lifts the barbell • The storeowner increases the price & The girl hopes for a pony

48. Study 2 - Experiment • Idea: redo the Richardson et al experiment, fixing problems • Sentences are all intransitive, and differ only in the direction of their verb (only concrete sentences discussed here) • The chair toppled vs. The mule climbed • Separate conditions for sentences encoding up vs. down

49. Study 2 - Norming • Sentences in each pair of conditions were closely balanced along two factors (no signif. diff.) • Sentence reading (push the button as soon as you understand the sentence) • Sentence meaningfulness rating (1-7) • Verbs in each condition were rated as strongly related to up or down • up/down rating (1-7) • Subject nouns were controlled for their upness or downness in each pair of conditions • So any differences would derive from differences in the verb (possibly affecting the sentence)

50. Study 2 - Experiment • Method (65 subjects) • After fixation cross (1s), one of two sentence types • Literal Up: The mule climbed • Literal Down: The chair toppled • 150 msec later, a circle or square appears in a quadrant of the screen: up, down, right, or left (for 150 msec) • In critical trials, up or down. As many up- or down-related sentences were followed by an object on left or right • To ensure that subjects listened to the sentences for content, a comprehension task was interleaved • Hypothesis: interference when sentence direction coincides with the object location