1 / 30

Situation Models and Embodied Language Processes

Situation Models and Embodied Language Processes. Franz Schmalhofer University of Osnabrück / Germany. Memory and Situation Models Computational Modeling of Inferences What Memory and Language are for Neural Correlates

zion
Download Presentation

Situation Models and Embodied Language Processes

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Situation Models and Embodied Language Processes Franz Schmalhofer University of Osnabrück / Germany • Memory and Situation Models • Computational Modeling of Inferences • What Memory and Language are for • Neural Correlates • Integration of Behavioral Experiments and Neural Correlates (ERP; fMRI) by Formal Models

  2. Charles A. Perfetti, Michal Balass, Jessica Nelson, Chin-Lung Yang &Edward Wlotko University of Pittsburgh funded by Alexander von Humboldt Foundation Uwe Friese, Markus Raabe, Karin Pietruska Ho-Ming Chow Niki Vavatzanidis Anke Karabanov University of Osnabrück Mark GreenleeRoland Rutschmann University of Oldenburg Acknowledgements

  3. Outline • Behavioral Data and Computational Modeling on Inferencing in Text Comprehension • Event-related potentials (ERP) on the reading of inference-related words • An fMRI-experiment on inference processes and the verification of inference related statements

  4. Text comprehension and Inferencing • Mary heard the ice-cream van coming. • She remembered the pocket money. • She rushed into the house.

  5. Lexicon Meaning Morphology Syntax -argument structure Linguistic System Orthographic Units Word Representation Mapping to phonology Visual Input General Knowledge Comprehension Processes Situation Model Phonology Syntax Morphology Inferences Text Representation A blueprint of the reader (Perfetti, 1999) Parser Meaning and Form Selection Word Identification Writing System Phonological Units Orthography

  6. The KiWi-model (Schmalhofer, 1998) common sense related domain knowledge situation model text repres. Sensory encoding text direct experience

  7. A mapping of mental and brain processes • Beeman, Bowden & Gernsbacher (2000) • Information supporting predictive inferences initially activated in RH • Long & Baines (2002) • Hemispheric differences in representations • Tapiero & Fillon: Emotional inferences and hemispheric differences • Mason & Just (2004) • Inferencing by right hemisphere language network and reasoning network (dorso-lateral prefrontal cortex) • Ferstl (2003, HC); Ferstl & von Cramon (2001) • Shift from local to global aspects  precuneus • Inferencing  fronto-median cortex (FMC) • Situation model (dep. on content)  vFMC

  8. Predictions from the KIWi-model (explicit and paraphrase)

  9. Predictions from the KIWi-model (inference and control conditions)

  10. How many nodes and links had to be newly constructed

  11. Text Materials in ERP experiment

  12. ERP study of inferencing • Use ERPs to examine inference processes. • Vary the accessibility of referents required by the integration (new construction versus preexisting traces) referent previously introduced, possibly inferred or control condition • Measurements taken on a single word that occurs early in the second sentence

  13. The N400 component in ERP-studies • N400: A negativity shift around 400 ms is associated with an incongruent meaning. • “He smeared the bread with socks.” • A marker for semantic processing; sensitive to semantic congruence effects in sentence contexts (Van Petten & Kutas, 1990) • Sensitive to global comprehensibility effects in text (St. George, Mannes, & Hoffman, 1994)

  14. General Procedure of ERP-study Participants: • 32 adult readers Procedure: • Slow-SVP presentation • 600 ms SOA word-to-word • 300ms exposure duration • 300ms interval • Last word of sentence followed by additional 300ms interval Materials: • 120 two-sentence passages for each participant; 30 in each condition; Comprehension Probes (T-F) at random after 25% of trials

  15. Summary (ERP) • A common set of assumptions about construction and integration processes in text comprehension is useful for accounting for behavioral data as well as neural correlates (e.g. ERP) • Inference possibilities yield a cognitive preparation for target concept, but different from explicit and control conditions • Distinction between text and situation representation accounts for behavioral data (priming) and ERP-data quite well

  16. Text Materials of fMRI Experiment

  17. fMRI-Lab

  18. fMRI Lab • 1.5 T Siemens Sonata fMRI-Scanner (Siemens Medical Solutions) • 8 Channel Head Coil • Lumitouch optical response device • Visual Stimulation and recording of responses were controlled by E-Prime and PC

  19. Behavioral Results • Results in Milliseconds • Differences are significant except the following pairings: • Explicit – Paraphrase • Filler – Predictive

  20. fMRI: General Procedure • 13 adult skilled readers • 108 (72 + 36) trials; 18 trials per condition • Experiment is divided into 3 Sessions à 36 trials (6 per condition) • Pseudorandom order of trials • Conditions and domains (themes) were counterbalanced by a Latin Square Design • Different response delays were equalized at the end of each trial to receive a constant trial length

  21. Time sequence of one trial

  22. fMRI - Procedure BOLD sensitive T2 weighted functional sequence (TR = 3s, TE = 50 ms, FOV = 192mm) Rotated app. 10° relating to AC-PC line, covering prefrontal, parietal and temporal regions in full and the majority of the occipital cortex 3 Sessions à 324 continuously acquired scans (app. 48 Minutes) T1 weighted structural sequence (TR= 1900 ms, TE=3.93ms, FOV = 256 mm) Acquisition at the end of the experiment (app. 10 Minutes)

  23. fMRI – Data Analysis • Data were preprocessed and analyzed using The SPM2 Software Package (Functional Imaging Laboratory, Welcome Department of Imaging Neuroscience, London) • Statistical Analysis was conducted by calculating a Random Effects Analysis • Referring to the problem of multiple comparisons only those clusters with a corrected p < 0.05 were considered as significantly activated

  24. The last third of reading modeled as a block Statement Verification Task modeled as a block Reading Sentence versus Pseudoword Reading Sentence Reading in Inference Condition versus Sentence Reading in Explicit Condition Statement Verification Inference vs Explicit Inference vs Paraphrase Control vs Explicit fMRI – Data Analysis

  25. Word-Reading vs Pseudoword-Reading

  26. Summary (fMRI-Experiment) • fronto-median wall: • reasoning, inferencing, situation model integrations (Ferstl, Mason & Just; Robertson & Gernsbacher) • Precuneus, Cuneus, occipital lobe: • Imagery, perception tasks, memory related imagery operations • Posterior Cingulate: • Memory retrieval • Prefrontal: Inferencing • BA9/10: theory of mind

  27. Integration of Fields and Methods Philosophy Linguistics Language sytems Cognitive Neuroscience Cognitive Science Computer Science Electronic machines How the mind works? How the brain enables the mind? Psychology Human Behavior Medicine - Physiology Brain Research Components of the brain, How does it work?

  28. What the course was about: Sth. Old and Std. New • Memory and Situation Models • Computational Modeling of Inferences • What Memory and Language are for • Neural Correlates (ERP, fMRI) • Integration of Behavioral Experiments and Neural Correlates (ERP; fMRI) by Formal Models

More Related