1 / 31

How do we process text with spatial information?

How do we process text with spatial information?. Marijn E. Struiksma*, Matthijs L. Noordzij**, Bas F.W. Neggers*** & Albert Postma* *Universiteit Utrecht **Radboud Universiteit Nijmegen *** Universitair Medisch Centrum Utrecht. Spatial Language.

dotty
Download Presentation

How do we process text with spatial information?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. How do we process text with spatial information? Marijn E. Struiksma*, Matthijs L. Noordzij**, Bas F.W. Neggers*** & Albert Postma* *Universiteit Utrecht **Radboud Universiteit Nijmegen *** Universitair Medisch Centrum Utrecht

  2. Spatial Language • Spatial language: spatial configuration of the world • Simple sentences • Search directions, identifying people • Complex route finding instructions • Generate mental map • Also blind people

  3. Spatial Language • Aim: gain insight in underlying process and nature of mental representations • Research with sighted and blind • Different input modalities

  4. Spatial Language Processing • Propositional model • Verbal strategy • Strategic model • Verbal strategy or • Visuo-spatial strategy

  5. Different Strategies • Noordzij et al. (2005) • Sentence-sentence • Sentence-picture • 80% expectancy

  6. triangle left of circle 80% 20% Different Strategies + triangle left of circle + Spatial condition

  7. triangle and circle 20% 80% Different Strategies + triangle and circle + Non-spatial condition

  8. Different Strategies Fig. 1 from Noordzij et al. (2005) • Spatial: slower on unexpected pictures • Propositional model: unexpected stimuli – general switch cost Identical for spatial and non-spatial • Strategic model: unexpected stimuli – general switch cost and incompatibility for both sentence and picture

  9. Dual-representational model • Automatic propositional representation • Additional visual-spatial representation • Strategically dependent on context • Neuroimaging parietal areas: • Understanding spatial terms • Visual-spatial representation

  10. EEG: processing spatial sentences • Noordzij et al. 2006 • Similar sentence-sentence and sentence-picture paradigm • Similar behavioral pattern • Event-related potentials (ERPs)

  11. EEG: processing spatial sentences • ERPs Fig. 4 from Noordzij et al. (2006) Parieto-occipital activation for spatial sentences, expecting a picture. visual-spatial strategy

  12. fMRI: processing spatial sentences • Sentence-sentence and sentence-picture paradigm • 100% expectancy for S2 • RT faster for picture than sentence, especially for spatial

  13. fMRI: processing spatial sentences • fMRI Fig. 2 from Noordzij et al. (2008) Spatial > Non-spatial for both stimulus modalities: activity in left Supramarginal gyrus (SMG)

  14. Nature of mental representations • How is spatial language processed in the absence of vision? • Is language processing different for different input modalities? • Is the nature of mental representations modality specific?

  15. fMRI blind & sighted • Sentence-sentence paradigm • Comparing spatial and non-spatial • Auditory version • Scanner-details: • 3T Philips Achieva scanner • PRESTO-SENSE sequence • TR = 500ms • Voxel-size: 4*4*4 mm

  16. Subjects • 13 congenitally blind • Age 36.5 ± 9.8 • 7 male, 6 female • 5 right-handed, 5 left-handed, 3 ambidexter • 13 sighted controls • Age 37.2 ± 11.2 • 8 male, 5 female • 6 right-handed, 4 left-handed, 3 ambidexter

  17. Spatial Sentence Comprehension • Compare two sentences • Do they describe the same situation?

  18. Spatial Sentence Comprehension • Block-design: 4 sessions, 18 blocks per session, 2 trials of 7.5s per block • 4 conditions

  19. Spatial Sentence Comprehension Position = Left of/Right of Size = Taller than/Smaller than Conjunction = Together with Age = Older than/Younger than Position(15s) Size(15s) Conjunction(15s) Interval (6-9s) Age(15s) Interval (6-9s) Interval (6-9s) Interval (6-9s) Presented pseudo-randomly

  20. Performance • Performance is good • No difference CB or SC

  21. Behavioral Results: RT • Main effects of Space and Category • No group difference

  22. Results: Supramarginal Gyrus • Contrast: Position > Combination • ROI around SMG from Noordzij et al. (2008) • Conjunction CB and SC ROI, T=2.5, p = .045 (corrected)

  23. Results: Supramarginal Gyrus • Contrast: Relational > Combination • ROI around SMG from Noordzij et al. (2008) • Conjunction CB and SC ROI, T=2.5, p = .045 (corrected)

  24. Contrast Estimates SMG

  25. fMRI results • Evidence for reorganization • Difference CB and SC for task vs. rest • Occipital areas • No dimension modulation Whole brain, T=4.0,k=10 voxels, p < 0.03

  26. Conclusions 1 • Behavioral results: sighted can generate propositional and visual-spatial representations • Target area: left SMG • Left SMG not influenced by • context (picture or sentence) • input modality (visual or auditory)

  27. Conclusions 2 • Blind also activate left SMG and visual areas • Visual areas are not modulated by Space or Category • Left SMG important for processing spatial prepositions • Other function: ordering on 1 dimension • Activation in left SMG is modality-independent + hardwired

  28. Questions ?

  29. Verb Generation • Block-design • 3 types of blocks • Rest • Non-words • Words

  30. Results • Conjunction CB+SC • Language areas • bilateral Broca, Wernicke • Covert verb generation: • precentral gyrus • supplementary motor area • Cerebellum Word - Nonword Whole brain, T=4.0,k=20 voxels, p < 0.007

  31. Results: Reorganization • Difference CB-SC • CB significantly activate occipital cortex • BA 19, 31, 37 CB vs. SC Whole brain, T=4.5, k=10 voxels, p < 0.019

More Related