Cognitive Processes during Text Comprehension

1. Cognitive Processes during Text Comprehension Paul van den Broek University of Minnesota

2. Outline • Components of text (discourse) comprehension • Product • processes • A framework that captures the complex interactions between on-line processes during reading and the gradual construction of a memory representation of the text • Connections to cognitive neuroscience

3. Computational Model Cognitive Theory Neuro-imaging Studies

4. 1A. The product of discourse comprehension: Episodic Memory Representation • A network representation, consisting of elements from text and semantic memory connected by semantic relations (e.g., Kintsch, 1988; Graesser & Clark, 1985; Trabasso & van den Broek, 1985) • During reading, the reader activates information relevant to the current text, establishing semantic relations • between parts of the text • between text and background knowledge

5. Referential and Causal relations • Example The lady gave the waiter $100 He returned to give her the change • In complete texts, relations are more complex: • May require background knowledge • May extend over long distances • May require coordination of multiple pieces of information • The language may signal: The moon exerts gravitational pull on the earth, and thus played an important role in the development of life on earth.

7. Memory as a function of number of connections in network

8. Priming as a function of network distance

9. Evidence for Network Representation • The more connections an event has, the better it is remembered • Reminding someone of one part of the text primes parts that are related more than parts that are not related (e.g., naming times, lexical decisions, speeded recognition) • Events with many connections are judged more important, more often included in summaries

10. 1b. Cognitive processes during comprehension • Limited attentional capacity • As information becomes activated, other information becomes deactivated • Fluctuating activations of concepts as reader proceeds through text

11. Fluctuating activations-A view from the sentence • At each reading cycle, four possible sources of activation: • Current input • Carry over from prior text • Episodic memory representation of prior text (occasionally physical text) • Semantic (background) knowledge • Activation from these sources occurs through distinct processes

12. Comprehension processes • Two mechanisms for accessing memory (Hagoort M): • spread-of-activation (memory-based, associative) processes • coherence-oriented processes • Resulting in • Look-backs • Reinstatement from episodic memory • Retrieval from semantic memory • Connecting inferences (content in Working memory)

13. Testing hypotheses about reading processes • Naming, speeded recognition, lexical decision • Speed of response is indication of activation(e.g., O’Brien & Myers, 1987; McKoon & Ratcliff, 1990; Singer, 1994) • Reading times, Gaze duration/eye fixations, intonation in reading aloud

14. Example: Lexical Decision … … John dropped the banana peel on the floor Mary fell on her back … … ‘slip’ but also ‘eat’, ‘yellow’’, ‘pain’

15. Memory-based and Coherence-oriented processes • Balancing act: • limited attentional (working memory) capacity • Need for coherence –standards of coherence (Hagoort: C) • Standards depend on: • Individual differences • Reading goals/instructions • Linguistic input properties • genre • syntax, vocabulary • connectors • Input order • Standards of coherence pertain to both • Which types of relations matter (causal, referential, others) • The strength of the required relations • ‘Good enough’ , shallow vs. deep processing

16. 2. Connecting processes and product –the Landscape Model • The above mechanisms and processes do not take place in isolation • simultaneously • dynamically (different combinations) • interactively (Memory-based may support/interfere with coherence-oriented) • To keep track of the various processes: computational model (cf. Hagoort)

17. 1. A young knight rode through the forest. 2. The knight was unfamiliar with the country. 3. Suddenly, a dragon appeared. 4. The dragon was kidnapping a beautiful princess. 5. The knight wanted to free her. 6. He wanted to marry her. 7. The knight hurried after the dragon. 8. They fought for life and death, 9. Soon, the night’s armor was completely scorched. 10. At last, the knight killed the dragon. 11. He freed the princess. 12. The princess was very thankful to the knight. 13. She married the knight.

19. Gradual emergence of a memory representation • At each cycle, activated concepts are added (or strengthened) in memory representation of the text • Co-activated concepts are connected (or existing connections strengthened) (Hagoort: U): • asymptotic (delta) rule • as function of strength of activations of the concepts • Computationally: changing connection matrix • Concepts form interconnected clusters/cohorts • Basis for re-activation in subsequent cycles

20. Interaction process-product • Reading is the dynamic interaction between process/activation and gradually emerging mental representation • Over time: • Fluctuating activations • Different processes (memory-based and coherence-oriented, with subprocesses) • Changes in episodic memory representation

21. Connecting to neuroscience Computational Model Cognitive Theory Neuro-imaging Studies

22. Connections to neuro-imaging studies • By providing detailed description of component processes discourse models can inform neuro-imaging research (and, at same time, be tested) • Avoid molar questions, such as where does causal coherence take place, where does referential coherence take place? • Component processes are recruited by both, at different text points • Basis for experimentation • Insights about brain function provide constraints for theoretical models • On-line record of activities can be matched to time course of neural activity

23. Constraints on theoretical model • Semantic constraints influence generation of inferences during reading: • the stronger the constraints, the more quickly and reliably inference is generated • Possible model: particular inference is generated in particular (set of brain structures, constraints influence efficiency and ease • But possible hemispheric specialization: generation of associates under high constraints: more LH; low constraints: RH (Beeman, Virtue, Sundermeier, Marsolek, others)

24. Neural Pathways From the Eye to the Brain

25. Divided Visual Field Paradigm Lexical Decision Task Inference word or non-word was presented quickly to the participant's left or right visual field

26. Bridging Inference Example (Virtue, van den Broek, & Linderholm, 2006) Todd was enjoying a vacation with his friends in the French Riviera. He took off his shirt and shoes, and he went for a walk on the beach. At first, Todd didn't see all of the beautiful shells in the sand as he waded out into the water. High Constraint Suddenly he called out to his friends holding his bleeding foot. Low Constraint Suddenly he called out to his friends holding a seashell. Target Word cut

27. Facilitation for Low & High Constraint Bridging Inferences

28. Predictive Inference Example Tom and Krista were standing together holding hands. Both of them were a little nervous, but mostly excited about today. Tom imagined the future as he looked at Krista. High Constraint They were just pronounced as man and wife. Low Constraint They were just announced as college graduates. Target Word kiss

29. Facilitation for Low & High Constraint Predictive Inferences

30. Conclusions Inferences under different degrees of constraints involve different processes –not same processes executed with different degrees of efficiency/ease

31. Neurological processes • Cognitive processes in computational models can be matched with ongoing neuro-imaging data

32. Methods • This is the story of a boy named Tuk who lived in the Artic. He wanted to show that he could be brave by hunting for big animals like his father who was a great hunter. Some people do not like the idea of hunting, but Tuk’s family relies on animals for most of their food and clothing needs… • Narrative • 118 clauses

33. Methods • Magnetoencephalography (MEG) • 248 channel axial gradiometer system (Magnes 3600WH, 4d-Neuroimaging, San Diego, CA) • Sampling: 1017.25Hz (0.1 - 400 Hz) • Non-invasive • Real-time recording • Task • Reading comprehension

34. Results “Tuk the Hunter” setting events/actions integration (temporal, spatial) (causal, explanations)

35. Conclusion • During reading, multiple processes take place, in dynamic and interatcive ways • Discourse models described processes during reading in considerable detail • Mechanisms • Spread-of-activation type processes • Coherence-oriented processes • Fluctuating activations involving • four sources of activation • subprocesses • Standard of coherence modulate the mix • E.g., Causal, referential • Gradual emergence of coherent memory representation • We have just scratched the surface of neurological investigations of text comprehension • Subprocesses (memory-based activation, reinstatement, etc.) rather than global comprehension processes (causal coherence, spatial layout).