Situation Models and Embodied Language Processes

1. Situation Models and Embodied Language Processes Franz Schmalhofer University of Osnabrück / Germany • Memory and Situation Models • Computational Modeling of Inferences • What Memory and Language are for • Neural Correlates • Integration of Behavioral Experiments and Neural Correlates (ERP; fMRI) by Formal Models

2. Text comprehension • Mary heard the ice-cream van coming. • She remembered the pocket money. • She rushed into the house.

3. Kintsch, Welsch, Schmalhofer & Zimny (1990)

4. Original/Explicit: Mary heard the ice cream van coming. Paraphrase: Mary noticed the ice cream van coming. Inference:Mary picked up her pocket money. False/Incorrect: Mary was 50 years of age. How the strengths of the different memory representations can be empirically determined

5. Propositional Representations Many cognitive science theories assume that knowledge and/or the meaning of sentences is represented by propositions, semantic nets and the like (e.g. Anderson, 1976; Kintsch, 1974; Collins & Quillian, 1969; Schank 1975, Schank & Abelson, 1978) Example: The propositional representation of the sentence George loves Sally [LOVES(GEORGE,SALLY)] Compare to: Two word sentences of children during language learning; Protolanguage (Bickerton, 1981, 1995)

6. The difference between pictures and perceptions Situation models are formed by perceptual symbols

7. Comprehension Comprehension includes a large range of topics in cognitive psychology: • pattern recognition, • knowledge representations, • Working memory, • Recognition and recall, • learning, problem solving and decision making Kintsch, W. (1998). Comprehension as a Paradigm for Cognition

8. The Construction-Integration Model (Kintsch 1992) Comprehension: a two phase process Construction: Constructing mental units and interconnecting them in a network Integration: Integration of constructed units via a context sensitive process

9. The Construction-Integration Model (Kintsch 1992)

10. Text Comprehension Up to the 1980`s language comprehension was mostly viewed as the representation of the meaning of the text itself (focus on propositional representations) Now language is viewed as a set of processing instructions (Zwaan, 2004) on how to construct a mental representation of the described situation (mental model or situation model) (Johnson-Laird, 1983; van Dijk & Kintsch, 1983)

11. Situation Models as Event Indices (Zwaan and Radvansky, 1998): The Event-indexing model of Zwaan & Radvansky (1998) suggests that readers monitor five indexes (aspects) of the evolving situation model at the time when they read stories: • Protagonist • Temporality • Causality • Spatiality • Intentionality Or more generally: space, time, causes, agents, intentions; In other words: everything that is relevant for planning actions and predicting future perceptions

12. Text general, in a natural language relatively short sentence may describe a single attribute of a concept The function first requires one argument. The argument of the function first must be a list. The function first returns the first element of the argument. Text of LISP function description Examples specific, possibly a large set of examples required exemplification of many attribute instances (FIRST `(A B))  A(FIRST `((A) B))  (A)(FIRST `(A (B C)))  A(FIRST `(A))  A How do people acquire knowledge from different materials? Properties of The Function First Informational Equivalence of different learning materials

13. A unifying model (KIWi-Model)Schmalhofer (1998) common sense related domain knowledge situation model text repres. Sensory encoding text direct experience

14. Regressive eyemovements during reading

15. Reading times for studying examples

16. Memory retrieval after learningfrom text or examples text novices text experts example novices example experts

17. Correct Responses in Example Verification Task as a Function of Different Amounts of Text

18. Summary of the tested predictions of KIWi-Model • KA is a goal-driven process, consisting of construction and integration phases • Text and examples can be equated for informational contents • The material-related representations are constructed by general heuristics, the situation model depends on domain knowledge • Experts construct and use deep knowledge (situation model), novices rely on material-related representations • Integrative KA (perception, language and memory) instead of the dominance of one source

19. Text Comprehension(learning from text) memory issues encoding processes retrieval representational issues Concept Formation(learning from examples) search processes hypothesis formation concept identification Research on and “memory paradigm“ ”Problem solving paradigm“ Two alternative ways of acquiring knowledge

20. Comparison to Other Cognitive Models

21. Importance of Unified Theories • Consistent and consensual theories drive cumulative progress • The chronology of research produces a need for readjustment of the mappings between theoretical constructs and empirical data • Natural Texts versus Textoids • Changes in Task Types (Priming, Reading time, Memory) • Changes in Available Methods (lateralized presentation,fMRI, ERP) • Possibility that some controversies can be resolved by synthesis into a unified theory • For Example: Predictive and Bridging Inferences

22. “Mary heard the ice-cream van coming.She remembered the pocket money.She rushed into the house.” What types of inferences are there and when are they drawn? Referential Case structure assignment(role: e.g. agent) Causal antecendent Superordinate goal Thematic Character emotional reaction Causal consequence Bridging Predictive Types of inferences (Graesser et al., 1994)

23. Landscape of Inferences (from Graesser, 2003; HC) TYPES OF INFERENCES • Referential • Case structure role assignment • Instantiation of a noun category • Superordinate goal • Superordinate goal or action • Instrument • Causal antecedent • Causal consequence • Character emotional reaction • Emotion of reader • State • Themes • Author‘s intent Causal consequence: The inference is on a forecasted causal chain, including physical events, psychological events, and new goals, plans, and actions of agents. Causal antecedent: The inference is on a causal chain that bridges the current explicit action, event, or state to the previous passage context.

24. Construction and persistence of predictive and bridging inferences(e.g. McKoon & Ratcliff, 1986; Potts et al., 1988; Keefe & McDaniel (1993)

25. Kintsch‘s (1998) C-I theory Multi-level representation: surface-level, propositional text representation and situation model Processing cycles with construction-integration phases Enhancing assumptions Situation models are built from perceptual symbols (Zwaan et al. 2001); they often build a visuo-spatial representation (Fincher-Kiefer, 2002) Instead of a nominal distinction of inference types, like predictive, bridging, causal etc. inference,a functional description of cognitive processes Similar to object constancy in visual perception, a situation constancy is postulated in the formation of situation models Inferencing achieves this situation constancy, i.e. inferencing as a pattern completion process Overarching theoretical assumptions

26. The 2nd processing cycle for the explicit and predictive conditions

27. Model predictions of the inference encoding scores for Keefe & McDaniel data MODEL DATA Input Cycle 2 3 2 3 ______________________________________________________ Explicit 33 32 30* 33* Predictive Inference 24 6 35* 4 Bridging Inference 22 22* ______________________________________________________

28. Evaluation of experimental predictions • Experiment: 2 (instructions) x 4 (text) mixed factorial design • 1) situation condition: elaborate on the context described in the passage • 2) concentrate on the precise wording of the sentences • Textmaterials from McKoon & Ratcliff (1986), Potts et al. (1988), …. • Latencies in word pronunciation task • Sentence recognition task – Pr(yes=old) as dependent measure

29. Reaction time in ms in Pronunciation Task (to inference targets) Predictive Bridging Explicit Control Situation 569 556 571 614 Word 595 576 550 609

30. Text- and situation focused reading (3-rd processing cycle) ModelData Reading Focus text situation text situation __________________________________________________________________ Condition Explicit 23 63 59* 43* Predictive inference 5 61 14 45* Bridging inference 21 41 33* 58* _________________________________________________________________

31. Sentence Recognition Task Explicit probe • The cameraman was preparing to shoot closeups. Inference probe • The actress was pronounced dead. Elaboration probe • The actress died from her injuries. Inconsistent probe • The actress lived a long life.

32. Pr („explicitly mentioned“) in sentence recognition task Text TypesPredictive Bridging Explicit Control _________________________________________________________________________________ Reading focus text situation text situation text situation text situation _____________________________________________________________________________ Explicit .72 .82 .75 .94 .77 .86 .51 .52 Inference .22 .29 .27 .36 .93 .87 .15 .11 Elaboration .24 .33 .26 .42 .23 .38 .15 .16 Inconsistent .11 .12 .14 .05 .11 .04 .12 .11

33. Strength of situational representation for the critical consequence as d‘-values (from elaboration and inconsistent statements)

34. Griesel, Friese & Schmalhofer (2003)

35. Modeling predictive and bridging inferences in comparison to explicit statements • Differences in interconnectivity are the key: • High interconnectivity at situation level (predictive) compared to • High interconnectivity at the propositional level (explicit) Schmalhofer, McDaniel, and Keefe (2002) • Focus on situation and time course may even keep predictive inferences activated in a later processing cycle (this was also the model prediction) McDaniel, Schmalhofer, and Keefe (2001)

36. Application to Beeman, Bowden and Gernsbacher (2000) data • Differential contribution of LH an RH in inference generation LH fine semantic coding Strong activation of small semantic fields RH coarse semantic coding Weak activation of large semantic fields Both hemis process in parallel with sharing at critical times Activation of hemis assessed at predictive and bridging inference points • Mapping to CI model LH is verbatim and propositional RH is situation

37. Bob took his daughter Karen out of school for the day so she could enjoy a very historic event that would take place that morning. The shuttle sat on the ground in the distance, (1) waiting for the signal to be given (2). After a huge roar (3). and a bright flash, the shuttle disappeared into space (4). Leaving clouds of smoke in the wake (5), and the audience cheered. Experiment of Beeman et al. (2000) predictive LH - RH+ bridging LH + RH + predictive LH - RH +

38. Beeman, Bowden and Gernsbacher (2000)

40. Relative frequency of “yes”-responses in the verification task for left visual and right visual field presentations (Griesel et al., 2003)

41. Mean latencies in ms of the “yes”-responses in the verification task for left visual and right visual field presentations

42. Summary • Experimentation for differentiation; Theorizing for integration; • Theories of text comprehension can be instantiated to simulate data from multiple experiments in detail • Systematic relation of dependent and independent variables to the different conceptual entities in models • Integration of existing data and theories is exciting, especially in view of ERP and new brain imaging data, related to inferencing