Assessments of Reading at Deeper Levels of Comprehension

1. Assessments of Reading at Deeper Levels of Comprehension Art Graesser Department of Psychology and the Institute for Intelligent Systems

2. Main Messages • We need assessments of reading comprehension that systematically tap deeper levels of comprehension. • Good deep measures integrate reading skills/strategies with world knowledge and nonlinguistic components.

3. Graphemes and phonemes Morphemes and words Syntactic composition Linguistic style and dialect Explicit propositions Referents of referring expressions Common ground Discourse focus versus presuppositions Situation models & inferences Embedded dialog Configuration of multiple agents Genre, registers, rhetorical structures Plot configurations Local and global coherence Point of message -- theme Goals and attitude of author Five major categories(Kintsch, 1998; Graesser, Millis, & Zwaan, 1997) Surface code Propositional textbase Situation model Text genre and rhetorical structure Pragmatic communication Levels of Language & Discourse

4. What do we mean by deep comprehension?

5. Cognitive Processes Bloom (1956) Recognition Evaluation Recall Synthesis Comprehension (Inference) Application Analysis

6. Abstractness of Content (Mosenthal, 1996) • Most concrete: concrete person, action, thing • Highly concrete: observable attributes (action, types, attributes, amounts) • Intermediate: Procedures, goals, and manner • Highly abstract: cause, effect, reason, evidence • Most abstract: inferred theories & themes

7. Good Questions that Tap Pragmatic Communication (Questioning the Author , Beck, McKeown, Hamilton, & Kucan, 1997) • What is the author trying to tell you? • Why is the author telling you that? • Does the author say it clearly? • How could the author have said things more clearly? • What would you say instead?

8. Domain Knowledge Wisher & Graesser (2007) People (Agents) Other Taxonomies Causal Networks Spatial Layout Compositional Structure Procedures & Plans

9. Deep-Level Reasoning Questions (Graesser and Person,1994) LEVEL 1: SIMPLE or SHALLOW 1. Verification Is X true or false? Did an event occur? 2. Disjunctive Is X, Y, or Z the case? 3. Concept completion Who? What? When? Where? 4. Example What is an example or instance of a category? LEVEL 2: INTERMEDIATE 5. Feature specification What qualitative properties does entity X have? 6. Quantification What is the value of a quantitative variable? How much? 7. Definition questions What does X mean? 8. Comparison How is X similar to Y? How is X different from Y? LEVEL 3: COMPLEX or DEEP 9. Interpretation What concept/claim can be inferred from a pattern of data? 10. Causal antecedent Why did an event occur? 11. Causal consequence What are the consequences of an event or state? 12. Goal orientation What are the motives or goals behind an agent’s action? 13. Instrumental/procedural What plan or instrument allows an agent to accomplish a goal? 14. Enablement What object or resource allows an agent to accomplish a goal? 15. Expectation Why did some expected event not occur? 16. Judgmental What value does the answerer place on an idea or advice?

10. Analysis of 120 Multiple Choice Questions in Cognitive Psychology • 120 MCQ items of cognitive psychology textbooks • 4 test banks • 30 items per textbook • Items coded on the level in Graesser & Person taxonomy.

11. Challenges • How can we promote more diversity in the landscape of questions? • How can we encourage deeper questions? • How do we change: • Textbook writers • Professors & teachers • Students

12. OFF-LINE Recall Summarization Retrospective think aloud Recognition tests Cloze procedure Multiple choice tests Sentence verification Question asking & answering Response signal paradigm Ratings Word sorting ON-LINE Immediate think aloud Self-paced reading times Eye tracking Word naming latencies Lexical decision latencies RSVP-SOA Physiological recordings fMRI Evoked potential Cognitive and Behavioral Tests in Experimental Psychology

13. Open-ended Questions The sun exerts a gravitational force on the earth as the earth moves in its orbit around the sun. Does the earth pull equally on the sun?Explain why? EXPECTATIONS The sun exerts a gravitational force on the earth. The earth exerts a gravitational force on the sun. The two forces are a third-law pair. The magnitudes of the two forces are the same. MISCONCEPTIONS Only the larger object exerts a force. The force of earth on sun is less than that of the sun on earth.

14. Virtues of the Multiple Choice Format • Familiar to the educational enterprise • School systems • Textbook companies • Psychometric community • Easy to score • Objectively and systematically scored • Distracters add complexity

15. Analytical Schemes • Categories of distractors • Near miss with pedagogical point (versus obscurity) • Thematic versus unrelated distracters • Relation to associated text • Local versus global • Explicit versus inference • Main rhetorical content versus details • Qualitative causal analysis • How does a change in A affect B? • Increase, decrease, no change, versus indeterminate

16. What happens to the pins when the key is turned to unlock the door? • A) they drop • B) they rise • C) they remain stationary • (Graesser & Olde, 2003)

17. Closing Comments • Rand Reading Study Group (Snow, 2002) emphasized comprehension • Text • Reader • Activities • Sociocultural context • Training effective strategies of reading comprehension requires analysis of world knowledge and deeper levels of processing (edited by McNamara, 2007)

18. ?