Assessing Comprehension with Verbal Protocols and LSA

1. Assessing Comprehension with Verbal Protocols and LSA Keith Millis Joe Magliano Northern Illinois University

2. HOW TO FOOL A CAT

3. Once upon a time there was a rich lord.

4. He liked to collect little wood carvings of animals.

5. He had many kinds, but he had no carved mouse.

6. He called two skilled carvers to him.

7. He said, “I want each of you to carve a mouse for me.” PLEASE REPORT YOUR THOUGHTS

8. Sample thoughts • The, uh, he had all kinds of carved animals except a mouse. So he sent for some men to carve him a mouse. I guess he was going to pick which one he liked best. • He didn’t have a mouse for his collection of wood carved animals. And he wanted a mouse. • He really wanted a mouse.

9. What is LSA? • Latent semantic analysis • Statistical representation of word knowledge • Computes a metric of semantic similarity between any two units of language • Cosine ranging (usually) between 0 and 1.0

10. Why use LSA to assess comprehension? • Comprehension tests typically use multiple-choice formats • The “off-line” limitation, “surfacy” information • LSA enables an assessment of on-line processing as expressed by verbal protocols • Verbal protocols • Express thoughts of the reader, which convey the use of reading strategies • LSA can categorize thoughts on reading strategies • R-SAT: Reading Strategy Assessment Tool

11. Benchmark 1 at sentence N Benchmark 3 at sentence N Benchmark 2 At sentence N .04 .18 .55 Verbal protocol at sentence N Summary of Approach • Compute similarity between verbal protocols and “semantic benchmarks” – words representing different strategies • The magnitude and pattern of cosines reveal strategies

12. Types of reading/ comprehension strategies • Specific strategies • Paraphrasing, bridging & associative inferences • General strategies • The extent that the current text representation is elaborated with information from the reader’s world knowledge and the mental representation of the text

13. General reading strategies • Sentence-focused: primarily paraphrases the current sentence • Local: some elaboration of current sentence from the prior sentence • Global: uses prior text and world knowledge to integrate sentence with the theme of a text

14. Example: the Heart • (text…) The blood cannot get rid of carbon dioxide through the lungs. It becomes purplish, and the baby's skin looks blue. • Sentence focused: “The blood turns to a purplish color and the baby's skin turns blue.” • Local: “When the carbon dioxide does not escape the body, the baby's skin looks blue.” • Global: “The infant is essentially choking from the inside. The skin turning blue might have something to do with not receiving enough oxygen connected to the heart problems.”

15. General strategy and Information Source

16. Current sentence World knowledge Prior causal sentences .04 .18 .75 Verbal protocol at sentence N Benchmarks and source of information • (from theory) Good readers attempt to integrate current sentence with prior causal text along with appropriate inferences • Three benchmarks for each sentence

17. Example Benchmarks • Sentence from “Thunderstorm”: • “Usually, within an hour the amount and size of precipitation becomes too much for the updraft to support.” • Current sentence benchmark: • hour size precipitation amount becomes updraft support • Prior sentence: • cloud release develops start storm • World knowledge: • fall hold down heavy

19. Do cosines predict narrative comprehension? (from Magliano & Millis, 2003) • Simple fairy tales • ½ stories, participants thought-aloud to selected sentences • “Write down thoughts that come to mind as you read the current sentence in the context of the passage.” • ½ stories, participants read silently • Answered comprehension questions & recalled the passages • Took the Nelson-Denny • Cosines, between protocol and • Current sentence benchmark • Prior causal sentences benchmark

20. Beta Weights from regression

21. Beta Weights from regression * p< .05; ** p < .01

22. Can feedback based on LSA change reading strategies? • Feedback to student who is typing in self-explanations to a text • If feedback based on LSA changes the quality of self-explanations, then • It supports the validity of LSA for identifying strategies • Useful in a tutoring environment

23. All thunderstorms have a similar life history. Thunderstorms start with the development of large cumulonimbus clouds. Type your self-explanation below: The cumulonimbus clouds are the first to appear in a thunderstorm Get feed back Tutor’s feedback: Rate feed back OK. How appropriate/fitting is the feedback? Get Next sentence 1 2 3 4 5 6 Not at all very

24. Method • Participants receive brief self-explanation training • Participants told to evaluate feedback from new tutor • Randomly placed into one of four conditions: no feedback, general, full, random

25. Feedback conditions • General: • Sentence focused = “Ok.” Local = “Very good.” Global = “Excellent.” • Full: • General + specific • Specific feedback associated with paraphrases, bridges, association • “Very good. I like it when you use your world knowledge.” • Random feedback • Like Full but content is randomly chosen.

26. Example • Sentence: It becomes purplish, and the baby's skin looks blue. • Self-explanation: A lack of oxygen to the heart generates a blue hue on a baby's skin. • General: Excellent. • Full: Excellent. It seems to me that you are bringing up prior text in your self-explanation. • Random: Very good. It appears that you are paraphrasing for the most part.

27. General reading strategy

28. Specific reading strategies

29. Student appropriate ratings

30. Summary • LSA can be used to assess text comprehension • Narrative and expository texts • Identify general and specific reading strategies • Does not have the “off-line” limitations associated with MC tests • What are important factors to consider with young children?