Jack Mostow, Joe Beck, Juliet Bey, Andrew Cuneo, June Sison, Brian Tobin Project LISTEN

1. An Embedded Experiment to Evaluate the Effectiveness of Vocabulary Previews in an Automated Reading Tutor Jack Mostow, Joe Beck, Juliet Bey, Andrew Cuneo, June Sison, Brian Tobin Project LISTEN Carnegie Mellon University www.cs.cmu.edu/~listen Funding: NSF IERI, Heinz Endowments The Reading Tutor helps children read stories aloud. Should it preview new vocabulary? Within-student experiment design Effects on vocabulary and comprehension

2. Before story: randomly assign 4 new vocabulary words in story to different previews • Control: • Do not pretest or explain word meaning. • Test-only: • Pretest but do not explain word meaning. • Pretest word meaning, then teach a synonym: • Relate to a simpler word appropriate to the story. • Pretest word meaning, then teach a definition: • Give a short explanation appropriate to the story. • How does Reading Tutor pretest word meaning?… 2

3. Before story: pretest word meaning 3

4. Before story: teach a synonym 4

5. Before story: teach a definition 5

6. During story: assist reading(only with decoding, not word meaning) 6

7. During story: test comprehension by automatically inserting cloze questions 7

8. … then continue assisted reading 8

9. After story: post-test vocabulary 9

10. … by matching the 4 words to their definitions 10

11. Effects on vocabulary: post-test results Percent correct on post-test • N = 5668 trials • 1417 story readings • 364 students in grades K-9 • 7 urban & suburban schools • “Untaught”: just encounter • control • test-only • “Taught”: preview meaning • synonym • definition 56% 54% 52% 50% 48% control test-only synonym definition (95% confidence intervals of means) 11

12. Exclude words with hasty pretest responses • Correctness varies with speed • N = 4251 pretest responses • 10.8% are faster than 2 seconds • 3.3% are slower than 15 seconds • “Hasty” responses are guesses • At chance level (25%) • Exclude those words in subsequent analyses Percent correct on pretest 80% 60% 40% 20% 0% 0 5 10 15 Response time on pretest (in seconds) 12

13. More taught than untaught words were learned • Split by whether already knew • “Knew” = correct on pretest • Excluding hasty pretests • Compare per-student % correct • N = 291 students • If didn’t already know word: • Taught: 17% over chance • Untaught: 10% over chance • Preview > encounter alone! • P = .031 • Who learned?… Percent correct on post-test 60% 50% 40% 30% N=277 N=242 N=291 N=250 No Yes 13

14. Who learned taught words better than untaught? • Disaggregate by WRMT • N = 291 students • No difference: • For WC up to GE 3 • Taught > untaught words: • WC above GE 3 Percent correct on post-test 80% 60% 40% 20% 0% • 2 3 4 5 6 • Word Comprehension GE pretest score 14

15. Did teaching target, distractor(s), or sentence word(s) help? N = 2671 cloze questions on vocabulary words Effects on comprehension: cloze results 15

16. Who did better on cloze if target word was taught? • N = 152 students’ cloze performance with target word taught or not • Exclude cloze question if a distractor or sentence word was taught Percent correct on post-test 70% 60% 50% 40% 30% 20% 10% 0% 1 2 3 4 5 Word Comprehension GE pretest score 16

17. Benefits to vocabulary and comprehension • Briefly explaining a new word before a story helped child • Match the vocabulary word to its definition after the story • Better than just encountering it in the story • For students with Word Comprehension above GE 3 • And apparently • Answer comprehension questions involving the word • Whether as cloze target, distractor, or context • For students with Word Comprehension above GE 1 (!) • Do previews really help comprehension before vocabulary? • Thank you! Questions? Suggestions? 17

18. Project LISTEN’s Reading Tutor (video) • John Rubin (2002). The Sounds of Speech (Show 3). On Reading Rockets (Public Television series commissioned by U.S. Department of Education). Washington, DC: WETA. www.readingrockets.org. 18

19. Project LISTEN’s Reading Tutor (video) 19

20. Problem: support vocabulary learning • Efficacy varies by age and ability (NRP 2000). • What are the specific vocabulary instruction needs of students at different grade and ability levels? • Computers can facilitate vocabulary learning (NRP 2000). • What is the optimal use of computers in vocabulary instruction? • Our question: when does previewing new words help? • Which previews? • Which outcomes? • Which students? • Which words? 20

21. Within-student experiment design • The Reading Tutor helps children read stories aloud. • Before story: pick 4 random new vocabulary words in story • Pretest word meaning: “Which word means (definition)?” • Teach 2 of the 4 words • During story: • Assist reading: scaffold decoding but not meaning • Assess comprehension: insert cloze questions • After story: • Post-test all 4 words: “Which word means (definition)?” 21

22. Before story • show example -- simulate screen shot sequence 22

23. Before story: ask familiarity with word 23

24. During story: assist and assess reading • Assist: scaffold decoding but not word meaning • Assess: automatically insert cloze questions • Before a sentence containing a vocabulary word • Delete the word and use it as the target • Use 3 other vocabulary words as distractors • (not necessarily the preview words) 24

25. Results • Effect of previews on post-test • Effect of previews on cloze performance • By age • By word frequency 25

26. Source df F Sig ANOVA for % correct on post-test Corrected Model 6 3.6 .002 Intercept 1 .48 .491 WRMT WC GE 1 7.3 .007 Age 1 .61 .437 Word Length 1 1.1 .290 Gender 1 .41 .521 Treatment 1 4.7 .031 Gender * Treatment 1 .47 .493 • Significant predictors: • Previous vocabulary • Preview 26

27. Previous work • Aist 2002: factoids helped rare, single-sense words (grades 2-3) on next-day test • Vocabulary learning literature • Reinking & Rickman 1990: computer vocabulary help for 6th graders • Beck: • McKeown guidelines for explaining words • Mostow, Tobin, Cuneo 2002: cloze validation see Aist thesis 27

28. Who learned taught words better than untaught? 28