Micro-analysis of Fluency Gains in a Reading Tutor that Listens:

1. Micro-analysis of Fluency Gains in a Reading Tutor that Listens: • Jack Mostow and Joseph Beck • Project LISTEN (www.cs.cmu.edu/~listen) • Carnegie Mellon University • Society for the Scientific Study of Reading • 12th Annual Meeting, June, 2005 • Funding: National Science Foundation, Heinz Endowments 1

2. Wide vs. repeated guided oral reading • Guided oral reading builds fluency [NRP 00]. • Typically repeated oral reading • Is repeated reading better than wide reading? • Unclear! [Kuhn & Stahl JEP 03] • Past work analyzed reading rates on passages and word lists. • E.g. scramble word order to study context effects [Levy, …] • This talk: use finer-grained Reading Tutor data. 2

3. 2002-2003 database: 8 diverse schools 600 students (K-6) 26,000 sessions 600,000 sentences read 4 million words heard Project LISTEN’s Reading Tutor: Rich source of guided oral reading data 3

4. Reading speeds up with practice: example • Initial encounter of muttered: • I’ll have to mop up all this (5630 ms) muttered Dennis to himself but how • 5 weeks later (different word pair in different sentence): • Dennis (110 ms) muttered oh I forgot to ask him for the money • Word reading time = latency + production time  1/fluency • Beck et al. [TICL 04] used latency to assess proficiency (R2>.8). • What predicts word reading time? 4

5. Proficient readers are faster. 5

6. Long words are slower. 6

7. Reading time speeds up over successive encounters, but by less and less. 7

8. Linear model of word reading speedup • Predicted variable is speedup on the same word • Reduction in word reading time • From one (“practice”) encounter of a word • To the next (“test”) encounter • N = 243,172 opportunities for speedup • By 352 students (gr 1-6) with WRMT pretest scores • Include rereading as practice but not as test • Exclude encounters after the first 8 • Exclude 36 stop words (the, a, …) 8

9. Predictors of speedup • What is the student’s reading level? • How many letters long is the word? • How often has the student seen the word before? 9

10. Contextual predictors • Has the student seen the practice sentence before? • (Students have not seen test sentences before.) • Has the student seen the practice word pair before? • Has the student seen the test word pair before? 10

11. Results Higher readers speed up less: 3 ms less per grade level. Longer words speed up more: 2.4 ms more per letter. Speedup averages 18 ms per encounter (for the first seven). Speedup averages 30 ms after the first encounter, then 3 ms less after each subsequent encounter. 11

12. Context effects A new practice sentence helps 27 ms more than an old one. Wide reading beats rereading! A new practice word pair helps 21 ms more. A test word pair seen before is 13 ms faster. 12

13. Are these effects statistically reliable? • Standard errors are 0~3 ms. • But: Linear model ignores dependencies • Within student (other than proficiency) • Within word (other than length) • So: Compute p by counting N students, not encounters. • Restrict to students with 50+ encounters in each condition • Compare how many students do better in one condition vs. the other. • Use sign test to compute significance. • New practice sentences beat old ones for 100 of 135 students (p = .000). • No marked differences between groups • New practice word pairs beat old ones for 151 of 222 students (p = .000). 13

14. Does word speedup relate to test score gains? • In contrast, # encounters doesn’t predict gains (R2<.001). 14

15. What affects fluency growth? • Practice on specific words • Especially in new sentences • Practice on specific word pairs • Helps if tested, but new practice pairs are better • Practice in decoding new words • Not analyzed in this talk 15

16. Conclusions • Wide guided oral reading beats repeated reading! • Correlational, not experimental; what uncontrolled sample bias? • Excludes rereading but not recency effects • Relevant to human-guided oral reading? • Is rereading more motivating for poor readers? • Wide reading requires more text and more guidance! • Micro-analysis of tutor data finds subtle (ms) effects • Large, fine-grained, longitudinal sample • Speech recognition is imperfect • But unlikely to be biased wrt our variables • Thank you! Questions? • See papers & videos at www.cs.cmu.edu/~listen. 16

17. Tutoring: Dr. Joseph Beck, mining tutorial data Prof. Albert Corbett, cognitive tutors Becky Kennedy, linguist Joe Valeri, activities and interventions Listening: Dr. Evandro Gouvea, acoustic training John Helman, transcriber Dr. Mosur Ravishankar, speech recognizer Programmers: Andrew Cuneo, application Field staff: Kristin Bagwell Julie Sleasman Dr. Roy Taylor Grad students: Kai-min Chang, LTI Cecily Heiner, MCALL Ayorkor Mills-Tettey, RI Interns: Alisa Grishman Brooke Hensler James Leszczenski Rachel Minkoff Research partners: DePaul U. Toronto U. British Columbia Ghana Advisory board Kathryn Ayres, children’s stories Rollanda O’Connor, reading Charles Perfetti, reading … Schools www.cs.cmu.edu/~listen Thanks to fellow LISTENers & friends 17