Recovering from Reading Recovery: A long-term program evaluation

1. Recovering from Reading Recovery: A long-term program evaluation JoAnna Cogan-Ferchalk, Ed.S., NCSP Matthew Ferchalk, Ed.S., NCSP

2. Background & Philosophy • Developed in New Zealand, by Dr. Marie Clay in the 1970's • The goal of RR is to close the gap between at-risk readers and their average peers (Schwartz, 2005) • Incorporates Vygotsky’s theory of the Zone of Proximal Development, where the student is able to read at a higher rate with help from the teacher than would be possible alone (Clay & Cazden, 1990) • Whole language background-- phonics is not taught explicitly to children • It is taught “incidentally” if an opportunity arises (Schwartz, 2004) • Details of print and letter/sound relationships should be gained through writing, not reading instruction. This knowledge is considered limited when compared to comprehension of the text (Clay, 1993; Grossen et al., 1997)

3. How it works • Students are selected for RR by teacher ratings and by screening with the Observational Survey (the lowest students are not always enrolled) • Students receive daily 30 minute lessons during their first grade year • Instruction includes use of magnetic letters to help write words, and teaching to read for meaning • Most students graduate from RR after 12-20 weeks • ELCO’s RR teachers have adopted a more structured phonics element (CREC, 2004; see also Saginaw Public Schools, 1992)

4. School District Demographics • Approximately 2400 students • 16% special education placement rate • Rural, farming community in Pennsylvania • Reading Recovery has been in place for 11 years

5. Literature on RR • Several studies have found RR to be moderately effective (Marina & Gilman, 2003; Pinnell, Lyons, DeFord, Byrk, & Seltzer, 1994; Schwartz, 2005) • There are complaints that these studies are flawed (Center & Wheldall, 1992; Heibert, 1994; Hoff, 2002; Rasinski, 1995) • Unequal treatment time • Not using the lowest readers • Other studies have found that students do not attain the same reading proficiency as their peers (Huggins, 1999), which has been attributed to lack of explicit phonics instruction (Pressley, 1998) • When explicit instruction in phonics is added to RR researchers have found greater improvement than with traditional RR (Chapman, Tunmer, & Prochnow, 2001; Iverson & Tunmer, 1993) • Few studies have examined long term effects of RR. Those that do typically high rates of special education placement in later years (Ramaswami, 1994)

6. Research Questions • How do students who have been through the RR program score on the PSSAs and curriculum-based assessments (CBA) in reading? • If students demonstrate improvement in reading scores after RR, do these gains persist over time? • Do students who have been through RR make the same rate of reading progress as their peers who did not need supplemental instruction? • What is the prevalence of placement into programs such as Title I and Special Education after students have graduated or were discontinued from RR?

7. Method • Review existing data for students who were enrolled in RR and are currently in 5th, 6th, and 7th grades • 8 students per elementary building per year (n= 96 students) • 3rd and 5th grade PSSA reading scores • CBA reading scores from 1st to 5th grade • Special programs placement rate following RR • Review existing data from a randomly selected comparison group (n=96)

8. Reading Recovery Below Basic 24% Basic 38% Proficient 19% Advanced 19% Total Proficient 38% District wide data Below Basic 14% Basic 22% Proficient 30% Advanced 34% Total Proficient 64% 5th Grade PSSA data, Cohort 1

9. Reading Recovery 1st quartile 28% 2nd quartile 22% 3rd quartile 28% 4th quartile 22% Upper quartiles 50% District wide data 1st quartile 14% 2nd quartile 20% 3rd quartile 34.5% 4th quartile 31.5% Upper quartiles 66% 3rd Grade PSSA Data, Cohort 2

10. Reading Recovery Below Basic 27% Basic 14% Proficient 50% Advanced 9% Total Proficient 59% District wide data Below Basic 18% Basic 12% Proficient 47% Advanced 23% Total Proficient 70% 5th Grade PSSA Data, Cohort 2

11. PSSA scores from 3rd to 5th grade declined from 1298 to 1238 The difference was not significant at 0.05 Across time PSSA, Cohort 2

12. Reading Recovery 1st quartile 35% 2nd quartile 30% 3rd quartile 22% 4th quartile 13% Upper quartiles 35% District wide data 1st quartile 17% 2nd quartile 22% 3rd quartile 31% 4th quartile 30% Upper quartiles 61% 3rd Grade PSSA Data, Cohort 3

13. Short-term, 2nd grade, 75% of the comparison group were at mastery or instructional levels Compared to 9% of RR students Long-term, 5th grade 90% of the comparison group were at mastery or instructional 81% mastery Compared to 80% of RR students 20% mastery CBA Cohort 1

15. Short-term, 2nd grade 82% of the comparison group were at mastery or instructional levels Compared to 32% of RR students Long-term, 5th grade 95% of the comparison group were at mastery or instructional levels 42% mastery Compared to 82% of RR students 29% mastery CBA Cohort 2

17. Short-term, 2nd grade 43% of the comparison group were at mastery or instructional levels None of the RR students had reached an instructional level Long-term, 4th grade 90% of the comparison group were at mastery or instructional levels 71% mastery Compared to 67% of RR students 25% mastery CBA Cohort 3

19. Special Education Cohort 1 19% Cohort 2 22.7% Cohort 3 12% District wide rate is approximately 16% Title I Cohort 1 1 year- 100% 2 years- 33.3% 3 years- 14.3% Cohort 2 1 year- 100% 2 years- 31.8% 3 years- 27.3% Cohort 3 1 year- 100% 2 years- 40% 3 years- no data Special programs placement rates

20. How do students who have been through the RR program in the ELCO school district score on standardized tests of achievement and curriculum-based assessments (CBA) in reading? • On the PSSA exams, fewer students in RR scored in the proficient range as compared to district wide data • This held true across all cohorts • On CBA data, fewer students in RR scored at the instructional and mastery levels than did students not in RR • This held true across all cohorts

21. Do students who have been through RR make the same rate of reading progress as their peers who did not need supplemental instruction? • When examining the CBA data it appears that RR students started out well below their peers, and remained below them over the course of elementary school • RR’s goal is to close this gap in achievement • Over time more RR students scored in the instructional and mastery levels on the CBA measures; however, they did not approach the level of mastery attained by their peers

22. If students demonstrate improvement in reading scores after RR, do these gains persist over time? • In general, students did not demonstrate initial gains in reading (as demonstrated by CBA) • By 5th grade, students in RR were still behind their peers (as demonstrated by CBA & PSSA scores) • A jump was noted in fourth grade CBA scores; however this cannot be attributed to RR as it occurred three years after the intervention. This may also be the result of CBA scores no longer being kept for students in special education.

23. What is the prevalence of placement into programs such as Title I and Special Education after students have graduated or were discontinued from RR? • In cohorts 1 & 2, students were placed into special education at a higher rate than the district average (19 & 22.7% vs. 16%) • In cohort 3, the rate was lower (12% vs. 16%) • 100% of students in RR received Title I services in 1st grade • 40-60% of students in RR received Title I services in 2nd and 3rd grade

24. Conclusions & Implications • The goal of RR (bringing at-risk students to the level of their peers in a short period of time) does not appear to have been met • Given that RR is not designed for the lowest readers, it excludes students who are in need of help, and may lead to misleading reports of effectiveness • The practice of dropping students if they do not make adequate progress leaves some students without much needed intervention • School psychologists are in a unique position to recommend programs that have a broader research base and can provide more benefits to more students (i.e.- PHONICS!!!) • We can also take an active role as a researcher and/or program evaluator to keep our districts informed of beneficial programs

25. Limitations • Quality of data kept in the school • No “pre RR” reading levels of students available • Different data sets exists for each cohort • Too many confounding variables to accurately determine the effects of RR • 100% of students in RR received Title I for at least one year, and many more received years of support • Attrition • When students in RR were placed in special education, their CBA scores were not recorded

26. Recommendations for future research • More long-term research is needed to examine the effects or RR • Research should include initial reading levels of at-risk students

27. References • Capital Region Education Council. (2004). Who we are and what we do. Retrieved online September 26, from http://www.crec.org/rr/index.shtml#what. • Center, Y., & Wheldall, K. (1992). Evaluating the effectiveness of Reading Recovery: A critique. Educational Psychology, 12(3/4) • Clay, M. M. (1985). Observation survey of early literacy achievement. Auckland, New Zealand: Heinemann. • Clay, M. M. (1993). Reading Recovery: A guideline for teachers in training. Portsmouth, NH: Heinemann. • Clay, M. M. (1998). An observation survey of early literacy achievement. Auckland, New Zealand: Heinemann. • Clay, M. M. & Cazden, C. B. (1990). A Vygotskyan interpretation of Reading Recovery. In L. Moll (Ed.), Vygotsky and education: Instructional implications and applications of sociohistorical psychology (pp. 206-222). Cambridge, England: Cambridge University Press. • Chapman, J. W., Tunmer, W. E., & Prochnow, J. E. (2001). Does success in the Reading Recovery program depend on developing proficiency in phonological processing skills? A longitudinal study in a Whole Language instructional context. Scientific Studies of Reading, 5(2), 141-177.

28. Dyer, P. C. (1992). Reading Recovery: A cost-effectiveness and educational outcomes analysis. Spectrum: Journal of Research in Education, 10(1), 110-119. • Elbaum, B., Vaughn, S., Hughes, M. T. & Moody, S. W. (2000). How effective are one-to-one tutoring programs in reading for elementary students at risk for reading failure? A meta-analysis of the intervention research. Journal of Educational Psychology, 92(4), 605-619. • Evidence-based research on Reading Recovery. (2002). Report to House and Senate Education Committees. Retrieved July 1, 2002, from, http://www.nrrf.org/rrletter_5-02.pdf. • Glynn, T., & Crooks, T. (1992). Reading recovery in context: Implementation and outcome. Educational Psychology,12(3/4), 249-262. • Grossen, B., Coulter, G., & Ruggles, B. (1997). Reading Recovery: An evaluation of benefits and costs. Retrieved on July 1, 2002, from, http://darkwing.uoregon.edu/~bgrossen/rr.htm. • Hoff, D. J. (2002). Researchers urge officials to reject Reading Recovery. Education Week, 21(39), 1-3. • Huggins, R. (1999). Longitudinal study of the Reading Recovery program: 1994-1998. Urban Education. (ERIC Document Reproduction Service No. 430 067). • Iversen, S. & Tunmer, W. E. (1993). Phonological processing skills and the Reading Recovery program. Journal of Educational Psychology, 85, 112-126. • Marina, B., & Gilman, D. A. (2003). Is Reading Recovery worth the cost? (ERIC Document Reproduction Service No. ED 473 957).

29. Moore, P., Lysy, D. C., Bamford, R., & Cook, L. A. (1999). A look at Reading Recovery programs in Maine schools: A supplement to the State of Maine Reading Recovery report and evaluation, 1997-1998. Maine University: Center for Early Literacy. (ERIC Document Reproduction Service No. ED 463 533). • Munoz, M.A. (1999) Reading Recovery and the at-risk first grade students. (ERIC Document Reproduction Service No. ED 471 186). • Reading Recovery Executive Summary, 1984 to 1998. (1999). Columbus, OH: Reading Recovery Council of North America. (ERIC Document Reproduction Service No. ED 434 318). • Report of the National Reading Panel: Teaching Children to Read. (April 2000). National Institute of Child Health and Human Development. NIH Pub. No. 00-4769. Retrieved July 1, 2002, from, http://www.nichd.nih.gov/publications/nrp//findings.htm. • Pinnell, G. S., Lyons, C. A., DeFord, D. E., Bryk, A. S., & Seltzer, M. (1994). Comparing instructional models for the literacy education of high-risk first graders. Reading Research Quarterly, 29, 9-39. • Pressley, M. (1998). Reading instruction that works: The case for balanced teaching. New York: Guilford. • Ramaswami, S. (1994). The differential impact of Reading Recovery on achievement of first graders in the Newark School District. Newark, NJ: Newark Board of Education, Office of Planning, Evaluation and Testing. (ERIC Document Reproduction Service No. ED 374 180). • Rasinski, T. V. (1995). Commentary on the effects of Reading Recovery: A response to Pinnell, Lyons, DeFord, Bryk, and Seltzer. Reading Research Quarterly, 30, 264-270.

30. Saginaw Public Schools, Department of Evaluation Services. (1992). Compensatory education product evaluation: Reading Recovery Program 1991-92. Saginaw: Author. (ERIC Document Reproduction Service No. ED 350 587). • Shanahan, T., & Barr, R. (1995). Reading Recovery: An independent evaluation of the effects of an early instructional intervention for at-risk learners. Reading Research Quarterly, 30, 958-995. • Schawrtz, R. M. (2005). Literacy learning of at-risk first-grade students in the Reading Recovery early intervention. Journal of Educational Psychology, 97(2), 257-267. • Tunmer, W. E., & Chapman, J. W., (2001). The Reading Recovery approach to preventative early intervention: As good as it gets? Palmerston North, New Zealand: Massey University.