1 / 27

What Was Learned from a Second Year of Implementation

What Was Learned from a Second Year of Implementation. William Corrin, Senior Research Associate MDRC. IES Research Conference Washington, DC June 8, 2009. Presentation Overview. Project Background Do teachers need more than a year to master new practices?

nayef
Download Presentation

What Was Learned from a Second Year of Implementation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. What Was Learned from a Second Year of Implementation William Corrin, Senior Research Associate MDRC IES Research Conference Washington, DC June 8, 2009

  2. Presentation Overview • Project Background • Do teachers need more than a year to master new practices? • Year 2 Implementation Findings (and Year 1 Comparison) • Impact findings • Exploratory findings • Two-Year Teachers and Replacement Teachers • Two-Year Teachers: Year 1 vs. Year 2 • Summary

  3. Project Background

  4. The ERO Study • The ERO study is an impact evaluation of two supplemental literacy programs targeted at 9th grade students with limited reading skills. • Key research question: • What are the impacts of the two interventions, together and separately, on ninth-grade students’ reading achievement and reading behaviors?

  5. The Interventions • Two literacy programs targeted at 9th grade students with limited literacy skills (2-5 years below grade level) • Reading Apprenticeship Academic Literacy (RAAL) • Xtreme Reading (XR) • The programs are supplemental reading classes • Designed as full-year courses that replace a 9th grade elective class (rather than a core academic class) • The two programs share common goals and instructional principles

  6. Delivery of the Interventions • Each high school implemented one of the two programs for ninth graders for two school years • 2005-06 (Cohort 1) and 2006-07 (Cohort 2) • One ERO teacher per high school • Experienced full-time ELA or social studies teacher was trained to teach the ERO reading course • Responsible for 4 sections of 10-15 students each • Training and technical assistance were provided to teachers by the developers • Summer training institutes, school-year off-site booster training sessions and on-site coaching

  7. Implementation Data Collection • To answer questions about how well the programs were put into place • Allows us to answer questions about how implementation compared across program years: Do teachers do better with more experience teaching these programs? That is, is implementation stronger in the second year compared to the first? • Allows us to investigate associations between implementation and impacts

  8. Measuring Implementation Fidelity • Two site visits in Year 2 (one per semester; only one site visit in Year 1) • Implementation fidelity was assessed on two dimensions: • Classroom learning environment • Comprehension instruction • Classroom observation ratings were obtained for 6 constructs common to both programs, and 7 program-specific constructs • Ratings are based on a 3-point scale

  9. Measuring Implementation Fidelity (cont.) • Composite ratings for each of the two dimensions were calculated, and then used to classify sites as: • “well aligned” (average rating > 2) • “moderately aligned” (average rating = 1.5 – 1.9) • “poorly aligned” (average rating < 1.5)

  10. Implementation Findings

  11. Implementation Findings • ERO teachers in Year 2 • Of the 34 ERO teachers in Year 2: • 25 had taught the entire first year of the study (13 RAAL, 12 XR) • 2 had taught part of the first year (both XR) • 7 were new to the programs (4 RAAL, 3 XR) • All Year 2 teachers taught the entire year

  12. Implementation Findings (cont.) • Implementation fidelity in Year 2 • In terms of classroom learning environment, 1 school was rated as poorly aligned with program models (vs. 4 schools in Year 1) • In terms of reading comprehension instruction, 1 school was rated as poorly aligned with program models (vs. 9 schools in Year 1) • 23 schools were rated as well aligned on both dimensions (vs. 16 schools in Year 1) • Similar pattern of implementation findings at RAAL and XR schools

  13. Implementation Findings (cont.) • Implementation fidelity rated higher in Year 2 • In terms of classroom learning environment, the average rating at the spring site visit was 2.5 in Year 2 compared to 2.2 in Year 1 • In terms of reading comprehension instruction, the average rating at the spring site visit was 2.3 in Year 2 compared to 1.9

  14. Learning EnvironmentBy Site Visit

  15. Comprehension Instructionby Site Visit

  16. Impact Findings

  17. Random Assignment and Analysis • Random assignment: • School-Level • 34 schools randomly assigned within 10 districts to RAAL or Xtreme Reading (17 RAAL sites and 17 Xtreme Reading sites) • Student-Level • In each high school, eligible students were randomly assigned to either: • Enroll in an ERO class (ERO group) or • Take one of the regular elective classes (non-ERO group) • Analysis: Impacts are estimated by comparing the outcomes of students in the ERO and non-ERO group

  18. Impact Findings: Reading Achievement • Reading achievement was measured using the GRADE, a standardized reading assessment. • Overall positive impact on reading comprehension • 0.09 SD for Cohort 1 • 0.08 SD for Cohort 2 • No impact on vocabulary • Same result for both cohorts

  19. Impact Findings: Reading Behaviors • Cohort 1 • Estimated impacts on three reading behaviors were not statistically significant (frequency of school-related reading, freq. of non-school-related reading, use of reading strategies) • Cohort 2 • Estimated impact on students’ use of the reading strategies taught by the programs is 0.09 SD and statistically significant

  20. Exploratory Analyses

  21. Implementation Fidelity by Teacher Experience and Year • Implementation ratings by year and by teacher experience:

  22. Implementation Fidelity by Teacher Experience and Year • Overall implementation fidelity: • Of the 25 sites with two-year teachers, 19 had higher overall implementation ratings in the second year compared to the first year • Of the 9 sites with replacement teachers, 8 sites had higher overall implementation ratings with the replacement teacher compared to the original teacher

  23. Composite Fidelity Scores by Site Visit

  24. Variation in Impacts • Impacts on Student Outcomes by Teacher Experience Teaching ERO: Year 2

  25. Variation in Impacts (cont.) • Impacts on Student Outcomes by Implementation Year: Schools w/ Two-Year Teachers (N=25)

  26. Supplemental literacy courses for 9th grade students can have a positive impact on students’ reading comprehension. These programs can be implemented with reasonable fidelity in a diverse array of high schools, and implementation fidelity can improve over time. On average, implementation fidelity was stronger in the second year. There is suggestive, but not convincing, evidence that teachers gained greater mastery of the programs having a second year to teach it. However, impacts on students reading comprehension remained the same. (Impacts on students’ use of reading strategies were greater.) Summary

  27. Further questions? • William Corrin • william.corrin@mdrc.org • (212) 340-8840 • Marie-Andree Somers • marie-andree.somers@mdrc.org • (212) 340-8825 • Paul Strasberg • paul.strasberg@ed.gov • (202) 219-3400

More Related