Evidence of Impact

1. Evidence of Impact

2. 2 What is the Scope of Massachusetts Reading First? Public school subgrants 83 schools in 36 districts serving approximately 1,300 teachers and 14,000 students (K-3) 4 additional schools may be funded in January 2006 47 non-public schools received funding Additional dissemination through: State-funded John Silber Early Readers Program (formerly BayState Readers) involving 31 districts and 36 schools Project Focus (17 districts pairing RF and non-RF schools) Universal availability of Teacher Reading Academy materials and opportunities for training of trainers

3. 3 The Evaluation is Organized Around a Logic Model

4. 4 The Evaluation Has Two Main Components Process evaluation Examines implementation Provides on-going feedback to Mass DOE Documents the program model Outcomes evaluation Measures changes in teaching and learning Identifies promising practices Fulfills federal accountability requirements

5. 5 Selected Findings � Teacher Knowledge Analysis of the pre/post professional development assessment reveals that the TRAs had an immediate positive impact on participants� knowledge. Mean scores increased by 13 percentage points � a statistically significant increase That knowledge level increased slightly over the course of the year, indicating that receiving on-going PD and support were of some benefit to participating teachers. Matched post-test and follow-up records available for only 248 teachers (compared to over 1000 matched pre/post tests) Increase from 76% correct to 79% correct is small but statistically significant

6. 6 Student Outcomes Assessments Dynamic Indicators of Basic Early Literacy Oral Reading Fluency subtest (DIBELS ORF), University of Oregon Administered spring grade 1 and fall/spring grades 2-3 The Group Reading Assessment and Diagnostic Evaluation (GRADE) published by AGS Administered fall and spring for grades 1-3 MRFP students considered performing �at or above grade-level� if they meet or exceed stanine 5 (38th percentile) The Massachusetts Comprehensive Assessment System (MCAS) Reading test administered to third graders each spring

7. 7 Selected Findings - Summary of GRADE Data Cross-grade level summary of the percentage of students �at or above grade-level� For cohort 1 students this figure increased from 31% in the fall of 2003 to 61% in the spring of 2004 and 64% in the spring of 2005 � an increase of 107% over 2 years For cohort 2 students this figure increased from 30% in the fall of 2004 to 51% in the spring of 2005 � an increase of 70% in their first year For cohort 1: All grade levels and third grade subgroups showed the most marked improvement from baseline to year 1. All grade levels showed modest improvement from year 1 to year 2. All of the third grade subgroups also showed gains. However only those for low income, African American and Hispanic/Latino students were statistically significant.

8. 8 Selected Findings - Summary of MCAS Data From spring 2003 to spring 2004 there was no significant change in the proficiency rate among students in cohort 1 schools. This is consistent with the statewide trend. These schools showed a 77% increase in proficiency among LEP students compared to only a 26% increase statewide. In contrast to the statewide figures, there was improvement as evidenced by an 11% decrease in the percentage of students performing at the warning level. Notable subgroup performance includes: LEP (40% decrease vs. 10% statewide), Hispanic/Latino (20% decrease vs. 10% statewide) and low income (12% decrease). Baseline data for cohort 2 schools show that 34% of their students were proficient and 18% performed at the warning level on the spring 2004 MCAS.

9. 9 Selected Findings - RF assessments as predictors of MCAS Massachusetts selected DIBELS and GRADE because they were believed to be good predictors of third grade reading proficiency. 2003-2004 data for third grade RF students support the expectation that fall DIBELS Oral Reading Fluency (ORF) and fall GRADE assessments are correlated with scores on the MCAS reading test. These data provided the opportunity to predict fall scores that are suggestive of MCAS proficiency. It is likely that a fall 3rd grader: scoring 90 words per minute or better on DIBELS ORF will attain proficiency on the MCAS with GRADE performance at stanine 6 or better will attain proficiency on the MCAS

10. 10 Year 3 Evaluator�s Report We are currently working on our annual evaluation report, which will include a much more thorough analysis of the student assessment data Much of the student data analysis will follow guidelines recently disseminated by the Eastern Regional Reading First Technical Assistance Center (ERRFTAC), including: Examination of spring-to-spring performance changes within RF schools Comparing performance of students returning to RF schools to those who are new to RF schools (in-lieu of comparison schools) Looking at the percentage of students at grade level as well as those who are seriously behind Using Hierarchical Linear Modeling to determine statistical significance Presenting data tables with average scores, standard deviations and effect sizes The Evaluator�s Report should be publicly available in December

11. 11