Evaluating the NYC Core Knowledge Early Literacy Pilot: Year 3 Report

1. Evaluating the NYC Core Knowledge Early Literacy Pilot: Year 3 Report Research and Policy Support Group February 2012 1

2. Executive Summary – Quantitative Analysis • Core Knowledge Reading (CKR) students had significantly greater gains in Year 3 than comparison school students on nearly all measures. • Overall Achievement • Spring scores for 2nd grade CKR students were greater than that of comparison students on all tests.* • High vs. Low Performers • The CKR intervention had an impact for all students, regardless of their incoming fall reading scores, but the effects were strongest for students with lower incoming scores. • CKR Program Exposure • Students new to the CKR program this year (who also had lower incoming fall scores) displayed greater increases on the WJ Brief Reading than those who had been in the program longer. CKR students who had been in the program for multiple years displayed the highest spring scores. • Post- Pilot CKR Participation • CKR Kindergarten and 1st grade classes implementing the CKR program this year scored significantly higher than comparison schools in the spring on TerraNova Reading. *All differences were statistically significant except TerraNova Reading

3. Executive Summary – Surveys & Site Visits • Surveys • The teacher and administrator surveys indicate high levels of satisfaction with the CKR Program and a preference for the program over other 2nd grade reading curricula. • Most teachers found that students were very engaged with the content of the curriculum and that it was successful at sparking enthusiasm for reading. • Site Visits • Fidelity to the program was quite high. The main difficulty noted was in allotting an uninterrupted 60 minutes for program implementation. • Most of the observed classrooms fully utilized the materials provided and students were largely attentive during lessons, participated in discussions, and were curious about the content of the materials. • In surveys and interviews, teachers noted difficulty with differentiation of instruction using the provided materials for students with special learning needs, such as English Language Learners and Students with Disabilities.

4. Methodology: A multi-method, longitudinal research design Focus of the Evaluation Hypothesis:Students taught with the Core Knowledge Reading (CKR) Program will gain reading competencies and content knowledge (science and social studies skills) at a faster rate than their peers. PROGRAM YEAR 3: 2ND GRADE Assessments of 2nd Grade Students (at 10 CKR schools & 10 comparison schools)* • Pre- and post-test of literacy skills • Tests of science and social studies skills at the end of the year • Additional test of content-specific literacy skills (the Core Knowledge Assessment for CKR students only) at the end of the year Teacher and Administrator Surveys (at 10 CKR schools): • Assesses satisfaction with and impact of CKR Case studies (at 4 CKR schools): • Classroom observations, administrator & teacher interviews • ADDITIONAL GRADE LEVELS • Assessments of Kindergarten and 1st Grade Students** (at 9 CKR schools & 9 comparison schools) • Tests of literacy, science and social studies skills at end of the thirdyear • In Year 1 and 2, Reading First schools were used as a subgroup in the evaluation. However, these schools were no longer Reading First in Year 3. • ** Kindergartners and 1st graders in CKR pilot schools no longer received full CKR program supports and two schools did not test additional students.

5. Demographics: While the comparison schools for the evaluation were initially picked for their similarities to the pilot schools, analyses control for demographics to account for any demographic differences that have emerged over the past two years. Percent of 2nd Grade Students CKR Students (N = 700) Comparison Students (N = 357)* * N = the number of students for whom both fall and spring data were available for any WJ test. A random sample of half of the students in the comparison schools were selected for testing. All students in CKR schools were tested. * * ** Analyses in all slides exclude students with disabilities. Data from 08-09 was used to select comparison schools (data presented here are from the 2010-11 school year). * Difference between CKR and Control groups statistically significant. ** ELA percents based on total number of students who took the exams in the schools in this evaluation.

6. Evaluation of Achievement Gains

7. Quantitative Results Overview:CKR students displayed higher scores and gains on nearly all tests. Note: All analyses control for student demographic characteristics. *Significant only at one-tailed level. Two-tailed p=.067 7

8. Quantitative Results Overview: CKR students’ literacy gains were more than double the gains of students at demographically similar comparison schools. Average Fall-Spring Gain in Scale Score Points Woodcock-Johnson (Brief Reading Test) CKR Students Displayed A Significantly Greater Change From Fall to Spring Than Comparison School Students (N = 683) (N = 352)

9. Impact of time in program:CKR had the largest impact among students who were new to the program (who had lower starting scores) However, the highest ultimate (spring) scores were found among those who had been in the program the longest. Average Fall and Spring Scale Score Points By Years in CKR Program in Year 3 Woodcock-Johnson (Brief Reading) +0.8 pts +2.6 pts + 3.2 pts

10. Impact of time in program:The longer second grade students had been in the CKR program, the higher their TerraNova Reading scores were. Average Spring Scale Score Points By Years in CKRProgram TerraNova (Reading Test) 50th percentile Nationwide (606) 10th percentile Nationwide (553) Scores are not adjusted for demographics.

11. Effects of CKR Following Participation in the Pilot: Kindergarten students in schools implementing CKR showed higher TerraNova Reading spring scores than students in comparison schools in both the year they participated in the pilot and the current year. Spring Scores TerraNovaReading Test CKR Kindergartners in 2008-2009 Kindergarteners in Year 1 CKR Kindergartners in 2010-2011 Kindergarteners in Year 3 Scores are not adjusted for demographics.

12. Effects of CKR Following Participation in the Pilot: First grade students in schools implementing CKR showed higher TerraNova Reading spring scores than students in comparison schools in both the year they participated in the pilot and the current year. Spring Scores TerraNovaReading Test CKR First Graders in 2009-2010 First Graders in Year 2 CKR First Graders in 2010-2011 First Graders in Year 3 Scores are not adjusted for demographics.

13. Administrator and Teacher Surveys 13 13

14. Administrators reported satisfaction with the CKRprogram. Will your 2nd grade classrooms be using the CK Reading Curriculum next year? (n = 9) Would you recommend the CK Reading curriculum to other school administrators you know? (n = 9) Administrators’ overall satisfaction with CK Reading (n = 9) * No 2 1 No Not Sure Neutral 1 7 Very Satisfied 7 Somewhat Satisfied Yes Yes Each school is unique. I have a significant amount of ELL’s and SWD's other schools may not. As I have said before, children gain in Phonics and Phonemic Awareness. They need more basic vocabulary development and comprehension skills and making a writing connection. We feel that it is not preparing them for the state standards for reading and Writing. The writing connection is missing. We don't feel there is enough aligned to the new CCSS and although the children are able to get the Phonics, and Phonemic Awareness down; the comprehension and independent thinking skills are lacking… * Administrators had five options for answering this question: Very Satisfied; Somewhat Satisfied; Neutral; Somewhat Dissatisfied; Very Dissatisfied . 14 14 14

15. Most teachers were satisfied with CKR and reported that CKR was better than other reading programs. • Teachers’ Views: • “The students enjoyed learning new concepts which sparked their reading and the need to know. Although their skills were limited they were always ready for a new concept or a new presentation of skills. Because this was a new program for me as a teacher it sparked my interests as well.” • “Over more than twenty years of teaching I have experienced a number of reading programs. Phonics, spelling, vocabulary and writing and comprehension skills were integrated throughout and the sequence made sense. Enrichment suggestions and remediation suggestions were presented throughout the materials.” Percent of Respondents 77.0% 72.2% Very Satisfied Much Better Somewhat Satisfied Somewhat Better Teachers’ overall opinion of CK Reading compared with other reading programs (n =13)b Teachers’ overall satisfaction with CK Reading (n = 18)a Note: Calculations were based only on responses that selected an answer other than N/A, which resulted in a lower overall N count. Other Answers: Question a Neutral=2 (11.1%); Somewhat dissatisfied=2 (11.1%); Very Dissatisfied=1 (5.6%); Question b About the Same=0; Somewhat Worse=2 (15.4%); Much Worse=1 (7.7%). In Year 2, Overall satisfaction was 82.7% very/somewhat satisfied, while overall opinion of CK relative to other programs was 50% much/somewhat better 15 15

16. Teachers indicated that both CKR strands had clear goals, were engaging, and that they had sufficient time to complete the lessons. In Year 2, teachers had rated the Skills Strands higher on these measures.* Percent of Respondents Strongly Agree Listening and Learning Strand Skills Strand Somewhat Agree Goals of lessons are clear (n = 16) Neutral: N=1 (6.3%) Disagree: N=0 (0.0%) Strongly Disagree: N=1 (6.3%) I have enough time to complete daily lessons (n = 16) Neutral: N=1 (6.3%) Disagree: N=2 (12.5%) Strong y Disagree: N=1 (6.3%) Students find activities engaging (n = 16) Neutral: N=1 (6.3%) Disagree: N=2 (12.5%) Strongly Disagree: N=1 (6.3%) Goals of lessons are clear (n = 16) Neutral: N=2 (12.5%) Disagree: N=1 (6.3%) Strong y Disagree: N=2 (12.5%) Students find activities engaging (n = 16) Neutral: N=2 (12.5%) Disagree: N=1 (6.3%) Strongly Disagree: N=2 (12.5%) I have enough time to complete daily lessons (n = 16) Neutral: N=1 (6.3%) Disagree: N=0 (0.0%) Strong y Disagree: N=2 (12.5%) * In Year 2, those that rated Strongly Agree or Somewhat Agree: Goals of lessons are clear =96.6%;Students find activities engaging=78.6%; I have enough time to complete daily lessons= 72.4%. Note: Calculations were based only on responses that selected an answer other than N/A, which resulted in a lower overall N count. 16 16

17. Compared to other programs they taught, most teachers found CKR much or somewhat better at teaching content/background knowledge. Teachers’ Views: “I like the content reading, the students really enjoyed learning about different areas and different people. I always got the question-what are we going to learn next? Note: Calculations were based only on responses that selected an answer other than N/A, which resulted in a lower overall N count. 17

18. Teachers believed the CKR program was better than other programs they have previously used at accommodating different learning needs and teaching vocabulary. Teachers’ Views: “The writing component was weak and did not really teach students how to write a multi paragraph writing. Most of your writing pieces were quick and easy. They didn't really show the writing process or teach students how to write extended responses.” Note: Calculations were based only on responses that selected an answer other than N/A, which resulted in a lower overall N count. 18

19. Site Visits and Interviews 19 19

20. Measuring fidelity to the CKR curriculum • Examining implementation fidelity allows us to better determine: • 1) whether achievement gains can be attributed to the CKRprogram • 2) which components of the CKR program teachers are struggling to implement and those they are implementing successfully • We took several different approaches to measuring fidelity in the CKR Pilot schools: • Site visits with classroom observations • Site visits were conducted at 4 randomly selected Pilot schools • 11 classrooms were observed • Interviews with administrators and teachers • Interviews were conducted with 16 teachers and 3 administrators • Additional teacher and administrator survey questions 20

21. During the site visits, we used a classroom observation protocol developed from the Core Knowledge Reading Pilot Observation Form These components were taken from the Core Knowledge Reading Pilot Observation Form These components were added as additional measures of student engagement 21

22. Nearly all classrooms demonstrated high fidelity to the CKR program. In particular, we observed high levels of student engagement in lessons. Number of Classrooms (N=11) Observed During Visit Not Observed During Visit Students are attentive to their teacher during instruction Full 60 minutes is allotted for both the Listening and Learning and Skills Strands* Teachers have Skills Strand materials posted and used Word Walls effectively Small group and/or individual practice time Students respond appropriately to their teacher’s directions, questions, and assignments * 60 minutes for the Strands typically not allotted uninterrupted. 22

23. Overall, teachers and administrators reported high fidelity to the program. One area where they reported struggling to implement the program was in finding time for daily small group instruction. 23

24. Comparison of 3rd grade 2012 NY State ELA test scores among: Students who participated in the CKR Pilot as second graders in the 2010-2011 vs. students who were assessed at comparison schools in that year. Students who had previously participated in the CKR Pilot as second graders (one year in the pilot) versus those who had previously participated in the CKR Pilot for multiple years. Exploration of associations between 3rd grade scores on the 2012 NYSELA test and previous performance on tests of reading comprehension (WJIII, TerraNova) while participating in the pilot as second graders. Next Steps: Year 4 NYCDOE/RPSG’s Year 4 Evaluation Track student achievement through 3rd grade on the NYS English Language Arts (ELA) test 24