Evaluating the NYC Core Knowledge Early Literacy Pilot: Year 2 Report

1. Evaluating the NYC Core Knowledge Early Literacy Pilot: Year 2 Report Research and Policy Support Group CONFIDENTIAL – PLEASE DO NOT DISTRIBUTE 1

2. Executive Summary – Achievement Gains • By nearly all measures, Core Knowledge Reading (CKR) students made significantly greater gains in Year 2 than their peers • Overall Achievement • When compared to peers (both peers at demographically similar schools and peers at Reading First schools), 1st graders taught with the CKR program made more progress in reading and social studies. • Spring science scores were significantly greater for students in pilot schools than comparisons schools, but not significantly different from students in Reading First comparison schools. • High vs. Low Performers • High and low performing CK students made larger gains than comparison students • Year 2 vs. Year 1 of the Program • While significant gains were found in almost all domains, the magnitude of these gains was smaller in Year 2 than in Year 1. • Both students with only one year of CKR instruction and those with two years of CKR instruction made greater gains than their peers at comparison schools.

3. Executive Summary – Surveys & Case Studies • Surveys and case studies indicate overall high levels of administrator and teacher satisfaction with the CKR Program and high levels of fidelity in program implementation. • While survey respondents had favorable reactions to CKR, they indicated that they felt CKR rated similarly to other programs they had used. • Surveys and case studies revealed that teachers are struggling with how to differentiate instruction with CKR, in particular for special populations. • Case study schools’ fidelity to the program aligned with their achievement gains: schools with low fidelity demonstrated less gains

4. Methodology: A multi-method, longitudinal research design Focus of the Evaluation Hypothesis:First Graders taught with the Core Knowledge Reading (CKR) Program will gain reading competencies and content knowledge (science and social studies skills) at a faster rate than their peers. • YEAR 2 • Assessments (at 10 CKR schools & 10 comparison schools, 3* of which are Reading First schools): • Pre- and post-test of literacy skills • Additional tests of literacy skills at end of each year • Tests of science and social studies skills at end of each year • Teacher and Administrator Surveys (at 10 CKR schools): • Assesses satisfaction with and impact of CKR • Case studies (at 5 CKR schools): • Classroom observations, administrator & teacher interviews * Last year there were 4 Reading First comparison schools. This year there were 3 Reading First comparison schools: one comparison school used Reading First in Year 1 but not in Year 2.

5. Overall, CKR and comparison schools had similar demographic profiles. However, the differences between groups, especially % of Black/ Hispanic students, was significantly larger than last year. As a result, all analyses control for student demographics. Percent of Students (Number of Students for School Size) CKR Students (N = 781) Comparison Students (N = 343)* • N = the number of students for whom both fall and spring data were available. • A random sample of half of the students in the comparison schools were selected for testing. All students in CKR schools were tested. Note: Data from 08-09 was used to select comparison schools (data presented here are from the 2009-10 school year).

6. Evaluation of • Achievement Gains

7. CKR students displayed greater gains & higher Spring scores in nearly all literacy and content knowledge domains relative to comparison students Note: All analyses control for student demographic characteristics. 7

8. CKR students displayed nearly 2x greater literacy gains relative to students at demographically similar comparison schools The magnitude of the group difference was smaller than last year (in Year 1, CKR students’ gains were 5X greater than their peers). Average Fall-Spring Gain in Scale Score Points Woodcock-Johnson (Brief Reading Test) CKR Students Significantly Higher p < .0001 (N = 758) (N = 342)

9. CKR students displayed 6x greater literacy gains relative to students at Reading First comparison schools The magnitude of the group difference was smaller than last year (in Year 1, CKR students’ gained 14.3 pts while Reading First schools declined 4.1 pts). Average Fall-Spring Gain in Scale Score Points Woodcock-Johnson (Brief Reading Test) CKR Students Significantly Higher p < .0001 (N = 233)* (N = 100)* * Data presented represent the three comparison schools that are Reading First schools and the three CKR schools who were statistically comparable matches. 9

10. Gains at Each School: All of the schools making statistically significant gains were CKR pilot schools Significantly Positive Gain Non-significant Gain Significantly Negative Gain 10

11. Gains at Each School in Year 1 and Year 2: Some CKR schools that made significant gains in Year 1 did not make significant gains in Year 2 and vice versa Significantly Positive Gain Non-significant Gain Significantly Negative Gain 11

12. At all achievement levels, CKR students demonstrated greater literacy gains than students at comparison and Reading First schools While the lowest performing students made greater gains than students at comparison schools, this difference was not significant. All other differences were statistically significant. Average Fall-Spring Gain in Scale Score Points Woodcock-Johnson (Brief Reading Test w/ Demographic Controls) CKR & All Comparison Schools CKR & Reading First Schools * * * * * * * CKR Schools Comparison Schools Reading First Schools *Significantly different at p < .05 12

13. CKR students made significant gains regardless of whether it was their first or second year in the program CKR students with two years of the program made larger gains than their peers. Similarly, CKR students with one year of the program made larger gains than their peers. CKR Years 1 and 2 (n = 527) Comparison Years 1 and 2 (n = 154) CKR New in Year 2 (n = 237) Comparison New in Year 2 (n = 189)

14. Administrator and Teacher Surveys 14 14

15. Administrators reported satisfaction with the CKR program Would you recommend the CK Reading curriculum to other administrators you know? (n = 7) Will your 1st grade classrooms be using the CK Reading program next year? (n = 7) Administrators’ overall satisfaction with CK Reading (n = 7*) Very Not No Dissatisfied Sure 1 1 1 Very Satisfied Somewhat 3 Satisfied Yes Yes 3 6 6 “Although the CK program has a lot of good components, its lack of a writing curriculum and differentiated instruction is a major concern for a program being used for 1st grade.” * Three administrators did not complete the survey. 15 15 15

16. Administrators reported that 1st grade CKR teachers’ practice was about the same as it was the previous year (prior to CKR) Collaborating with other first grade teachers (n = 7) Collaborating with teachers in other grades (n = 7) Discussing/ sharing ideas on teaching strategies w/ other teachers (n = 7) Using assessment data to drive instruction (n = 7) Much less than last year Somewhat more than last year Somewhat more than last year Much more than last year Much more than last year Much more than last year N/A 1 1 Much more than last year Somewhat more than last year 1 1 2 3 2 About the same as last year 3 1 About the same as last year About the same as last year About the same as last year 4 3 3 2 1 Somewhat less than last year Administrators’ Views: “I like CKR. I see improvement from last year…However…it’s the balance that I would like to see CK achieve. More of the sight words that children in early childhood should acquire…I would like to see complete alignment between Pre-K, K, and 1st, and now we’re going to 2.” 16 16 16

17. Most teachers were satisfied with the CKR overall but only half reported that CKR was better than other reading program • Teachers’ Views: • “I think the skills training really works…the Skills Strand was kind of similar [to last year] because we did Foundations [which also had] lots of cards and decoding.” • “I used the Guided reading program. I liked the small groups and being able to move students faster according to their ability with the Guided Reading but I love the phonics and comprehension part of the Skills Strand.” Percent of Respondents 82.7% Very Satisfied Somewhat Satisfied 50.0% Much Better Somewhat Better Teachers’ overall opinion of CK Reading compared with other reading programs (n = 28)b Teachers’ overall satisfaction with CK Reading (n = 29)a Number of teachers selecting the “neutral” response: question a = 1 (3.2%); question b = 6 (19.4%). 17 17

18. Strongly Agree Somewhat Agree Teachers thought both CKR strands had clear goals, were engaging, and that they had sufficient time to complete the lessons. In Year 1, teachers rated the Skills Strand more positively than the L&L Strand and had concerns about time. Percent of Respondents Skills Strand Listening and Learning Strand 96.6% 93.1% 78.6% 72.4% 72.4% 68.9% Goals of lessons are clear (n = 29) d I have enough time to complete daily lesson (n = 29) f Students find activities engaging (n = 29) e Goals of lessons are clear (n = 29) a Students find activities engaging (n = 29) b I have enough time to complete daily lesson (n = 29) c Number of teachers selecting the “neutral” response: question a n = 0; question b n = 2 (7.1%); question cn = 2 (6.9%); question d n = 0; question e n = 6 (20.7%); question f n = 4 (13.8%). 18 18

19. Much Better Somewhat Better Teachers compare CKR with other programs they have taught: As in Year 1, decoding skills are a strength of CKR while differentiation of instruction is rated less highly. Percent of Respondents 68.9% 62.0% 55.2% 51.7% 44.8% 31.0% Sequence of instruction (n = 29) b Compehen-siveness of program (n = 29) d Ability to engage students and spark enthusiasm for reading (n = 29) e Accommodations for different learning needs (n = 29) f Teaching decoding skills (n = 29) a Teaching content/ background knowledge (n = 29) c Number of teachers selecting the “about the same” response: question a n = 4(13.8%); question b n = 8 (27.6%); question cn = 3 (13.8%); question d n = 9 (31.0%); question e n = 11 (37.9%); question f n = 9 (31.0%).

20. Site Visits and Interviews 20 20

21. Measuring fidelity to the CKR curriculum • Examining implementation fidelity allows us to better determine: • 1) whether achievement gains can be attributable to the CKR program and • 2) which components of the CKR program teachers are struggling to implement and those they are implementing successfully • We took several different approaches to measuring fidelity in the CKR Pilot schools: • Site visits with classroom observations • Site visits were conducted at 5 randomly selected Pilot schools • 15 classrooms were observed • Interviews with administrators and teachers • Interviews were conducted with 13 teachers and 7 administrators • Additional teacher and administrator survey questions

22. During the site visits, we used a classroom observation protocol developed from the Core Knowledge Reading Pilot Observation Form These components were taken from the Core Knowledge Reading Pilot Observation Form These components were added as additional measures of student engagement

23. Most classrooms demonstrated high fidelity to the CKR program. In particular, we observed high levels of student engagement in lessons. We observed fewer classrooms engaging in the full 60 minutes of recommended CKR instruction. Number of Classrooms Out of 15 Observed During Visit Not Observed During Visit Full 60 minutes is allotted for both the Listening and Learning and Skills Strands Small group and/or individual practice time Students are attentive to their teacher during instruction Students respond appropriately to their teacher’s directions, questions, and assignments Teachers have Skills Strand materials posted and used Word Walls effectively 23

24. Overall, teachers and administrators reported that high fidelity to the program. One area where they reported struggling to implement the program was in finding time for daily small group instruction. 24

25. Example of a school with high fidelity

26. Example of a school with high fidelity Administrators’ Views

27. Example of a school with lower fidelity

28. Example of a school with lower fidelity

29. There was an alignment between achievement gains and fidelity to the curriculum Three Site Visit Schools Did Not Significantly Gain or Declined in Performance from Fall to Spring: • One of these schools was the low fidelity school described previously. • One of these schools had a teacher new to teaching first grade who had been in and out of the classroom, and the relationship between the principal and this teacher was strained. • All schools spent less than the required 60 minutes on the Listening and Learning Strand and the Skills Strand. • One of these schools also had a very strong focus on writing, which would occasionally cut into their other strand time. Significantly Positive Gain Non-significant Gain Significantly Negative Gain

30. Core Components of Year 3 Evaluation CKR assessments and longitudinal examination of relationship between literacy and content knowledge Survey/ site visits at comparison schools Assess administrators’ and teachers’ thoughts/ plans re: 3rd grade Assess achievement gains of Kindergarten and 1st graders - CKR grades that are not receiving as much intensive support (Year 4) Track student achievement through 3rd grade (NYS tests) Next Steps: Year 3 Recommendations for Year 3 Implementation • Provide targeted support to teachers on: • Differentiating instruction for students requiring different accommodations • Finding time to complete lessons and have them allot a full 60 minutes to lessons (or adjust appropriately if not allotting full 60) 30