Rochester Arts Impact Study

1. Rochester Arts Impact Study Deborah Harloff, RCSD Executive Director of Visual and Performing Arts Andrew MacGowan , III Project Administrator and Arts Impact Study Evaluator

2. The research on arts integration is based on the hypotheses that there is a positive impact between integrated arts instruction and increased student achievement .

3. What is arts integration? • Arts Integration is the process of combining content of the arts with other classroom content such as English Language Arts, Science, Social Studies, or mathematics. This is done by making clear content connections between the various curricular areas. • Visual art • Dance • Theatre • Music

4. The Arts as an intervention • Little empirical data • Student motivation

5. 1st Time 2006 • 2006-2009 • Over 200 applicants • Ranked 13th nationally of 25 awardees • Perfect score on evaluation design

6. “True Experimental Design”What and Why • “Medical model,” “True Experimental,” “Randomized Trial” all mean the same thing • Until NCLB, very few (< dozen) randomized trials in education • The most rigorous form of evaluation • “quasi-experimental” is the next best form of evaluation

7. The RCSD Design • 9 randomly selected elementary schools (out of 39) • 180 teachers • 4,000 students • 7 grade levels (K-6) • Remaining 29 schools served as the control group • Approximately 30,000 students in all, over the course of four years (baseline year + three years of treatment) • Statistically “robust”: results can be believed

8. Measures • Sixteen major New York and national standardized tests were used, in English Language Arts, Mathematics, Science and Social Studies

9. Components of the grant • Ten session teaching artist residency • Additional arts experience • Supplies to implement the residency • Ten hours of PD for teachers

10. Year One: Selection • Computer-generated randomization of schools • Introduce project to building administrators • Interview and hire teaching artists • Create planning document • Grade level planning sessions

11. 4 Schools Opt Out • Based on the following rational: • the school based planning team (SBPT) was concerned about the ten hours PD requirement • the teachers at one of the schools were concerned that they would be writing the curriculum for the teaching artist • another school cited the concern about accepting a new program when the school had a new principal.

12. Years Two & Three • Year two one school decided to opt out of the program • During year three, some of the faculty at one school refused to meet thereby essentially opting out of the program. • Due to scheduling conflicts and organizational skills one teaching artist was not retained. • A different teaching artist moved out of state and could not complete the project.

13. Year One: Implementation • Grade level planning meetings were scheduled • Schedules were arranged to accommodate teaching artist availability as well as classroom continuity. • Materials and supplies needed for the project were discussed.

14. Years Two & Three: Implementation • Due to inconsistencies in planning meetings, a planning document was developed to create uniform lesson and unit plans • Schedule protocol remained the same for year two.

15. Introduction of the 4 Art Forms • The art form that was integrated at the school was randomly selected . • Two schools were assigned as music • two were assigned as dance • Three schools were assigned as theatre • Three were assigned as visual art

16. Skills and Content Taught • Visual Art • Music • Theatre • Dance

17. Arts Impact Study PD • Professional development (PD) to reinforce understanding of integration techniques and strategies in music, dance, theater and visual arts. • Professional reading • 96% approval rating • Hands on • Models strategies • Provides materials

18. PD for Teaching Artists • Program objectives, program implementation and classroom residencies. • Operational practices were taught to the teaching artists • Techniques to reinforce ELA, Math, Science and Social Studies with the teaching artists’ area of expertise. • Collegial sharing - discussed best practices and successful strategies for co-teaching and co-planning lessons with classroom teachers.

19. Main Conclusions • Using both national and NY tests, positive and significant effects, from 5% to 30% improvements • Across most grade levels K – 6 • Students with Disabilities showed gains • English Language Learners showed gains • Gains in both low and high performing schools • Gains in ELA and Math • Science and Social Studies (not statistically significant)

20. More conclusions • The lower the grade level, the greater the effects (e.g., the younger the student, the greater the effects); • The more economically disadvantaged the student, the greater the effects. • Major, unexpected finding: Arts Impact acts as a protective factor for students who did not have the benefit of attending a higher performing school.

21. Top conclusions (charts to follow) • Kindergarten through 2nd grade students in the treatment schools enjoyed the greatest degree of gains, including Students With Disabilities. • By Year 3, Kindergarten students realized a 14% differential, 56% of the treatment students reached Hi/Scope Child Observation Record (COR) Proficiency Level 4 (on the COR 5-point scale), versus 41% of the control students. • The gains in Year 3 for Kindergarteners With a Disability were even greater, with 25% more gaining on the COR Proficiency Level 4 (on a 5-point scale), 30% for the control students vs. 55% for treatment students. • The comparative gains for Kindergarten English Language Learners in Year 2 saw an approximate 4% advantage. • By Year 3, 1st grade students on Free or Reduced-Price Lunch (an indication of poverty), realized a 6% comparative gain in Reading, Language Arts and Mathematics (identical in all three) as measured by the CTB Terra Nova (which is configured to New York State Standards.)

22. More conclusions . . . • By Year 3, 1st grade English Language Learners realized a 12% comparative gain in Reading, an 18% comparative gain in Language Arts and a 13% comparative gain in Mathematics. • By Year 3, 1st grade Students With Disabilities realized a 15% comparative gain in Reading, a 6% gain in Language Arts and an 8% gain in Mathematics, compared to control school students. • For reasons we do not yet understand the intermediate grades (3 – 6) see greater gains for students in 4th and 6th grade than in 3rd and 5th grade. • In low performing schools, 4th grade students realized a 13% higher rate of English Language proficiency and a 10% higher rate of Mathematics proficiency (as shown by the New York State Tests), compared to low performing control schools. • In low performing schools, 6th grade student gains were noteworthy -- a 13% higher rate of treatment students achieving English Language Arts proficiency (New York State Levels 3 and 4), than control school students, and a 20% higher rate of treatment students achieved Mathematics proficiency compared to students in the control schools.

23. VII: Rochester Kindergarten Students Improvement in English Language Arts, Math, Science and Behavior 90 80 70 60 50 Percent Scoring at or above 4 on the Total K-COR 5 level Scale Control Schools Program Schools 40 30 20 (p<.001) 10 0 Fall, 2006 Spring, 2007 Fall, 2007 Spring, 2008 Fall, 2008 Spring, 2009 (p<.01) (p<.001) Martin F. Gardiner, Ph.D. Gains in Kindergarten – all students

24. V: Rochester 1st Grade Students With Disabilities Progress in English Language Arts and Math 45 40 35 30 25 Percent Scoring at or above Grade Level Control Schools Program Schools 20 • Year 1 15 10 5 0 NY Lang. (2006-2007) (p < .01) NY Math (2006-2007) Reading Reading NY NY NY Math NY Math Reading (2006-2007) 2007-2008 2008-2009 Language Language 2008-2009 2007-2008 (p< .05) 2007-2008 2008-2009 (p < .01) Martin F. Gardiner, Ph.D. Gains for Grade 1 ELL

25. V: Rochester 1st Grade Students With Disabilities Progress in English Language Arts and Math 45 40 35 30 25 Percent Scoring at or above Grade Level Control Schools Program Schools 20 • Year 1 15 10 5 0 NY Math (2006-2007) Reading Reading NY NY NY Math NY Math NY Lang. (2006-2007) (p < .01) Reading (2006-2007) 2007-2008 2008-2009 Language Language 2008-2009 2007-2008 (p< .05) 2007-2008 2008-2009 (p < .01) Martin F. Gardiner, Ph.D. Gains for Grade 1 Students with Disabilities

26. Quantitative Data • 143 teacher interviews were conducted at the end of the third year of the grant project • 1. What worked best?

27. 2. What is your evidence of student learning?

28. 3. What activities that the teaching artist modeled have you tried or would you do?

29. 4. What support do you need to continue with arts integration?

30. Did it work? • In short, Yes. 30,000 students and 16 standardized tests.

31. Summary • Overall, we have concluded the earlier the intervention, the greater were the effects sizes: We observe statistically significant positive effects in Kindergarten, first and second grade. We also observed the higher the poverty as determined by the Free and Reduced Price Lunch code the greater the effect size

32. Evaluation Tips • If you are contracting with an independent evaluation outfit, get your Non-Disclosure Agreement done ASAP; • Work with US DoE on this – and feel free to borrow from Rochester’s model (available upon request) • Go with unique IDs, so you can use your Lunch Code data – which can reveal big differences in effects size • NEVER, EVER release Lunch Code information with personally-identifiable data

33. What Next? • The results of this project – and the independent evaluation of this project – pose significant policy implications for high-poverty urban schools, including those with high populations of English Language Learners and Students with Disabilities.