Meryle Weinstein, Emilyn Ruble Whitesell and Amy Ellen Schwartz New York University

1. Can Formal–Informal Collaborations Improve Science Literacy in Urban Middle Schools?  The Impact of Urban Advantage Meryle Weinstein, Emilyn Ruble Whitesell and Amy Ellen Schwartz New York University Improving Education through Accountability and Evaluation: Lessons from Around the World Rome, Italy October 3, 2012 Meryle Weinstein Emilyn Ruble Amy Ellen Schwartz

2. What is Urban Advantage? • Collaboration between New York City Department of Education and 8 New York City informal science education institutions • Led by American Museum of Natural History • Provides professional development to middle school science teachers and opportunities to students to engage in authentic science practice • Workshops for science teachers and school administrators • Science materials/equipment for schools, teachers, & students • Vouchers for class field trips, family field trips and visits • Launched in 2004-05 with 31 schools and in 2011-12 had 137 • Funded by NY City Council and DOE

4. Why Urban Advantage? • Increased calls for collaboration between formal and informal institutions, particularly around science • Growing aversion to science among students, particularly by the time they reach middle school • Middle school is time to grab students attention – “gateway” for high school science courses • Strong science instruction has been found to impact science persistence in high school

5. Our goal is to determine if the Urban Advantage program is effective. • Does Urban Advantage lead to increased student achievement? • UA students outperform non-UA students on 8th grade Intermediate Level Science Test • Modest impact • Magnitude increases over time • Students who attend UA schools are more likely to take and pass a Science Regents in 8th or 9th grade than students at non-UA schools

6. We make use of a rich longitudinal student level database. • Student-Level Data • NYCDOE Administrative Data, 2004-05 – 2009-10 • Socio-demographic characteristics, educational needs, and test scores

7. Raw performance data suggests UA is effective Student Weighted Mean Achievement, 8th Grade Intermediate Level Science (ILS) Test – Percent Proficient

8. Differences between UA and non-UA schools prior to joining UA are largely insignificant. Standard deviations are in parentheses Bold indicates differences are statistically significant at .05 level or less % Proficient is the percent scoring in levels 3 or 4

9. Basic Model Yijt= β1ijt + β2PreUA ijtβ3PostUA ijt+ β4STijt+θj + εijt • Y = individual student outcome • PreUA = indicator variable for whether school joined UA in next year • PostUA = indicator variable for whether school has joined UA • ST = vector of student characteristics • θ= school fixed effect •  = random error term

10. * p<0.05, ** p<0.01, *** p<0.001 Year, LEP, and Special Education dummies not shown Robust clustered standard errors in parentheses

11. Robustness Checks • Controlled for prior achievement • Magnitudes are smaller but still significant when controlling for prior math or reading scores • Lagged Math Scores, Post UA Yrs β=.037, p < .10 • Lagged Reading Scores, Post UA Yrs β=.045, p < .10 Caveat: Sample size decreases by 50,000 • No statistically significant findings for percent proficient

12. Linear Probability Coefficients, High School Outcomes * p<0.05, ** p<0.01, *** p<0.001 Robust clustered standard errors in parentheses Control variables not shown are: Black, Hispanic, Asian, Female, Poor, Special Education, LEP, and for Model 4 lagged_zmath.

13. Conclusions • Student performance increases with the implementation of UA and the magnitude of the difference increases over time. • Little change on ELA or math for 8th grade students, suggesting the effect is not merely reflecting coincident overall school improvement • UA also contributes to post-8th grade outcomes. Biggest impact is on the likelihood of taking the Living Environment Regents in 8th or 9th grade.

14. Policy Implications • First estimates of the impact of a science program on academic achievement • Inquiry as a method to approach science instruction is not emphasized in schools but more common in informal science institutions • Benefits of collaboration between formal and informal science institutions • Importance of strong partnerships between these institutions and between the institutions and the school district in which they work • Need for improved data linking teachers and students • Future research: Inside the black box

15. http://steinhardt.nyu.edu/iesp