Magnet Schools and Peers: Effects on Student Achievement

1. Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam Gamoran, Ellen Goldring and Keke Liu.

2. Research Questions • Has attending a magnet school caused an increase in mathematics achievement? • How large is the influence of peers on mathematics achievement? • How much of the magnet school effect remains after controlling for the influence of peers?

3. Study Setting • Middle Schools in a Large Southern District 1 selective academic magnet 4 non-selective magnets 5 student cohorts 6 years: 1998-99 through 2003-04 Grades 5 & 6

4. Admissions Lotteries • Oversubscribed magnets conduct lotteries • Students may enter multiple lotteries • Students who are not outright winners are placed on wait lists • Wait-listed students accepted until the first week of school

5. Non-lottery Admissions • Sibling preferences • Promotion from a feeder school • Geographic priority zone These students are not included in the study sample (though they do enter the calculation of peer characteristics).

6. Research Design • Lotteries assign students randomly to school type and to peers (magnet school peers vs. non-magnet peers). • Randomized design circumvents biases arising from self-selection of schools and peers.

7. Limitations of Design • Results may not generalize beyond lottery participants. • Effects are relative (magnet schools vs. mix of non-magnet schools attended by lottery losers).

8. Lottery Participation

9. Grade 5 Enrollments

10. Potential Pitfalls • Substantial non-compliance, especially among winners of non-academic lotteries, attenuates estimated treatment & peer effects based on comparison of winners and losers. Remedy: Use lottery outcomes as instruments to predict probability of attending magnet school, outcomes interacted with peer variables at magnet & zoned schools as instruments for peer characteristics.

11. High rates of attrition from district can introduce systematic differences between treatment and control groups. Remedies: Control for student characteristics (race, income, ESL, special ed, gender, prior achievement). Analyze attrition patterns for evidence of differences between winners and losers.

12. Participating in multiple lotteries increases chances of winning. “Multiple participants” may differ in ways related to achievement. Remedy: Control for the combination of lotteries each student entered. Winners are compared to losers who entered the same combination.

13. Lotteries randomly assign students to magnet school peers or peers in their neighborhood (zoned) school, but lotteries do not determine the characteristics of the latter—residential decisions do. Remedy: Control for characteristics of the peers in the zoned school.

14. Peer Characteristics • Percentages black, low income (free & reduced-price lunch program), special ed, ESL, female • Absenteeism rate • Disciplinary incidents (rate per student) • Intra-year mobility • Prior achievement in math and reading

15. Model (Summary) • Two treatment variables (academic magnet, composite non-academic magnet) • Variation in peers resulting from lottery outcomes • Other controls (student characteristics, peers at the zoned school, lottery participation indicators, year by grade effects)

16. Findings • When model does not include peer characteristics - Academic magnet, + 18% in grade 5, drops to +10% in grade 6 (% of normal year growth) - Non-academic magnet, no grade 5 effect, +54% in grade 6

17. When models include peer characteristics - Reducing percent black from 75% to 25% increases scores by 60% of normal year growth. - Effect of percent low income is about half that large. - Other peer characteristics have no statistically significant effect.

18. - Controlling for either percent black or percent low income, the effect of the academic magnet disappears. - The large 6th grade effect in the non-academic magnets remains substantially undiminished.

19. Checking Alternative Interpretations • Are peers a proxy for heterogeneous response to treatment? Check: Interact magnet treatment indicators with all observed student characteristics. Finding: Peer effects are undiminished.

20. Are peers a proxy for teacher quality? Check: Control for teacher quality by including teacher fixed effects. Finding: Peer effects are undiminished.

21. Attrition, Academic Magnet

22. Attrition, Composite Non-Academic Magnet

23. Potential Attrition Biases • Lottery losers are more likely to leave the system than winners. • Losers are also more likely to leave when they can afford private schooling. These tend to be higher-achieving students. Result: Losers who remain in the system have lower achievement than winners who remain.

24. Unfavorable peers at zoned school make losers more likely to leave system. • Effect greatest among those who can afford private schooling. Result: Quality of peers positively correlated with losers’ achievement. Estimated peer effects appear too strong.

25. Checking Attrition Bias • Are rates of attrition correlated with variables that predict individual achievement (race, income, prior achievement)? • Yes, but not differently for winners and losers.

26. Conclusions • For at least some students in some places, magnet schools have a positive effect on academic achievement. • There are very strong peer effects on middle school achievement. Do not appear to operate through behaviors readily quantified with administrative data (attendance, disruptions, mobility).