Allison Atteberry University of Virginia Susanna Loeb Stanford University James Wyckoff

1. Do First Impressions Matter?Improvement in Early Career Teacher EffectivenessThurs, Feb 21st: Panel III (1:30– 3:15pm)New Research on Training, Growing and Evaluating Teachers Allison Atteberry University of Virginia Susanna Loeb Stanford University James Wyckoff University of Virginia

2. Motivation • The difference between effective and ineffective teachers have substantial effects, not only on standardized tests outcomes, but on later life outcomes • (Chetty, Friedman, & Rockoff 2011) • Teachers vary widely in their ability to improve student achievement • (Gordon, Kane, & Staiger 2006; Rivkin, Hanushek, & Kain 2005; Rockoff 2004; Sanders & Rivers 1996) • Teachers improve, on average, during the first three to six years of their careers. • (Papay and Kraft 2011; Harris & Sass 2009; Ost 2009; Clotfelter, Ladd, & Vigdor 2006, 2007; Rockoff 2004) • Growing interest in how can we use measures of teacher effectiveness in practical ways to improve the quality of the teacher workforce. Some possible mechanisms include: • improved teacher recruitment and formal preparation • targeted mentoring or professional development interventions • performance-based incentives • dismissals of ineffective teachers

3. Goals of the paper • Why we focus on early career • Can identify struggling teachers sooner for help • Examine the likelihood of future improvement • Make strategic pre-tenure dismissals to improve teacher quality • Less is known about stability of value-added in this period • Main Question: How well do teacher value-added measures in the first two years predict future teacher performance? • What are the mean returns to experience in the first 5 years? • How much do teachers vary around that overall pattern? • How predictive is initial performance to future performance? • What are the tradeoffs between potential errors in predictions?

4. Overview of Approach • Characterize early career performance • Use VA scores (shrunk) as year-specific estimates of teacher effectiveness • Group into quintiles based on scores for elementary school teachers • What is “initial” performance: • 1st year only? 2nd ? Mean of both years? Consistent in first 2 years? • Future Performance: • Years 3, 4, or 5? Mean of all three? • Examine and predict future performance (years 3 thru 5) in relation to initial performance. For example: + • Present evidence about errors in prediction, potential tradeoffs

5. Analytic sample 15.3%

6. Mean returns to experience Student Achievement Returns to Experience in Early Career, Across Various Studies

7. Variation around the mean

8. Variation around the mean

9. Variation around the mean

10. Mean Value Added over Experience, by quintile of initial performance

11. Quintile transitions

12. Distribution of future scores, by quintiles of initial performance

13. Distribution of future scores, by quintiles of initial performance

14. Distribution of future scores, by quintiles of initial performance

15. Distribution of future scores, by quintiles of initial performance 76.5 %

16. Distribution of future scores, by quintiles of initial performance

17. Distribution of future scores, by quintiles of initial performance 68.9 %

18. Adjusted R-squared estimates

19. Individual predicted future scores

20. Potential errors in categorizing teachers • This discussion lends itself naturally to a consideration of the tradeoffs associated with identifying teachers as low-performing based on imperfect measurements from a short period of time in the early career. • H0: Teacher is at least average (not ineffective) in the long run • Type I Error • Reject true null hypothesis • Falsely predict a teacher will be low-performing when she turns out to be at least average in the long run. • Type II Error • Fail to reject a false null hypothesis • Fail to identify a teacher as ineffective when she actually is ineffective in the long run.

21. Departures by Future Performance Quintile Based on Early Career Performance Type II: Identify a teacher as at least average who is actually low performing in the long run Type I: Identify a teacher as below average who is actually at least average or better in the long run

22. conclusions • VA quintiles in the first two years appears to predict well the members of the lowest quintile of teaching in the future • However, there is a somewhat high “cost” to these predictions—a fair number of teachers who began in the lowest quintilelater become at least average • That said, very few of the top performing teachers began in the bottom quintile • That is, its very unusual for teachers to go from “extremely good” to “extremely bad” during the early career. • Its useful to frame tradeoffs as Type I and Type II errors, both of which are costly to students. • Could using multiple measures to reduce both types of errors

23. Next steps • Middle school teachers • Revisit Attrition Question: • Who has consistent VA data? • What are the reasons teachers appear to be missing VA in years when they are still teachers? • Is there evidence that these missing data points are selective or strategic? • Return to the variability in growth in the early career. • Initial performance is one correlate of future performance • However some teachers who initially struggle make up more ground than others. • What are correlates of growth in the early career?

24. Do First Impressions Matter?Improvement in Early Career Teacher EffectivenessTHANK you. CONTACT: Allison Atteberry University of Virginia acma@virginia.edu Susanna Loeb Stanford University sloeb@stanford.edu James Wyckoff University of Virginia wyckoff@virginia.edu