Sensitivity of Teacher Value-Added Estimates to Student and Peer Control Variables

1. Sensitivity of Teacher Value-Added Estimates to Student and Peer Control Variables October 2013 Matthew Johnson  Stephen Lipscomb Brian Gill

2. Value-Added Models (VAMs) Used Today Differ in Their Specifications

3. Research Questions • How sensitive are teacher VAM estimates to choice of control variables? • Are estimates for teachers with more students from disadvantaged backgrounds affected by this choice? • Does the substitution of teacher-year level average student characteristics in place of classroom averages impact teacher VAM estimates? • Does allowing for relationship between current and lagged achievement to vary based on student demographic characteristics matter for teacher VAM estimates?

4. Data • Data from a northern state and a medium-sized urban district in that state • District has more minority and low-income students than state average • Estimate separate VAMs using state data and district data • More control variables available in district VAMs • For peer characteristics, use teacher-year level averages in state VAMs, classroom averages in district VAMs • Each VAM uses three years of teaching data from 2008-2009 through 2010-2011

5. Baseline Model • Explore sensitivity to several specifications: • Exclude peer average characteristics (X̅i,t) • Exclude student characteristics (Xi,t) and peer characteristics (X̅i,t) • Add scores from two prior years (Yi,t-2) • Interact free/reduced lunch status with baseline scores • Estimate all models using the same set of student observations • Control for measurement error in prior test scores using an errors-in-variables approach • Empirical Bayes (shrinkage) adjusted estimates

6. Student and Peer Characteristics

7. Correlation of 8th-Grade State Teacher VAM Estimates Relative to Baseline Specification Baseline: Student characteristics, peer characteristics, and prior scores from t-1 Findings are based on VAM estimates from 2008–2009 to 2010–2011 on the same sample of students.

8. Percentage of 8th-Grade Reading Teachers in Effectiveness Quintiles, by VAM Specification Findings are based on VAM estimates for 3,347 reading teachers in grade 8 from 2008–2009 to 2010–2011. Correlation with baseline = 0.946.

9. How Are Teachers in One District Affected? • District has relatively large fraction poor and minority students Percentile Rank of District Teachers in State Distribution

10. Using Additional Controls in District Data Baseline: Student characteristics, peer characteristics, and prior scores from t-1 Findings are based on VAM estimates from 2008–2009 to 2010–2011 on the same sample of students.

11. Teacher-Year Average Student Characteristics vs. Classroom Average

12. Different Relationship Current and Prior Test Scores for FRL Students • Correlation of teacher effect estimates with baseline model above 0.99 for both subjects

13. Conclusions • Teacher VAM estimates highly correlated across specifications • Choice of control variables • Use of teacher-year level averages in place of classroom averages • Interaction between FRL status and prior scores • Choice of control variables can impact estimates for teachers of disadvantaged students

14. Context for Results • Other researchers have examined correlations in teacher effect estimates when different same-subject assessments are used as outcomes for teacher VAMs • The highest correlations these authors found are: • Lockwood et al. (2007): 0.46 • Sass (2008): 0.48 • Concoran et al. (2011): 0.62 • Lipscomb et al. (2010): 0.61 • Papay (2011): 0.54

15. For More Information • Please contact • Matthew Johnson • MJohnson@mathematica-mpr.com • Stephen Lipscomb • SLipscomb@mathematica-mpr.com • Brian Gill • BGill@mathematica-mpr.com