Tim R. Sass

1. Research Using StateLongitudinal Data Systems: Accomplishments and Challenges – The Case of Florida Tim R. Sass

2. Overview • The Status of Statewide Longitudinal data Systems • Projects Underway Using Statewide Data from Florida • Challenges to Using Statewide Longitudinal Data for Research • What Lies Ahead

3. The Status of State Longitudinal Data Systems • Data Quality Campaign’s “10 Essential Elements” • A unique statewide student identifier that connects student data across key databases across years • Student-level enrollment, demographic and program participation information • The ability to match individual students’ test records from year to year to measure academic growth • Information on untested students and the reasons they were not tested • A teacher identifier system with the ability to match teachers to students • Student-level transcript information, including information on courses completed and grades earned • Student-level college readiness test scores • Student-level graduation and dropout data • The ability to match student records between the P–12 and higher education systems • A state data audit system assessing data quality, validity and reliability

4. The Status of State Longitudinal Data Systems • States That Can Track Student Gains and Match Teachers to Students • Arkansas, Delaware, Florida, Georgia, Hawaii, Kentucky, Louisiana, Mississippi, New Mexico, Ohio, South Carolina, Tennessee, Utah, West Virginia, Wyoming • North Carolina researchers do this indirectly through the identity of the teacher administering exams in elementary school • Some states have only recently acquired the ability to match students and teachers • Kentucky (2006-07), Wyoming (2006-07)

5. The Status of State Longitudinal Data Systems

6. Research Projects Using Statewide Longitudinal Data From Florida • School Choice • “Charter Schools and Student Achievement in Florida” (Sass (2006)) • Mature charters on par with traditional public schools in math and yield higher reading scores than the average traditional public school • Charter school competition associated with small improvement in math achievement in traditional public schools • “Charter Schools and Educational Attainment” (Booker and Sass) • Will investigate the impact of charter high school attendance on high school graduation and college attendance in Florida and in Chicago • “Vouchers and Special Education” (Sass) • Will analyze the decision to accept a voucher and the effects of vouchers on the supply of private schools and the achievement of special education students who remain in public schools

7. Research Projects Using Statewide Longitudinal Data From Florida • Peer Effects • “Classroom Peer Effects” (Burke and Sass) • Preliminary evidence suggests that peer influences are significant and the effects are non-linear • Low-achieving (lowest quintile) students worse or no better off in classes with more high-achieving (top quintile) peers • Mid-level students generally worse off with more low-achieving peers and better off with high-achieving peers • High-achieving students better off with more high-achieving peers and worse off with a greater fraction of low-achieving peers

8. Research Projects Using Statewide Longitudinal Data From Florida • Teacher Training • “Teacher Training, Teacher Quality and Student Achievement” (Harris and Sass (2007)) • Experience matters, especially over the first five years • Pedagogical content knowledge matters, both pre-service and in-service • Other forms of preparation do not seem to impact teacher effectiveness • No significant correlation between a teacher’s college entrance exam scores and later effectiveness as a teacher • Teacher Preparation Programs • In conjunction with researchers from New York, Louisiana and Ohio, will be analyzing the relative effectiveness of teachers who obtain certification through various routes

9. Research Projects Using Statewide Longitudinal Data From Florida • National Board Certification • “The Effects of NBPTS-Certified Teachers on Student Achievement” (Harris and Sass (2007)) • NBPTS certification provides a positive signal of teacher effectiveness in some cases, though the ability of NBPTS certification to identify high-quality teachers varies considerably across grades and subjects • Results are sensitive to the exam used to measure student achievement • No evidence that the NBPTS process itself enhances teacher quality • Little of no evidence of positive spillovers from NBPTS certified teachers to non-NBPTS-certified teachers

10. Research Projects Using Statewide Longitudinal Data From Florida • Teacher Labor Markets • “A Longitudinal Analysis of Teacher Mobility: Intra-District Moves, Inter-District Moves and Departure Decisions” (Feng (2007)) • Exit decisions influenced by teacher salaries relative to other occupations and the proportion of minority students in their school • Inter-district moves determined by relative salaries across districts and minority enrollment • Intra-district movement toward schools with higher student achievement and fewer minority students • Minority teachers’ job decisions less effected by minority student enrollment than those of their white colleagues

11. Research Projects Using Statewide Longitudinal Data From Florida • Accountability • “School Accountability and Teacher Job Choice” (Feng, Figlio and Sass) • Will investigate the impact of school accountability pressures on teachers decisions to move between schools or exit teaching entirely • Will exploit the fact that Florida changed the mechanism for grading schools which lead some schools to face increased accountability pressure and lessened the pressure faced by other schools

12. Challenges to the Use of Statewide Longitudinal Data • Achievement Tests and the Measurement of Student Achievement Gains • Relatively little known about the effects of alternative assessment measures on measured impacts • Test score scaling • Validity of using achievement tests to measure intertemporal gains in achievement • Criterion-reference exams versus developmental scale scores • Norming scores by grade and year • Choice of test instrument • Some evidence that results differ across tests (Harris and Sass (2007)

13. Challenges to the Use of Statewide Longitudinal Data • Methodology • How well do “quasi-experimental” methods control for selection biases? • Recent evidence on value-added models versus random assignment by Cantrell, Fullerton, Kane and Staiger (2007) • Inconsistent Estimation Methods and Specification in “Value-Added” Achievement Models • Student and teacher effects versus student and teacher covariates • Random versus fixed effects models • Gain scores versus partial persistence models • Inclusion/exclusion of Peer Effects

14. Challenges to the Use of Statewide Longitudinal Data • Use of Teacher Effects to Measure Teacher Quality • Not much known about inter-temporal consistency of individual teacher effects • Koedel and Betts (2006) • Little evidence on sensitivity of teacher effect estimates to changes in model specification • Use in policy research versus use for accountability and merit pay

15. The Future of Research Using Statewide Longitudinal Data • Cross-State Analyses • Due to lack of data, most statewide research so far has been limited to Florida, North Carolina and Texas • Many studies using large urban school districts, such as New York, Chicago and Los Angeles • Data availability in other states will help determine if previous findings are robust across geographic areas • Variability in effectiveness of NBPTS-certified teacher in Florida and in North Carolina found by Ladd, Sass and Harris (2007) • May be able to exploit cross-state variation in policy environments • RAND charter school project

16. The Future of Research Using Statewide Longitudinal Data • Linking Value-Added Measures to Qualitative Evidence • Comparison of Teacher Observations to Value-Added Measures of Teacher Quality • Rockoff, et al. • Comparison of Principal Evaluations of Teachers to Value-Added Measures • Harris and Sass (2007) • Matching Survey Evidence to Administrative Data