Informing Policy: State Longitudinal Data Systems

1. Informing Policy: State Longitudinal Data Systems Jane Hannaway, Director The Urban Institute CALDER www.caldercenter.org

2. State of U.S. Education • ½ of minority students graduate from high school • 4 grade level gap between white and minority students by 12th grade • 15% of minorities earn BAs w/in 9 years of 9th grade

3. The WILL and the WAY • The WILL • Left, Right, Center • Agreement on education crisis • Strange bedfellows • The WAY • Few, but growing, guideposts

4. Finding the WAY with Evidence-A New Day- • Who has the evidence? • States have the makings of the evidence • Where are the makings? • State administrative data systems • Why do states have it? • Important effect of NCLB • Why important? • Address questions never before possible

5. Research Background: What We Know • Teachers matter- single most important schooling contributor to student outcomes • Wide variation in teacher effectiveness. Some teachers are simply much better than others • Standard measures of teacher quality not much related to effectiveness, but directly related to spending.

6. Research Background:What We Don’t Know • What is it about teachers that matters?

7. 3 Research Probes • Teacher Maldistribution • Teacher Selection • Teacher Mobility

8. Teacher Maldistribution 1 • Comparison of VA of teachers in high/ low poverty schools • North Carolina and Florida • Findings • Low poverty - higher va, but not much • High poverty – larger variation in school

9. Teacher Value-Added at Percentiles by School Poverty Levels (North Carolina-Math)

10. Teacher Value-Added at Percentiles by School Poverty Levels (Florida- Math)

11. Novice teachers are less effective than experienced teachers. • Returns to experience taper off 3-5 years.

12. Distribution of Value-Added of Elementary Math Teachers in High Poverty Schools Solid line: Novice teachers Dash line: Teachers with 1-2 years of experience Dotted line: Teachers with 3-5 years of experience

13. Distribution of Value-Added of Elementary Math Teachers in Lower Poverty Schools Solid line: Novice teachers Dash line: Teachers with 1-2 years of experience Dotted line: Teachers with 3-5 years of experience

14. Teacher Maldistribution 2 • New York City • Phasing out of emergency certification • Introduction of alternative route teachers

15. LAST Exam Failure Rate of Elementary Teachers by Poverty Quartile, 2000-2005

16. LAST Exam Failure Rate of New Teachers by Poverty Quartile, 2000-2005

17. Predicted Effectiveness For Highest and Lowest Poverty Schools Narrows by 25%

18. Can change predicted effectiveness by selection up-front • Meaningful difference based only on attributes, though monitoring, development and selective retention also needed

19. Teacher Selection • Teach for America • North Carolina • Secondary school • Mainly math and science

20. TFA Findings – high school Student FE, Math subjects All TFA coefficients are significant at the .05 level.

21. Teacher Mobility • Mobility highest at most challenging schools • The worst teachers are the first to leave • General tendency to move to more affluent schools

22. Topic of the Day: Performance Incentives • Objective?? • Recruitment/ selection • Retention/ deselection • Increase performance thru effort

23. Issues • How good are the measures? • Individual vs school rewards? • Teachers without test scores?

24. VA Measures • Problems • Year to year variability • Measurement error • Sorting • How serious? • Less serious for policy research • More serious for individual stakes

25. Predicting Performance • Using first 2 yrs of performance – top to top/ bottom to bottom quintile • Goldhaber and Hansen (NC): 46%/ 44% • Koedel and Betts (SanDiego): 29%/ 35% • Sass (Florida): 22-32%/ 24-24%

26. Policy Implications • Use VA freely for research • Use VA carefully for individual teacher judgments • Important information • Corrorboration • More years are better • Move tenure decision out!

27. Research Questions • Are teachers in high poverty schools more/ less effective (value-added) than teachers in lower poverty schools? • Do school factors affect differences in the value-added of high poverty and lower poverty teachers? • Do teacher qualifications affect differences in the value-added of high poverty and lower poverty teachers?

28. Data • Florida (2000/01- 2004/05) • Elementary • Student achievement – FCAT-SSS • Grades 3-10 • Teacher links • Assignment, certification, experience, education • North Carolina (2000/1-2004/5) • Elementary • Student achievement • EOG – grades 3-8 • EOC – secondary subjects • Teacher linked through proctor and verification • Assignment, certification, experience, education

29. Definitions • High poverty elementary schools (>70% FRL students) • Lower poverty elementary schools (<70% FRL students) • Very low poverty schools (<30% FRL students).

30. NC Student-Teacher Link EOC student-level records Aggregate to EOC test classrooms by school, year, subject, proctor id Decision Rules Match if teacher and proctor id identical and  fit statistic < 1.5.

31. Sample Restrictions • Exclude charter schools • Exclude schools that switch high poverty to lower poverty status • Only classrooms w/ 10-40 students • Only self-contained elementary classrooms

32. Analytic Sample Note: We focus on elementary schools, grades 3-5 where poverty information is most reliable. We exclude teachers from charter schools and we exclude classrooms with <10 students or >40 students in our samples.

33. Methodological Challenges • Non-random sorting of teachers and students • Distinguishing teacher and school effects • Precision in Teacher Effects Estimates • Sources of Teacher Effectiveness Differentials

34. Descriptive Findings:Elementary Student Demographics

35. Descriptive Findings: Student Performance * Differences between the given estimate and the corresponding estimates for schools with 70-100% FRL students significant at ≤ 5% and ** differences significant at ≤ 1%.

36. Descriptive Findings:Teacher Experience

37. Descriptive Findings:Teacher Qualifications

38. Distribution of Value-Added of Elementary Reading Teachers in Lower Poverty Schools Solid line: Novice teachers Dash line: Teachers with 1-2 years of experience Dotted line: Teachers with 3-5 years of experience

39. Distribution of Value-Added of Elementary Reading Teachers in High Poverty Schools Solid line: Novice teachers Dash line: Teachers with 1-2 years of experience Dotted line: Teachers with 3-5 years of experience

40. Differences in Estimates of Teacher Value-Added

41. Magnitude of Differences in Value Added Estimates

42. Differences in Standard Deviations of Value-Added

43. Differences between Lower- and High-Poverty by Percentile of Teacher Value Added

44. Teacher Value-Added at Percentiles by School Poverty Levels (North Carolina- Reading)

45. Teacher Value-Added at Percentiles by School Poverty Levels (Florida- Reading)

46. Correlation of Teacher Qualifications and Value-Added

47. Sources of Difference in Teacher Value-Added Between High-Poverty and Lower-Poverty Elementary Schools

48. Sensitivity Analysis • School Effect • Empirical Bayes Adjustment

49. Conclusions • Teachers in high poverty schools, on average, are less effective than teachers in lower poverty schools. • Changing schools (high poverty/lower poverty) does not affect teacher effectiveness • There is greater teacher variation within high poverty schools than within lower poverty schools.

50. Conclusions (con’t) • Differences in teachers in High Poverty and Lower Poverty schools: • only weakly related to teacher qualifications • more strongly related to marginal effect of qualifications (experience) • not explained by school poverty level