Special Data Opportunities in Florida

1. Special Data Opportunities in Florida David N. Figlio University of Florida and National Bureau of Economic Research

2. What’s so special about Florida? • Florida has developed a remarkable ability to analyze data across a wide variety of settings • Facilitated through legislative authority – but reflective of exceptional nurturing of interagency relationships that facilitate data-sharing • Easier also to collect large datasets directly from school districts in areas not present in state data • Due in part to the relatively small number of highly-organized school districts that are acculturated to data-sharing and facilitating policy-motivated research

3. Two main special data opportunities in Florida • Florida Education and Training Placement Information Program (FETPIP) and K-20 Education Data Warehouse • Established in 1988 to document compliance with vocational education performance requirements • Expanded to become a system of collecting and sharing individual-level data across all education sectors (K-12, vocational education, community colleges, state universities) • Follows education, employment and earnings outcomes longitudinally for students – integrated into a relational database

4. Two main special data opportunities in Florida • Independent extensive surveys of school policies and practices • Collaborative effort between David Figlio (U. Florida), Dan Goldhaber (U. Washington), Jane Hannaway (Urban Institute) and Cecilia Rouse (Princeton U.) • Attempted census of all public schools in Florida • First conducted in 1999-2000, then followed up in 2001-02 and 2003-04 • Surveys of teachers in 2000-01, 2002-03, and 2004-05 in 288 representative elementary schools • Better than 70 percent response rates in all waves; over 80 percent longitudinal response rates

5. Examples of FETPIP data • Data from public two- and four-year institutions, as well as in-state private colleges and universities • In-state, data are collected on courses taken, programs of study, and attainment • Florida is working with the National Student Clearinghouse to include enrollment and credential records for a large fraction of out-of-state college students

6. Examples of FETPIP data • Using social security numbers, FETPIP marches student records with other outcome data for all students exiting Florida public institutions as well as some private exits • Examples include information on further education, job placement, compensation, military service, incarceration • Result: a remarkable tool for policy generation, evaluation and research

7. School Surveys • All “regular” public schools in 2001-02 and 2003-04; • 70%+ response rate in each year; • 2,095 schools responded in 2002; 81% of these responded in 2004.

8. Policies to improve low-performing students Lengthening instructional time Reduced class size for subject Minimum time required for tested subject instruction Minimum time required for non-tested subject instruction Scheduling systems Additional school resources Policies to improve low-performing teachers Teacher resources Teacher incentives Teacher autonomy District control Principal control School climate School policies/practices can be grouped into domains

9. Feeling the Florida Heat? How Low-Performing Schools Respond to Voucher and Accountability Pressure Cecilia Elena Rouse Princeton University & NBER Jane Hannaway The Urban Institute Dan Goldhaber University of Washington David N. Figlio University of Florida & NBER

10. There is little evidence on how schools respond to accountability pressure… • Improved teacher effectiveness and greater focus on basics; • Teaching to the test; • Cheating; • Reclassification of students; • “Strategic” suspension of students.

11. Our question: Do schools change their policies and practices in response to school accountability and voucher pressure?

12. Our approach: • Study effects of school accountability on student test score performance in Florida with a change in the A+ Plan for Education; • Analyze effects of accountability on schools using longitudinal data on school policies collected from surveys of school principals in 1999-00, 2001-02, and 2003-04; • Attempt to determine if the policy changes explain the test score effects.

13. We find… • Among elementary schools, student achievement significantly increases among F-graded schools; • F-graded schools appear to respond with policy changes; • These policy changes appear to explain non-trivial portions of the student gains, particularly in math.

14. Since 2002, Grade Points = percent students meeting levels 3+ in reading, writing, and math + percent students making “learning gains” in reading and math + the percentage of the bottom 25% that have improved scale points in reading.

15. Table 1: The Distribution of Elementary School Grades, by Year

16. Table 2: Transition Matrix in Predicted Grades Based on 2002 Grade Change, Elementary Schools(row percentages)

17. Appendix Table 1 (part 1): Mean School Characteristics in 2002

18. Appendix Table 1 (part 2): Mean School Characteristics in 2004

19. Table 3 (part 1): Regression-Discontinuity Estimates of the Effect of Receiving an F Grade in Summer 2002 on Student Performance

20. Table 3 (part 2): Regression-Discontinuity Estimates of the Effect of Receiving an F Grade in Summer 2002 on Student Performance

23. Table 4 (part 1): Regression-Discontinuity Estimates of the Effect of Receiving an F Grade in Summer 2002 on Student Performance

24. Table 4 (part 2): Regression-Discontinuity Estimates of the Effect of Receiving an F Grade in Summer 2002 on Subsequent Student Performance: Alternative specifications

25. Table 7 (part 1): Seemingly-Unrelated Regression and OLS Results of the Effect of Receiving an F Grade in Summer 2002 on School Policy in 2003-04

26. Table 7 (part 2): Seemingly-Unrelated Regression and OLS Results of the Effect of Receiving an F Grade in Summer 2002 on School Policy in 2003-04

27. Table 8 (part 1): OLS Results of the Impact of Receiving an F Grade in Summer 2002 on School Selected Individual Policies in 2003-04

28. Table 8 (part 2): OLS Results of the Impact of Receiving an F Grade in Summer 2002 on School Selected Individual Policies in 2003-04

29. Table 9 (part 1): The Effect of Including School Policy/Practice Variables on Regression-Discontinuity Estimates of the Effect of Receiving an F Grade in Summer 2002 on Subsequent Student Performance: Fifth-Grade Cohort of 2002-03

30. Table 9 (part 2): The Effect of Including School Policy/Practice Variables on Regression-Discontinuity Estimates of the Effect of Receiving an F Grade in Summer 2002 on Subsequent Student Performance: Fifth-Grade Cohort of 2002-03

31. In sum… • We estimate effect sizes in reading test scores among “F-graded” schools between 6-10σ and effect sizes in math between 7-14σ. • We also find that “F-graded” schools appear to focus on low-performing students, lengthen the amount of time devoted to instruction, adopt different ways of organizing the instructional environment of students and teachers, increase resources available to teachers, and decrease principal control. • These policies may explain at least 10% of the gains in reading and at least 25% of the gains in math.

32. Caveats…. • F-graded schools receive additional state assistance (e.g., assessment and course materials, increased professional development for teachers); • While our results suggest that schools respond to accountability in potentially educationally meaningful ways, we do not observe student performance along all dimensions.