Do Accountability and Voucher Threats Improve Low-Performing Schools?

1. Do Accountability and Voucher Threats Improve Low-Performing Schools? David N. Figlio and Cecilia Elena Rouse NBER Working Paper No. 11597 August 2005 JEL No. I20, I21

2. Aim of the paper Study the effects of the threat of vouchers and school stigma in Florida on the performance of low-performing schools

3. General Framework 1/3 • Florida’s program utilizes stigma by grading schools “A” (excellent) through “F” (failing) • Students in “F” schools are eligible for taxpayer-founded vouchers (Opportunity Scholarship Program) cheques gave to low income families in order to help them in the school choice. The families can choose among private or higher-rated schools • OSP has two rationales: • Fairness to reduce social stratification • more competition into the provision of education (in fact, based on the economic theory, an increased competition should force schools to improve)

4. General Framework 2/3 • 1996 Florida began rating schools based on their aggregate norm –referenced test performance introduction of the Critically Low Performing Schools List (four school rating groups) • 1998 introduction of the new Florida Comprehensive Assessment Test (FCAT) based on the SSS in grades 4 reading ,5 math, 8 and10 both in reading and math • 1999 introduction of the A+Plan. It dividedschool in five rating groups and it establish private school vouchers assigned in event of chronic low performance. Two types of tests: nationally norm referenced test FCAT-NRT and the curriculum based assessmentFCAT-SSS the two tests assess different set of skills. The FCAT-SSS tests a narrower set of skills but in greater depth than do the broader nationally NRT

5. General Framework: A + Plan 3/3 • In the early years of the A+ Plan, schools were identified as low performing: • if fewer than 60% attain level 2 in reading, fewer than 60% attain level 2 in math, and fewer than 50% attain level 3 in writing. • In addition, schools that improved by more than one grade level or retained an “A” were awarded an additional $90-$100 per student.

6. Evidence of test score improvements 0.3 s2

8. Improvements in students learning 1/3 • Real gains or statistical artifact? • If real gains due to : voucher threats or other elements of the accountability system such as grading stigma? There are several alternative hypotheses that might explain real gains: 1) mean-reverting measurement error 2) changing student characteristics 3) opportunistic teachers’ behaviours

9. Improvements in students learning 2/3 1) Mean – reverting measurement error • The gains obtained by “F” rated-schools were due to mean reversion. Indeed the gains obtained by these low rated schools in the following year had been larger than the average gains. This could be due to the “floor effect” and/or “ceiling effect”: Schools that scored highly on the test in one year are disproportionately likely to experience lower than average gains the following year • The school rating is based on test’s result of only one year, thereby it’s possible that many of the F-rated schools had only a transitory low grade due to low test scores. Their score would have increased in subsequent years even in absence of the A+ Plan

10. Improvements in students learning 3/3 2) Changing students’characteristics districts attempted to redraw school attendance area boundaries in order to improve the student characteristics of low rated-schools. 3) Opportunistic schools’ behaviours schools only focus on high-stakes grades test, teaching to the students the test’s matter.

11. Empirical framework 1/2 T ist is student i’s test score in school s in year t F is dummy variable indicating whether or not the school attended by student i received a failing grade of “F” in 1999. YEARt is a vector of year effects POSTt dummy variable indicating if the year is after the implementation of the A+ Plan (in these data it is equal to one if the year is 2000 and zero otherwise) GRADEist is a vector of dummy variables indicating the student’s grade in school f s is a vector of school-fixed effects m s school-level time trends predicted from the pre-1999 data e ist is a normally distributed error term  b key coefficient of interest. It measures the change in a student’s test score resulting from her school having received an “F.”

12. Empirical framework 2/2 • Utilizing a difference-in-differences framework • The Tist –Tist-1 is the outcome observed for two groups of test score in school s for two time periods. One of the groups is exposed to a treatment (A+Plan) in the second period but not in the first period. The second group is not exposed to the treatment during either period. • The dummy variable POSTt captures possible differences between the treatment and control groups prior to the policy change. F is captures aggregate factors that would cause changes in outcome even in the absence of a policy change. b multiplies the interaction term, F is and POSTt • This particular specification of the education production function allows us to control for the student’s prior academic achievement (by controlling for Tist-1) and this allows to have the right estimates of the effect of the A+ Plan.

13. Data • Student-level data from a set of participating school districts from 1995-2000. • These data include scores on: • FCAT-SSS Florida Comprehensive Assessment Sunshine State Standards • Norm-referenced NRT • Basic student demographic attributes, including information on student race, ethnicity, poverty status, limited English proficiency status, and disability status. • Data on third, fourth, and fifth grade NRT scores and FCAT-SSS scores for grade 4 in reading and grade 5 in math. These data include over 1500 schools • Focus on elementary grades (from 1 grade to 6 grade) because they comprised the vast majority of “F” schools • Dependent variable is the normal curve equivalent of the test scores • The mean is 50, the individual-level standard deviation is 21.06

15. Dependent variable 

17. Dummy variables 0.07 s Ss at 10%

18. Ability

19. Study changes in academic performance that resulted from the earlier policy of placing schools on a critically low performing list

20. Conclusion1/2 • Overall, they find that students showed significant improvement on the high-stakes exam (the FCAT-SSS) in both reading and math. • However, once they examine the effect of the accountability system on a nationally norm referenced test and control for student characteristics (by including the lagged test score), they find that there was no large-scale relative improvement in average NRT reading scores due to the A+ Plan among students in the lowest performing schools. • There were modest relative improvement in average NRT math scores, although this improvement appears largely concentrated in the high stakes grade. • And while they find some evidence that sanctioned schools put additional resources into low-performing students in math, more generally they find little evidence that these schools treated students differentially than low-performing schools that did not receive an F.

21. Conclusion 2/2 • A+plan is seen from Florida as a good tool to improve the low performing schools in this way all children can have a good quality of education. • Florida with the institution of this provision is sure that the improvement of the low performing schools conducts to an improvement in children’ learning. This would be logical • But as the authors show the low performing schools in order to avoid the stigma to receive an F (that is the main reason why the school improve and not for the voucher threat) they behave in an opportunistic manner for example focusing on the low performing children and leave aside the other subgroups of students, teaching the high-stake test contradiction with the two rationales of the OSP: fairness and competition