Decentralization and the quality of teaching

1. Decentralization and the quality of teaching Steven Rivkin April 2014

2. The decentralization of public schooling typically shifts authority and responsibility from the ministry of education to municipalities • Decentralization affects many aspects of schooling, but I focus on its effects on the quality of teaching and instruction

3. Outline of Talk • Discuss selected evidence on the effects of high-quality instruction and professional development • Discuss potential advantages and challenges of decentralization, focusing on the quality of teaching • Compare decentralization and achievement trends in the Czech Republic and Poland • Outline plan for teacher and principal evaluation efforts in Dallas, Texas

4. Evidence on the effects of teacher quality on future earnings • Investigates the effect of 4th-8th grade teacher quality in New York City on earnings • quality is measured as the contribution of a teacher to test score growth • This is only one dimension of teacher quality • math score 5th grade minus math score 4th grade • control for differences in family income and special needs • compares teachers within the same school who teach children with similar backgrounds

5. Rich data to learn about schools • New York tests all children in grades 3 to 8 every spring in math and reading • Information on income and special needs • School data link teachers to their students • Can follow students over time • Enables measurement of teacher quality • Tax data on earnings for these students as adults is matched with school data

6. Results • Value added to achievement is clearly a partial and flawed measure of teacher quality, BUT these results show it is strongly related to future student earnings • Having a teacher who is one standard deviation better in terms of raising achievement (moving from the middle of the distribution past 1/3rd of teachers) a student’s increases annual earnings in the future by roughly 1% on average • Multiply this effect by 25 students in the class and by the number of years a student works to calculate the total benefit of a better teacher (many $1,000s)

7. Benefits of Data on Schooling and Outcomes • Data make such research possible • Data enable schools to evaluate teacher effectiveness as part of the hiring, retention and pay processes • Data enable schools to evaluate principal effectiveness as part of the hiring, retention and pay processes • Data also provide information for parents and the public • In the US laws protect the privacy of students • Personnel information on employees including measures of teacher performance is generally private • Publicly available information on school performance shows average achievement, achievement growth and pass rates, graduation and attendance rates, and other averages

8. Relevant teacher effects publications of the research by Raj Chetty, John Friedman and Jonah Rockoff • “Great Teaching: Measuring its Effects on Students’ Future Earnings,” Education Next, Summer 2012 (not technical) • Measuring the Impacts of Teachers II: Teacher Value-Added and Student Outcomes in Adulthood,” American Economic Review, forthcoming (complete analysis) • “Big Study Links Good Teachers to Lasting Gain,” by Annie Lowrey, New York Times, January 6, 2012

9. Research on Benefit of Teacher Evaluation • Evaluation that potentially affects teacher pay and job retention provides incentives for teachers to increase effort • Evaluation of teachers also provides information on teacher performance that can be used by school administrators the mentoring of teachers in order to support their efforts to improve

10. Study of Teacher Improvement • A large US city put in a teacher evaluation plan as part of a performance pay system • teachers were observed and then met with the principal to discuss their performance and specific areas that needed improvement • average test score growth in math and reading for each classroom teacher was calculated • This study compares average growth after evaluation with average growth before evaluation

12. Relevant evaluation effects publications of the research by Eric Taylor and John Tyler • “Can Teacher Evaluation Improve Teaching?,” Education Next, Fall 2012 (not technical) • Source for plot • “The Effect of Evaluation on Teacher Performance,” American Economic Review, forthcoming (complete analysis)

13. US Department of Education funded study on intensive professional development for mathematics instruction • Average differences between the teachers in the treatment and control schools are unbiased estimates of program effects • Randomization eliminates systematic differences • Schools were randomly assigned to two groups • Teachers in the treatment schools received summer workshops, seminars and coaching (114 hours total) • Teachers in the control schools participated in any existing programs at the school but not the intensive professional development

14. Results after second year of program • No significant effect on mathematics achievement • Average score in treatment schools is not significantly higher than the average in control schools • No significant effect on teacher knowledge • These results cast doubt on the idea that in-service training is effective at improving the quality of teaching • These were expensive, highly regarded professional development programs • In the US there are a number of random assignment studies done by both academics and research firms that provide valuable information that can guide policy • In many cases these avoid expensive but ineffective programs

15. Citation • Garet, M., et al, (2011). Middle School Mathematics Professional Development Impact Study: Findings After the Second Year of Implementation (NCEE 2011-4024). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. • Report available on the IES website at http://ncee.ed.gov • This study was completed by analysts at policy research firms

16. Summary of Results • These three studies suggest that school leaders have an important role to play in raising the quality of instruction through the selection, mentoring, evaluation, and contract renewal decisions • Evidence suggests that large professional development programs or simple rules to govern teacher hiring are not likely to be effective • Rather it is a leader’s skill at identifying good teachers to hire, providing good feedback on teacher performance, and making decisions about which teachers to keep and which to let go that appears to be much more important • If school decentralization improves the quality of school leadership it may be a very good policy

17. School Decentralization • Responsibility for many decisions shifts from central authority to local institutions, often in response to dissatisfaction with the centralized system • Municipalities must develop the capacity to produce high-quality schools • A particularly important decision is the hiring of a principal, who will have responsibility for virtually all aspects of school operations

18. One view of the decentralization experiences of the Czech Republic and Poland • Much authority was transferred from the central education ministry to local municipalities in both countries • However • Central ministry retained much more involvement with regards to curriculum and evaluation of schooling in Poland

19. Percentage of decisions taken at each level of government in public lower secondary education (2011)

20. Decline in the Czech central ministry support for school administration and involvement in evaluation and curriculum may have been harmful • There was likely a loss in the capacity to support high-quality instruction • As Daniel Münich showed, the decline in support likely caused many school leaders to focus more attention on school operations and less on instruction • This does not appear to have been the case in Poland

21. Some municipalities in the Czech Republic may have devoted less effort to ensuring high education quality, particularly in terms of school principal selection and retention decisions • This may have been more of a problem in lower-income communities where parents may be less involved in pushing for a higher quality of education

22. The absence of national monitoring of educational outcomes in the Czech Republic complicates efforts to evaluate school performance • It is difficult to identify a decline in quality in a school, municipality or the country as a whole without evidence of outcomes that can be compared with other schools or years • It complicates efforts to manage and support teachers • Areas where students are struggling are not illuminated

23. The Czech response to salary increases in other occupations following labor-market decentralization was far weaker than in Poland • As shown by Professor Munich, teacher salaries relative to other occupations that employ college graduates increased much more in Poland than in the Czech Republic during the 1990s and 2000s • Many older teachers may choose to remain in teaching, because it is more difficult to switch occupations as you get older

24. But younger teachers and students with better opportunities are more likely to move away from teaching if salaries of teachers fall further below salaries in other occupations • Professor Munich will provide some evidence • The sharp decline in the school-age population in the Czech Republic during the late 1990s and 2000s likely masked the problem of teacher retention • Fewer new teachers were needed because of the decrease in the number of school-age children

26. Evidence highlights importance of high-quality instruction and school leadership • High quality teaching cannot be produced simply through regulations, including mandatory professional development or teacher education programs • Rather it requires good decisions regarding teacher hiring, teacher support and teacher mentoring • Good systems of evaluation of both teachers and school leaders support good decisions

27. Dallas Independent School District • Dallas school administration • Elected school board hires and evaluates the superintendent • Superintendent supervises area superintendents who govern different parts of the city • Area superintendents supervise school principals • This is a typical structure for the US

28. Dallas Evaluation Systems • Principal evaluation by school district administrators has been in place since 2013 • A teacher evaluation system has been proposed but not yet approved or implemented • Both systems combine supervisor observations of performance with data on student outcomes

29. Availability of Assessments is Crucial • The district is developing assessments available to supervisors in order to ensure that there is student outcome data for all subjects and grades • State test score data are available only for certain subjects and grades • District assessments cover all subjects including music, art and physical education • This is a very ambitious undertaking • The goal is to treat all teachers similarly

30. Teacher evaluations • Teacher evaluation combines the ratings based on observations of teaching completed by the principal typically with the assessment data • Assessments include estimates of student achievement growth • School principals have an incentive not to give very high ratings based on observations, because their own evaluation suffers if teacher ratings differ substantially from the student achievement growth and other outcomes

31. Addressing the serious problem of rating inflation • In previous evaluation systems it was typical for almost all teachers to be rated as good or excellent • There was little incentive for high standards • Student outcomes were not considered typically • This district requires some fraction of teachers and some fraction of principals to receive lower ratings and punishes principals who give a teacher with low student achievement growth or low outcomes in other areas a high rating

32. Basing pay on performance • The proposed evaluation system would eliminate the previous practice of basing teacher salary entirely on experience and level of education (often specified in union contracts) • Salary would be influenced by experience but determined largely by performance

33. Benefits of strong evaluation systems • Teachers and principals have strong incentives to work hard to raise the quality of instruction and produce better outcomes for students • More skilled individuals who expect to be successful as teachers or principals have a stronger incentive to enter teaching