Paying Teachers to Perform: The Effects of Bonus Pay in Pernambuco , Brazil

1. Paying Teachers to Perform: The Effects of Bonus Pay in Pernambuco, Brazil Barbara Bruns Lead Education Economist, Latin America and The Caribbean Region World Bank Claudio Ferraz Professor of Economics Pontifical Catholic University (PUC) Rio de Janeiro, Brasil Jessica Rodriguez, Tassia Cruz, Vitor Pereira OPCS –PSP GET –PREM Knowledge and Learning Series Washington, DC, May 8 2012

2. Overview • Teacher Bonus Pay (Pay for Performance) • Theory of action • Global evidence • Context for Bonus pay program in Pernambuco • Results to date • Next Steps

3. Teacher Pay for performance: theory of action (1) “Accountability strategy”: strengthens incentives/pressures for teachers to produce key results (growing use in OECD, MICs, 15 programs in Brasil since 2007) Theory of action: Bonus pay linked to student learning results can • Create short term stimulus for teachers to exert more effort and focus on learning results • Create long-term effects on teacher selection (by making profession more attractive to high performers) Logical policy in context of: • Consensus about the desired outcome: • student learning gains are key education result for growth and inequality reduction • Better measures of students learning • Difficulty monitoring the desired teacher behavior directly (effort and effectiveness in producing learning) • Perception that teacher salaries are a problem • Low and undifferentiated pay making it difficult to attract talent into teaching • No links between salary and performance (91% of salary variance in PE explained by yrs on job)

4. Teacher Pay for performance: theory of action (2) Design • Most common form is school-based bonus (individual bonus hard to implement) rewarding year on year learning gains • Relatively expensive (10-20% of wage bill) compared to other “accountability reforms” such as information and SBM (but not compared to across-the-board wage increases) Highly controversial • Corruptible: Extensive evidence of gaming and cheating on high stakes tests in US context • Debate on whether teacher effort is the binding constraint? Or teacher capacity? • Little evidence yet on longer-term selection effects

5. Teacher Pay for Performance: Global evidence • Seven rigorous evaluations outside US • bonus pay raised student learning outcomes 0.13-0.27 SD (highest for individual bonus in India) • Latest US evidence is different: even significant individual bonuses had no impact on student results (Nashville, NYC) • In several developing country cases bonus did not lower teacher absence (puzzling) • Most consistent “pathway” in JPAL studies (very low income settings) was extra teacher effort out of school

6. Teacher Pay for Performance: Global evidence • Research frontier now is to explain these results: • Context (low/high accountability?) Neal, 2011 • Design features? • Controllability (noise in performance measure + locality) • Size of bonus • Coverage/predictability of bonus • Deepen understanding of how incentives change teacher behavior inside the classroom • Positive: stimulate teacher effort (higher attendance, more preparation) • Negative: cheating, teaching to test

7. Teacher Pay for Performance: Global evidence

8. Teacher Pay for Performance: Global evidence PE PE

9. Bonus pay program in Pernambuco Brazil

10. Context for Bonus Pay in Pernambuco, Brasil • 27 States and 5,000+ municipalities deliver basic education • But federal govt. has put in place: funding equalization and high quality national “results measurement system” IDEB • Test scores • Pass rates • Pernambuco: • 950 State schools • 1 million students • Large schools (1,000 students and 30-40 teachers/per school) • Many teachers work two jobs (state and/or municipal or private school

11. Pernambuco Educational Context • Weak salary incentives for teachers (unified salary scale – 90% of salary explained by age, exp, and education • Weak sanctions and supervision (no dismissal for poor performance or high absence) • Limited results focus: • Teachers late or absent • Unused school libraries and computer labs

12. Pernambuco Educational Context In 2007, Pernambuco had some of the worst education outcomes in Brazil In 2008, the state government introduced a performance pay system for teachers: theBonus de DesempenhoEscolar (BDE) Reward for achieving 50% or more of annual school targets, modeled on IDEB

13. Targets sent to School Principals

14. Program Implementation • June 2008 • Bonus Program announced • Nov 2009 • 1st round classroom observations • April 2011 • Bonus paid • Nov 2008 • SAEPE exam • Aug 2009 • New targets • May 2010 • Bonus Paid • Nov 2010 • 2nd round classroom observations • Aug 2008 • Targets sent to schools • June 2009 • Bonus paid • Nov 2009 • SAEPE/ ProvaBrasil exams • August 2010 • New targets • Dec 2010 • SAEPE exam

15. Pernambuco Teacher Bonus: Research questions • Question 1: Did the bonus “work”? Did it improve test scores and pass rates? (more than they would have improved w/o the bonus) • Question 2: How did it work? • How did schools respond? What strategies? Change in strategies over time? • Did teacher behavior change in response to incentives? • Did effects vary by student, teacher, and school characteristics? • Were there negative effects? (eg. gaming, cheating) • Question 3: How do schools respond to non-linear performance targets and threshold? Did not winning the bonus encourage or discourage effort in subsequent year? • Research method: Difference in differences analysis plus classroom observation and directors’ interviews

16. Pernambuco School Bonus—Program Results

17. State IDEPE Index Results, 9th grade

18. State IDEPE Index Results, 11th grade (high school)

19. Impact Evaluation: Did the bonus “work”? • Strategy: Difference-in-differences • Can answer the question: Did the Pernambuco state schools improve more than they otherwise would have after the introduction of the bonus? • Method: exploit rich national and state-level performance data for the period pre- and post bonus to compare trends in: • Pernambuco state schools with other Northeast states

20. Pre-trend on test scores– Portuguese 5th grade

21. Pre-trend on test scores– Math 5th grade

22. Pre-trend on test scores– Portuguese 9th grade

23. Pre-trend on test scores– Math 9th grade

24. Impact Evaluation: Did the bonus “work”?

25. Post-bonus: PE state schools vs. other NE states Effects on test scores– Portuguese 5th grade

26. Post-bonus: PE state schools vs. other NE states Effects on test scores– Math 5th grade

27. Post-bonus: PE state schools vs. other NE states Effects on test scores– Portuguese 9th grade

28. ProvaBrasil: Pernambuco state vs. Northeast 9th grade 5th grade

29. Pass rate: Pernambuco vs. Other Northeast States Increase 10% in the pass rate

30. Overall Bonus Impact • Increase in test scores for 9th grade, both for Portuguese (.36 SD) and Math. (0.41 SD) using the difference-in-differences between Pernambuco and other NE states • Small effect on 5th grade math (.12 SD) and none for 5th grade language minor • Increases in approximately 10 to 12% on pass rates for 6th to 8th grades • No effects on pass rates for 1st to 5th grades (but very few state schools offer these grades)

31. Overall Bonus Impact • Why? Secretariat believes reason is less state school influence on 5th grade student performance • Fewer lower primary (5th grade) enrollments in state schools (officially, this level is decentralized to municipalities) • State policies have focused more resources on upper primary and secondary school • State policies such as bonus may have cumulative results (although results presented are short-term)

32. Research Question #2: HOWdid the bonus “work”? • Evaluation design: Classroom observations in a panel of 1800 classrooms in 300 schools • Can answer questions: • Do teachers behave differently in schools that achieve/don’t achieve the bonus? • Did teacher behavior change over time in Pernambuco state schools? • Are changes in teacher classroom practice correlated with results? • Surveys/interviews with school directors, teachers and regional supervisors • Can answer questions: • What strategies do schools use to try to reach targets? • How do these strategies correlate with bonus results?

33. Bonus Pay and Teacher Classroom practice in Pernambuco

34. Analysis of Panel of 220 Schools in Pernambuco – 2009 and 2010 (after introduction of school bonus program)

35. Slight increase in instructional time and decline in time spent on classroom management but increase in time off task (social interaction w/students) and no change in teacher absence from the class

36. Increase in teacher effectiveness keeping ALL students engaged(share of time entire class engaged with teacher) Small Group: 2 to 5 students Large Group: More than 5 students

37. Research Question 3: How do different targeting mechanisms affect bonus incentives?

38. Pernambuco school bonus: First round targets had sharp discontinuities (8th grade Portuguese)

39. Pernambuco school bonus: Second round targets (eliminated discontinuities)

40. Pernambuco school bonus: Third round targets (linear)

41. ….but maintained non-linearity in thebonus distribution

42. Preliminary (and incomplete) results • Losing the bonus by small margin seemed to create incentives for improvement compared to winning the bonus by small margin in year 1 • But effects heterogeneous based on school size and specific targets • Non-linear bonus assignment, at least close to discontinuity, does not seem to induce a discouragement effect • Much scope for further analysis of impact of target-setting mechanisms

43. Summary:what are we learning from Pernambuco? • Unlike US results, bonus pay can stimulate improvement in key school results within 1-2 years • Pernambuco results to date – even most conservative estimates – show a significant positive impact in the upper grades

44. Summary: What are we learning from Pernambuco? PE PE PE

45. Summary:what are we learning from Pernambuco? • Early evidence that it is possible to observe differences in classroom dynamics between improving and stagnant schools, and system-wide progress over time • (Hopefully) also some insights into design questions in setting up target-based bonus programs • Linear or nonlinear targets? • Linear or nonlinear bonus assignment? (everyone gets it or only those that pass a threshold for improvement – Sao Paulo vs PE) • Reward level of performance, value-added, or both? Should schools that remain at the top be rewarded (Sao Paulo vs PE)? • Should targets be multi-year? Renegotiated annually?

46. Summary:next phase of Brazil teacher incentives research • What strategies do schools most commonly employ in response to bonus incentives? (2012 qualitative survey and study) • How does school size (free-rider potential) and school “social capital” affect bonus attainment? (2012 analysis of existing survey data) • Does classroom-level improvement continue over time? Does it correlate with bonus? (panel of observations continuing in 2012 and 2013) • Do perverse impacts arise over time (as agents become more familiar with program and potential gaming strategies)? (unique set of “high stakes” and low stakes test data) • How to design effective bonus pay programs? (Comparative study of 3 yr. results in different Brazilian programs – MG, PE, SP, RJ – 2012/13) • Size of bonus? Design of targets? Reward level of performance, value-added, or both? Multi-year targets or renegotiated annually?

47. MuitoObrigados! Barbara Brunsbbruns@worldbank.org Claudio Ferrazcferraz@econ.puc-rio.br