Teacher Performance Pay: Evidence from the Andhra Pradesh Randomized Evaluation Study (AP RESt)

1. Teacher Performance Pay: Evidence from the Andhra Pradesh Randomized Evaluation Study (AP RESt) Government of Andhra Pradesh Azim Premji Foundation The World Bank DFID, UK Regional Conference on Quality Education for All New Delhi, 25 Oct 2007 Presented by Karthik Muralidharan, Harvard University Venki Sundararaman, World Bank

2. Why do we care? • Key policy challenge in India is to enable “inclusive growth” • Human capital investments are a critical enabler • Significant progress on enrolments through SSA, but levels of learning are very low • Large increases in budgetary allocations to education • Substantial inefficiencies in delivery of education services • Teacher absenteeism (~25%) and low teaching activity (<50%) • Over 90% of non-capital spending goes to teacher salaries • Correlations suggest that higher ‘levels’ of pay are not associated with better teacher performance • Performance pay for teachers is one frequently suggested option for improving school quality • Theoretical prediction about teacher incentives is ambiguous and empirical evidence is mixed • Identifying the causal impact is a central issue

3. The Role for Randomized Evaluations • Typical programs of governments are difficult to evaluate because they mostly use one of the following 3 procedures: • Case 1: Launch a program for the full population • Case 2: Launch a program for a target segment of the population • Case 3: Targeting based on expression of interest from the recipients • The problem in each of these cases is the lack of a meaningful comparison group that was similar in all other ways except for the program • Key idea of a randomized evaluation is to match/stratify schools on observables and then randomly assign some of them to a program or combination of programs • Since the program is randomly assigned, the remaining group of potential recipients provides a perfect control group of SIMILAR entities that did NOT receive the program • Randomized evaluation is universally considered to be the “gold standard” in research methodology

4. Questions/Contributions • Do teacher incentives improve test scores? • What, if any, are the negative consequences? • Should they be at the school or teacher level? • What is the impact of measurement & feedback? • How does teacher behavior change? • How cost effective is the incentive program? • How will teachers respond to the idea?

5. Location of Study • Indian State of Andhra Pradesh (AP) • 5th most populous state of India • Population of 80 Million • 23 Districts (2-4 Million each) • Close to All-India averages on many measures of human development

6. Sampling

7. Design Overview

8. Incentive Design • Teachers were eligible for bonus payments that paid Rs. 500 for every 1% increase in average test scores of their students • Bonus amount was set to roughly equal the spending on inputs and incentives • Teaching to the test/Effort Diversion • Less of a concern given extremely low levels of learning • Test-taking is an important skill in the Indian context • Test design can get progressively more sophisticated so that you cannot do well on the test without deeper knowledge/understanding • Threshold Effects • Minimized by making bonus a function of average improvement of ALL students • Teachers neglecting/penalizing weaker children • Mitigate/avoid this by tying incentives to “changes” from the baseline performance and assigning low scores to drop outs • Cheating/Paper leaks etc. • Potentially a big problem – here the testing is done by an independent outside testing team from Azim Premji Foundation with no connection to the school

9. Summary of Experimental Design • Baseline tests conducted in 500 sampled schools (June/July 05) • Stratified random allocation of 100 schools to each treatment (2 schools in each mandal to each treatment) (August 05) • Monitor process variables over the course of the year via unannounced monthly tracking surveys (Sep 05 – Feb 06) • Conduct 2 rounds of endline tests to assess the impact of various interventions on learning outcomes (March/April 06) • Interview teachers after program but before outcomes are communicated to them (August 06) • Communicate and provide bonuses in September 2006 and continue interventions for second year.

10. Specification

11. Impact of Incentives on Test Scores

12. Quantile Treatment Effects

13. Robustness/Heterogeneity of Treatment Effects • Incentive effect is positive (and mostly significant) across all 5 grades, all 5 districts, all 5 quintiles of question difficulty, and both rounds of endline testing • Cannot reject equality of treatment effect across sub-groups • No variation in incentive effects by HH demographics • Household affluence index, literacy index, caste, baseline score • No variation in incentive effects by teacher demographics • Rank, gender, experience or base pay • Only evidence of heterogeneous treatment effects is across school facilities • Interaction of incentives and school infrastructure index is significantly positive • Consistent with the idea that the incentives help to reduce the ‘slack’ in the use of existing inputs by teachers

14. Mechanical versus Conceptual - Examples

15. Impact of Incentives by Mechanical/Conceptual

16. Performance on Non-Incentive Subjects

17. Group versus Individual Incentives

18. How Much is Just Measurement?

19. How Did Teacher Behavior Change?

20. Impact of Para-teacher and Block Grants

21. Comparison of Inputs and Incentives

22. Teachers Liked the Program • Teachers interviewed in August 06 (before they know outcomes) • 75% of teachers say the program increased their motivation • 25% say their motivation was unchanged • 85% of teachers had a favorable opinion about the idea of bonus payments on the basis of improvement in student performance • 68% thought that the government should try and scale up this program in all schools • 75% were willing to accept a performance-pay system even under neutrality of the total wage bill • Teachers who show greater support for performance-pay (ex ante) are also likely to have performed better (ex post) • Implications for sorting into teaching profession

23. Policy Implications • Performance pay for teachers is likely to be a highly cost-effective policy for improving learning outcomes • Can combine elements of both group and individual-level performance pay • Can be largely cost/budget neutral when implemented in the context of an across the board salary increase • The broader point is that of creating a meaningful career ladder for teachers so that their professional trajectories depend on performance • Implementation details are critical and the key next step will be to build systems and infrastructure to do this

24. Ongoing and Future Policy-Focused Research • AP RESt is a long-term action-research project that is expected to continue at least until 2011 • 5-year MoU signed between GoAP and Azim Premji Foundation • We hope to systematically study the effectiveness of the most promising policy options to improve education in India • Performance Pay for Teachers (group and individual) • Para-teachers (locally hired under different contract structure) • Cash block grants to schools (focused on student-used inputs) • Student Incentives (based on levels and improvements of scores) • Extra regular teacher (can compare with para-teacher) • School Choice/Scholarships (including aggregate effects) • School Health (including various delivery models) • Teacher training programs • Studying each of these policy options in the same context makes AP RESt a unique test bed for research on education and service delivery in India