Behavioral Responses to Teacher Transfer Incentives: Results from a Randomized Experiment

1. Behavioral Responses to Teacher Transfer Incentives: Results from a Randomized Experiment INVALSI Conference on Improving Education through Accountability and Evaluation: Lessons from Around the World Rome, Italy October 4, 2012 Steven Glazerman  Ali Protik  Bing-ru Teh  Julie Bruch Neil Seftor

2. Policy Problem • Best teachers may not be working with the students who need them the most • Shift focus from improving productivity of the teacher workforce to composition • Big gaps in knowledge • Weak documentation of the policy problem • Lack of data on teacher transfer behavior • Lack of data on whether skills transfer • Controversy about teacher quality measures (value added)

3. Policy Response: Talent Transfer Initiative • $20,000 transfer incentive • Identify highest-performing (HP) teachers • Use value-added analysis, three years of data • Three pools: elementary, MS math, MS language arts • Top 20% are “highest performing” • Identify potential “receiving schools” • Recruit transfer candidates, arrange interviews • Support transfer teachers, issue payments • HP teachers already in potential receiving schools get retention stipend of $10,000

4. Research Questions • How do HP teachers respond to a monetary transfer incentive? • How do hard-to-staff schools respond to the opportunity to hire a HP teacher? • What impact do transfer teachers have in their new settings? • Did their skills transfer, i.e. were they portable? • Was “value added” the right metric?

5. Summary of Findings to Date • Implementation • Filling vacancies was feasible • Large pool of candidates needed • Meaningful contrast achieved • Intermediate impacts • Increased experience and credentials slightly • No significant impact on climate or collegiality • No change in how students assigned to teachers • TTI transfers used less & provided more mentoring • Impact on test scores and retention • Will be public in the final report (2013)

6. Study Design

7. Experimental Design • Identify potential receiving schools with a vacancy in a targeted grade/subject • Unit of randomization = teacher team • Team types can be: • Elementary self-contained math and reading • Middle school math • Middle school English/language arts (ELA)

8. Study Design, Illustration Randomization Block School A School B

9. Study Design, Illustration Randomly assign teacher teams (grade within school) to treatment or control Focal Teachers School A School B

10. Data

11. Ten Large, Diverse Districts in the Study Cohort 1: seven districts in five states Cohort 2: three districts in two more states

12. Data • Primary Data Collection: Surveys • Candidates • Receiving school teachers in study grades • Receiving school principals • Secondary Data • District-provided test scores and demographics • School-provided teacher rosters

13. Sample (Cohort 1) • 7 districts • Large, diverse • 5 county, 2 city • 1,012 transfer candidates • 63 transfers from 51 sending schools • 86 receiving schools • 124 teams randomized • 15,266 students • Below average prior achievement • 6% white, 48% African American, 72% free lunch

14. Behavioral Response to Incentives: Implementation Findings

15. Findings on Response to Incentives • Low takeup rates, most candidates do not apply • Not too low to fill positions (90% filled) • Hard to predict who transfers

16. Response to Incentives in 7 Districts: Takeup Rates

17. Types of Transfers by Change in School Achievement Ranks Before and After Transfer N = 63

18. Types of Transfers by Change in School Poverty Ranks Before and After Transfer N = 63

19. Who Filled the Vacancies?

20. Behavioral Response Within the Receiving Schools:Intermediate Impacts

21. Findings on Impacts on School Dynamics • Survey questions on degree of collaboration, mutual trust, or sharing ideas: no evidence of impact • Differential assignment of students to teachers: mixed evidence of impact • Mentoring and leadership: treatment led to more mentoring provided, less mentoring used

22. Mentoring Received and Provided to Others Receives Mentoring Mentors Others

23. Summary of Findings to Date • Implementation • Filling vacancies was feasible • Large pool of candidates needed • Meaningful contrast achieved • Intermediate impacts • Increased experience and credentials slightly • No significant impact on climate or collegiality • No change in how students assigned to teachers • TTI transfers used less & provided more mentoring • Impact on test scores and retention • Will be public in the final report (2013)

24. Future Work • Impacts on test scores and retention • Cost-benefit • Shadow price of raising test scores using CSR • Retention adjusted impacts, extrapolate into future? • Spatial analysis of mobility decisions • Related policies

25. Related Policies • Transfer groups of teachers (e.g. through reconstitution) • Additional screening criteria for HP teachers • Bonus conditional on performance in new school • Policy that spans district boundaries (e.g. statewide)

26. THE END(extra slides follow)

27. Summary of Prevalence Findings • Districts vary • Elementary and middle school differ • Overall pattern suggests: • Unequal access at middle school level • Less evidence for unequal access at elementary level

28. Prevalence of HP Teachers: Do Low-Income Students Have Equal Access?

29. Prevalence of Highest-PerformingMiddle School Math Teachers* Quintiles Based on Poverty * Statistically significant

30. Prevalence of Highest-PerformingMiddle School Language Arts Teachers* Quintiles Based on Poverty * Statistically significant

31. Prevalence of Highest-PerformingElementary Teachers Quintiles Based on Poverty

32. Results for Individual Districts Results, Five Districts at a Time

33. Prevalence of Highest-Performing Middle School Math Teachers (Districts A-E) Quintiles Based on Poverty

34. Prevalence of Highest-Performing Middle School Math Teachers (Districts F-J) Quintiles Based on Poverty

35. Overall, but Using Achievement to Divide Schools

36. Prevalence of Highest-PerformingMiddle School Math Teachers* Quintiles Based on Achievement * Statistically significant

37. Prevalence of Highest-PerformingMiddle School Language Arts Teachers* Quintiles Based on Achievement * Statistically significant

38. Prevalence of Highest-PerformingElementary Teachers* Quintiles Based on Achievement * Statistically significant

39. Components of Estimated Teacher Performance • Decompose value added estimate Total Performance Persistent Teacher Ability Returns to Specialization Noise, Luck, Measurement Error Transitory Performance

40. Prevalence of Highest-Performing Middle School ELA Teachers (Districts A-E) Quintiles Based on Poverty

41. Prevalence of Highest-PerformingMiddle School ELA Teachers (Districts F-J) Quintiles Based on Poverty

42. Prevalence of Highest-Performing Elementary Teachers (Districts A-E) Quintiles Based on Poverty

43. Prevalence of Highest-Performing Elementary Teachers (Districts F-J) Quintiles Based on Poverty

44. Study Design, Crossover Case School pair with matching vacancies in two grades. Randomization Block School A School B

45. Study Design, Crossover Case (cont’d.) School A School B

46. Teacher Team Dynamics

47. Team and Focal Teacher Analysis • Team-level • Impact estimate has intent-to-treat (ITT) interpretation Under zero resource allocation effect: • Focal teacher comparison • Impact estimate denotes the direct impact • Nonfocal teacher comparison • Impact estimate denotes the indirect impact

48. Interpretation/Analysis Issues • Dilution of direct effect • Non-compliers (unfilled vacancies) • Block-defined subgroups • High contrast transfers • High value added transfers • Complier blocks

49. Self-Reported Reasons For Not Applying Percentages, N = 680

50. How Are Students Assigned to Classrooms?Principal Report (N=57 Treatment, 54 Control) None of the differences are statistically significant.