How Much Value is Added? An Evaluation Plan for the Achievement Challenge Pilot Project

1. How Much Value is Added? An Evaluation Plan for the Achievement Challenge Pilot Project

2. What’s wrong? Teachers affect student performance, however… • Policy problem • General & specific teacher shortages • Measuring teacher effectiveness • Providing incentives to teachers • Need • System to recruit, retain, and reward high quality individuals in the teaching field Little Rock Merit Pay Year 1 Evaluation

3. Can current system rectify the problems? • Status Quo: • Single Salary System • Based on tenure and degree • Arguments for single system: • Fair • Simple (critics call it a “breathing bonus”) • Status quo • Concerns: • Lacks extrinsic reward for innovation, creativity, hard work • Lacks extrinsic reward for innovation • Does not encourage or reward outcomes • Does not recruit, retain, or reward effective teachers • If status quo isn’t working, what alternatives do we have? Little Rock Merit Pay Year 1 Evaluation

4. Policy Solutions • “Lump Sums” (recruit and retain) • Often in the form of lump increases • Intuitively lacks motivation to work harder • Differential Pay (recruit and retain) • Hard-to-staff schools • Specific subjects • Disadvantaged students • Merit Pay (rewards) • Teacher characteristics • Teacher behavior • Student performance gains Literature: Johnson, 2000; Lazear, 1996; Murnane & Cohen, 1986 Little Rock Merit Pay Year 1 Evaluation

5. Merit Pay Literature • Supporters believe performance improves: • Innovation • Work harder • Salary satisfaction • Opponents believe performance decreases • Counter-productive competition • Degraded work environment • Focus on high-performing students • Evidence:Very few evaluations • Policy questions: • Effects of merit pay programs on student performance? • Effects of merit pay programs on teachers? Little Rock Merit Pay Year 1 Evaluation

6. Possible Policy Implications • Possible options: • Improves student achievement, and teachers like program: • Improves student achievement, but teachers dislike program • Does not improve student achievement, but teachers like program; • Does not improve student achievement, and teachers dislike program. Little Rock Merit Pay Year 1 Evaluation

7. Achievement Challenge Pilot Project (ACPP) • Program Goals: • Increase student performance • Reward effective teachers • Make positive influences to school culture • Ultimately, recruits, retains, and rewards effective teachers • 5 elementary schools in Little Rock School District • Financial rewards based on student performance • payouts computed as NCE gains between fall and spring tests (SAT-9; SAT-10) • Meadowcliff payouts per student gain • Wakefield payouts based on class average gains Little Rock Merit Pay Year 1 Evaluation

8. ACPP Addresses Literature Concerns Table 1: Payouts for Wakefield for 2006-07 Little Rock Merit Pay Year 1 Evaluation

9. “Observable” School Characteristics Table 2: Demographic Characteristics of Schools in 2005-06 ACPP Evaluation Little Rock Merit Pay Year 1 Evaluation

10. Research Question 1: Student Effects • Question: • What is the impact of the ACPP on the math performance of students? • Method: • Student level fixed-effects regression model • Data provided by the Little Rock School District • Test scores • Stanford Achievement Test-9 (2003; 2004) • Iowa Test of Basic Skills (2005; 2006) • Reduces “gaming effect” • Demographic data • Race, Poverty (FRL), Gender, Age Little Rock Merit Pay Year 1 Evaluation

11. Methods: Data – Why Math Only? Table 3: Summary of Tests by Grade and Year for Fall 2006 Report ITBS 2005, Language subtest not administered to Grade 4 & 5 Little Rock Merit Pay Year 1 Evaluation

12. Methods: Analytic Strategy • Regression • Student-level Individual Fixed effects • Compares the difference in test scores for treatment students to the difference in test scores for control students • This model only applies to 4th and 5th grade students because they are the only students who possess • pre-gains (2002-03 or 2003-04 to 2004-05) • post-gains (2004-05 to 2005-06) • Meadowcliff removed – no pre-gain scores Little Rock Merit Pay Year 1 Evaluation

13. RQ1: Study Sample Characteristics Little Rock Merit Pay Year 1 Evaluation

14. Teacher Effects What are the attitudes regarding merit pay of ACPP teachers compared to those of teachers in the comparison schools? Little Rock Merit Pay Year 1 Evaluation

15. Teacher Survey • Advantages • Innovation • Work harder • Salary satisfaction • Disadvantages • Counter-productive competition • Degraded work environment • Focus on high-performing students • Teacher effectiveness Little Rock Merit Pay Year 1 Evaluation

16. Policy Implications & Conclusions • ACPP improves student performance • Student performance increased 3.5 NCE points • Teachers support the ACPP • Significantly more satisfied with ACPP than single salary system • Believe the program did not lead to counterproductive competition • Believe the school environment is more positive with ACPP • Teachers believe ACPP has positive impacts for students • Based on student performance increases and teacher opinions, program should be expanded to other elementary schools. Little Rock Merit Pay Year 1 Evaluation

17. Limitations & Policy Concerns • Receptivity is a factor • Teacher support may be vital to program success • Limited sample of teachers (58 treatment) • Limited sample of students (132 treatment) • All from same school • Only two grades used • Funding • $225,000 / school Little Rock Merit Pay Year 1 Evaluation

18. Survey Practice & Questions Little Rock Merit Pay Year 1 Evaluation