Teacher Quality, Teacher Evaluation, and “Value-Added”

1. Teacher Quality, Teacher Evaluation, and “Value-Added” Sean P. Corcoran New York UniversityEducation Policy BreakfastApril 27, 2012

2. How did we get here? • Research finds teachers are the most important school influence on student achievement • Teachers appear to vary widely in effectiveness, as measured by student gains on standardized tests • Teachers can have long-run measurable effects on life outcomes (e.g., Chetty et al., 2012)

3. How did we get here? • By many measures, teacher quality is inequitably distributed across students and schools • There is some evidence that teacher quality has declined over the long-run (Corcoran et al., 2004)

4. How did we get here? • If teachers are so important, what are we doing to ensure high-quality teachers can be found in every classroom, particularly for those students who need them the most? • The generally accepted answer among policymakers: not much, or at least current efforts are not working very well (e.g., see The Widget Effect)

5. Two key issues • What is teacher quality and how do we measure it? • What policies are most effective in improving the level and distribution of teacher quality?

6. The Teacher Quality Pipeline

7. What isteacher quality? • The easy (non-)answer: skills, practices, personal characteristics that positively impact desired student outcomes • Not a very helpful definition … but does make clear that it is ultimately outcomesthat indicate quality

8. The “old view” • Research and policy emphasized qualifications and experience as presumed indicators of quality

9. The “old view” – why? • Convenience – these measures are readily available and easily observable; a lack of data on outcomes themselves • Face validity– on their face, they seem sensible • Reward structure – traditional salary structure rewards these qualifications (e.g. MA, experience)

10. The “old view” • NCLB’s Highly Qualified Teacher provision: all teachers of core academic subjects must: • Have a BA or better in the subject matter taught • Have full state certification • Demonstrate subject matter knowledge

11. Policies that set high professional standards and barriers to entry ProfessionalDevelopment The Teacher Quality Pipeline

12. The “old view” • Surprisingly (or not) research has not found qualifications to be highly predictive of student outcomes (i.e. test scores), although some do better than others

13. The “new view” • “Teaching should be open to anyone with a pulse and a college degree—and teachers should be judged after they have started their jobs, not before” • Malcolm Gladwell, The New Yorker, 12/15/2008

14. The “new view” • “Success should be measured by results…That’s why any state that makes it unlawful to link student progress to teacher evaluation will have to change its ways.”PresidentBarackObama, July 24, 2009

15. The “new view” • In other words, let outcomes be the arbiter of quality • Great in theory, but whichoutcomes do we measure, and howdoes one measure teachers’ contribution to them? • How does one incorporate this information into personnel policies in ways that have desired effects?

16. Policies that focus on measurement and incentives Evaluation The Teacher Quality Pipeline

17. Measurement: outcomes • Outcomes: to date, whatever we have on hand • Typically, student growth on standardized tests in reading and math, grades 3-8 (though not for long) • Necessarily a subset of expected skills/outcomes • Necessarily a short-run outcome • Is our evaluation measure properly aligned with the goals we have for our educational system?

18. Measurement: value-added • Value-added: • The theoretical construct: a teacher’s unique impact on student learning • In practice, a statistic used to estimate this impact

19. Measurement: value-added • “Unique impact” implies causality – i.e. ruling other possible explanations for student learning • Several possible sources of error: • Systematic error (bias): attributing “value-added” to the teacher when it is really due to some other factor • Random error (noise): getting a “noisy signal” of the teacher’s contribution to learning

22. Measurement: value-added • So how can we attribute causality to a teacher? • If teachers were randomly assigned, this would be easy: systematic differences would almost surely be due to the teacher

23. Measurement: value-added • In the absence of this, we can instead devise a statistical model to account for other factors that explain differences in achievement

24. Measurement: value-added • Value-added is then defined as student achievement relative to predicted—in other words, there will always be a distribution of value-added + - 0

25. Value-added: bias • How confident are we that value-added measures isolate the unique contribution of individual teachers? • Classroom vs. teacher effects (esp. after 1 year) • Teacher vs. school effects • Mobile students • Tracking (e.g. Rothstein falsification test)

26. Value-added: bias • Does attributing outcomes to individual teachers even make sense? • Middle and high school settings • Team teaching • Evidence that teacher peers matter • The higher the stakes places on value-added measures, the more these questions matter

27. Value-added: noise • Even if value-added measures are not biased, they are still noisy—i.e. they are estimates with a high “margin of error” • More years of test results helps, although this may be “too late” to provide actionable information

32. Implications for policy • The promise of personnel decisions driven by outcomes has led to sweeping reforms of • Performance evaluations • Tenure and promotion, dismissal • Compensation • Principal evaluation • Evaluation of teacher training programs

33. Implications for policy • Race to the Top led numerous states to propose 50% or more of performance evaluations to be the “teacher’s impact on student achievement” • E.g. CO, FL, TN, NJ • Indiana: “negative” value-added teachers may not receive an effective rating, and tenure requires 3 years of effective ratings in a row • NY’s APPR: a somewhat more balanced approach

34. The Teacher Quality Pipeline

35. Implications for policy • What can we realistically expect from value-added based policies? • Not as much timely, actionable information as we might like – though perhaps useful as an early warning indicator • Crude differentiation of teachers at best, but more than current practice

36. Implications for policy • What are the risks and implications of a system based on high-stakes use of imprecise measures? • Mechanical applications are dangerous • Risk of improper attribution and “Type I errors” • Public reporting has minimal benefits and may do harm • Unnecessary diversion of resources • Unclear effects on entry into teaching profession

37. Implications for policy • Little is know about how value-added measures will be used in practice

38. References • Excellent and (mostly) non-technical resources: • Corcoran (2010) report for Annenberg • http://www.annenberginstitute.org/products/Corcoran.php • Harris (2010) Value Added Measures in Education • Koretz (2008) in American Educator • Braun (2005) primer for ETS • “Merit Pay for Florida Teachers: Design and Implementation Issues” (RAND 2007) • Rivkin (2007) CALDER policy brief • Harris (2009) and Hill (2009) point/counterpoint in the Journal of Policy Analysis and Management