Teacher Effectiveness

1. Teacher Effectiveness presented to Volusia Teachers Organization AFT Innovation Grant Subcommittees October 28, 2010 Dr. Serena Roberts

2. What does “teacher effectiveness” mean? • Think about the term “teacher effectiveness” and what it means to you • Turn to a partner and share your thinking • Based on what both of you have shared, come up with a common understanding to share with the group How does your collective understanding of teacher effectiveness relate to teacher evaluations? How does it relate to assessment-based accountability?

3. Objectives By the end of this session, participants will: • Understand terms relating to measurements of teacher effectiveness • Be aware of other state’s value-added models and the researchers associated with them • Know possible biases of different value-added models • Be able to compute simple gain scores • Be prepared to grapple with difficult questions about teacher effectiveness and ways different measurement models can play out in assessments of teacher effectiveness

4. Teacher effectiveness measurement context Part of a national movement The role of AFT’s Innovation grant Current focus with Race to the Top (RTTT) in Florida

5. The national context 2003 Rand Report, Evaluating Value-added Models for Teacher Accountability “Although VAM holds great promise, it also raises many fundamental and complex issues. . . . If these issues are not adequately addressed, VAM is likely to misjudge the effectiveness of teachers and schools and could produce incorrect generalization about their characteristics, thus hampering systematic effort to improve education” (p. iii).

6. The national context (continued) The entire Spring 2004 issue of Journal of Educational and Behavioral Statistics was devoted to value-added topics. In April 2008 the University of Wisconsin at Madison hosted the inaugural National Conference on Value-added Modeling.

7. Value-added terminology • Value-added assessment usually refers to a specific technique or assessment setting • Value-added measurement is usually used in discussing attempts to measure such intangibles as the impact of school environment, the long-term effect of prior teaching, teacher experience, and socioeconomic characteristic • Value-added modeling refers to the full range of statistical and mathematical formulations encompassing all value-added assessment techniques

8. Value-added models Layered Mixed Effects Models (LMEM) ignore sociodemographics, apportion credit for pupil learning growth among multiple schools, use incomplete observations Simple Fixed Effects Models (SFEM) highly correlated to LMEM so use of SFEM is preferred because of their simplicity; however, use of SFEM rather than HLM can be controversial Hierarchical Linear Models (HLM) The question of whether to adjust for minority status and poverty “is, perhaps, the most important issue faced when developing a school accountability system. Either inclusion or exclusion of them is likely to lead to a biased system. Which bias is most tolerable may depend on whether the system is to be a high-stakes one” (Tekwe, et al, 2004, p. 12).

9. Two important statewide value-added models Tennessee Value-Added Assessment System (TVAAS) Measures teacher effectiveness based on student gains, controlling for socio-economic status (SES) and other factors that influence achievement Before 2004 the model was criticized for absence of explicit control. In 2004 they introduced controls of SES and demographics. Controls had negligible impact on estimated teacher effects in TVAAS, but not on a SFE estimator with which the TVAAS was compared. Their explanation was that the use of covariance of tests in different subjects and grades results in a student’s history of test performance substituting for omitted background variables. Uses LMEM Sanders

10. Two important statewide value-added models Louisiana • Redesign of teacher preparation programs and use of effectiveness of experienced certified teachers as the point of comparison • Includes aspects of cost-benefit analyses and a skills-as-outcomes model • Looks at which practices are most effective and what characteristics of teachers are most powerfully tied to their instructional success • Coding and analysis of data about practices and characteristics to assess teacher effects • Results to be used to shape hiring practices • HLM uses prior achievement, Limited English Proficiency, gifted status, section 504 status, free or reduced price lunch status, student attendance, disability status, discipline record Noell

11. Other ways to look at value-added (micro and non-statistical) One way to look at the value one’s teaching adds is to gather data to document how one’s students do when they move to the next course. How do students who earn A’s in reading in your 2nd grade class do on their 3rd grade FCAT Reading exams? How do students who earn A’s in your Pre-Algebra class do in Algebra? The value added by changing a teaching practice sometimes called Response to Intervention (RTI) Non-statistical value added “When people talk about their experience with a really good teacher, they’re not talking about test scores. . . . They’re talking about a teacher who gave them self-confidence, the ability to learn, an interest and curiosity about certain subjects” (Pallas, 2010, as quoted by Stewart).

12. The role of the Innovation grant Subcommittees: Value Added Formative Assessment Alternative Assessment

13. Computing a simple gain score Please get into the following groups for this data exercise

14. Computing a simple gain score You will be given a data activity packet consisting of 8 sheets. Directions for each set of data are at the top of each sheet. Please do not look at subsequent sheets until you complete the previous sheet or are told to move to the next. If you have questions, please do not hesitate to ask for help.

15. Ranking teachers according to their gain scores Teacher 1, in School F, has a 74.3 average gain score in reading. Teacher 2, in School A, has a 150 gain score in reading. Based on this data, what can you conclude about the effect of each of these teachers on pupil learning? What other data would be helpful in coming to a conclusion? The importance of school effect

16. Other current resources The New Teacher Project “Teacher effectiveness should consider multiple sources of data and emphasize student learning outcomes.” Teacher evaluations and support for effective instruction (October 6, 2010) Teacher Evaluation 2.0 (October 2010) The Widget Effect: Our National Failure to Acknowledge and Act on Differences in Teacher Effectiveness (2009) Wisconsin Center for Education Research Using Teacher Performance Assessments for Human Capital Management (April 2010) New York Daily News “New York City plan to grade teachers with ‘value-added’ data is destructive, says Diane Ravitch” (October 25, 2010)

17. The role of the Innovation grant Subcommittees: Value Added Formative Assessment Alternative Assessment

18. Parting questions • If three teachers are ranked high, average, and low, using value-added methods, what is the difference among them in terms of the average number of test questions answered correctly by their students? • How many more correct answers does it take, on average, to raise a teacher from the low to the average ranking? • What are the factors on which these answers depend?

19. Parting questions Which of the 13 RTTT projects require knowledge of what we talked about today for effective implementation? • Expand Lesson Study * • Expand STEM Career and Technical Program Offerings • Increase Advanced STEM Coursework • Bolster Technology for Improved Instruction and Assessment • Improve Access to State Data • Use Data to Improve Instruction • Provide Support for Educator Preparation Programs • Improve Teacher and Principal Evaluation Systems • Use Data Effectively for Human Capital Decisions • Focus Effective Professional Development • Drive Improvement in Persistently Low-Achieving Schools** • Implement Proven Programs for School Improvement** • Include Charter Schools in LEA Planning

20. Parting questions One of the results of the Louisiana studies is: It is possible for teacher preparation programs to prepare new teachers whose students demonstrate achievement comparable to the achievement of students taught by experienced teachers. • How will this effect traditional pay scales? • How will it effect the improved teacher and principal evaluation systems mandated by Race to the Top?

21. Second data activity Please move to your grant subcommittee groups. You will be given a data activity packet consisting of 4 sheets. Directions are on each sheet. Please do not look at subsequent sheets until you complete the previous sheet or are told to move to the next. If you have questions, please do not hesitate to ask for help.