Measuring teachers' contributions to student learning in the non-tested subjects and grades

1. Measuring teachers' contributions to student learning in the non-tested subjects and grades Laura Goe, Ph.D. Research Scientist, ETS, and Principal Investigator for the National Comprehensive Center for Teacher Quality Webinar Presentation Supporting OSDE Teacher Leader Effectiveness Initiatives June 26, 2012

2. Laura Goe, Ph.D. • Former teacher in rural & urban schools • Special education (7th & 8th grade, Tunica, MS) • Language arts (7th grade, Memphis, TN) • Graduate of UC Berkeley’s Policy, Organizations, Measurement & Evaluation doctoral program • Principal Investigator for the National Comprehensive Center for Teacher Quality • Research Scientist in the Performance Research Group at ETS

3. The National Comprehensive Center for Teacher Quality • A federally-funded partnership whose mission is to help states carry out the teacher quality mandates of ESEA • Vanderbilt University • Learning Point Associates, an affiliate of American Institutes for Research • Educational Testing Service

4. The goal of teacher evaluation

5. Measures and models: Definitions • Measures are the instruments, assessments, protocols, rubrics, and tools that are used in determining teacher effectiveness • Models are the state or district systems of teacher evaluation including all of the inputs and decision points (measures, instruments, processes, training, and scoring, etc.) that result in determinations about individual teachers’ effectiveness

6. A non-tested subject example (1) • Industrial Technology 1: Working in small groups and independently, students will rotate through several units that emphasize the use of computer technology in a guided hands-on approach to learning.  The units cover flight, drafting, wood projects, metals, geometric design using plastics and mousetrap vehicles.  Each unit is designed to reinforce the core curricular areas of Math, Science, and History while having fun. • First Avenue Middle School, Arcadia, California http://fa.ausd.net/modules/cms/pages.phtml?pageid=163324&SID

7. A non-tested subject example (2) • The Great Depression: Describe the causes and effects of the Great Depression and give examples of the government’s response to conditions and events. Analyze the impact of New Deal programs and explain how the role of government changed during the 1930s. [Standard Indicators: USH.4.4, USH.4.5, USH.4.6, USH 4.8] • Indiana’s Core Standards • http://www.in.gov/edroundtable/files/SocStudies_FINAL3.pdf

8. A non-tested subject example (3) 4 types of musical behaviors: Types of assessment Slide used with permission of authors Carla Maltas, Ph.D. and Steve Williams, M.Ed. See reference list for details. Responding Creating Performing Listening Rubrics Playing tests Written tests Practice sheets Teacher Observation Portfolios Peer and Self-Assessment

9. Oklahoma State Department of Education Priority Academic Skills (PASS) District Arts Assessment Report http://ok.gov/sde/arts#Assess

10. Validity • Measures don’t “have” validity—depends on how the measure is used • There is still considerable work to be done in validating measures for teacher evaluation • Multiple measures may serve as a means of triangulating teacher effectiveness results • Herman et al. (2011) state, “Validity is a matter of degree (based on the extent to which an evidence-based argument justifies the use of an assessment for a specific purpose).” (pg. 1)

11. Recommendation from NBPTS Task Force (Linn et al., 2011) • Recommendation 2: Employ measures of student learning explicitly aligned with the elements of curriculum for which the teachers are responsible. This recommendation emphasizes the importance of ensuring that teachers are evaluated for what they are teaching.

12. Measuring teachers’ contributions to student learning growth: A summary of current models

13. School-wide VAM illustration

14. Differentiating among teachers • “It is nearly impossible to discover and act on performance differences among teachers when documented records show them all to be the same.” (Glazerman et al., 2011, pg 1)

15. What assessments are states using for the non-tested subjects and grades? • Existing measures • Curriculum-based assessments (come with packaged curriculum) • Classroom-based individual testing (DRA, DIBELS) • Formative assessments such as NWEA • Progress monitoring tools (for Response to Intervention) • National tests, certification tests (but these lack pre-tests) • Rigorous new measures created for subject/grade standards (by teachers, districts, SEAs, etc.) • The 4 Ps: Portfolios/products/performance/projects • Student learning objectives • Any measure that demonstrates students’ growth towards proficiency in appropriate standards

16. Tennessee approved assessments for non-tested subjects & grades

17. Who approves measures? • Teachers may select (and in some states create) measures but in all cases, there is an approval process for use of the measure for evaluation purposes • Approval process varies • Delaware: state must approve measures • New York: district must approve measures • Washington DC, Rhode Island, most other states: building principal approves measures

18. The 4 Ps (Projects, Performances, Products, Portfolios) • Yes, they can be used to demonstrate teachers’ contributions to student learning growth • Here’s the basic approach • Use a high-quality rubric to judge initial knowledge and skills required for mastery of the standard(s) • Use the same rubric to judge knowledge and skills at the end of a specific time period (unit, grading period, semester, year, etc.)

19. Collect evidence in a standardized way (to the extent possible) • Evidence of student learning growth • Locate or develop rubrics with explicit instructions and clear indicators of proficiency for each level of the rubric • Establish time for teachers to collectively examine student work and come to a consensus on performance at each level • Identify “anchor” papers or examples • Provide training for teachers to determine how and when assessments should be given, and how to record results in specific formats

20. Colorado Content Collaboratives • “…are P-12 educators from around the state coming together to identify and create high-quality assessments, which are aligned to the new Colorado Academic Standards and may be used in the context of Educator Effectiveness evaluations. The Collaboratives are a four-year initiative. The first year of work will focus on the reviewing and creation of fair, valid and reliable measures of student learning (see flow chart below). Those measures will then be piloted and peer reviewed for their utility in educator effectiveness evaluations.”http://www.cde.state.co.us/ContentCollaboratives/index.asp

21. Washington DC IMPACT:Instructions for teachers in non-tested subjects/grades “In the fall, you will meet with your administrator to decide which assessment(s) you will use to evaluate your students’ achievement. If you are using multiple assessments, you will decide how to weight them. Finally, you will also decide on your specific student learning targets for the year. Please note that your administrator must approve your choice of assessments, the weights you assign to them, and your achievement targets. Please also note that your administrator may choose to meet with groups of teachers from similar content areas rather than with each teacher individually.”

22. Rhode Island’s SLO language • “Student Learning Objectives are not set by educators in isolation; rather, they are developed by teams of administrators, grade-level teams or groups of content-alike teachers and, are aligned to district and school priorities, wherever possible.” (pg 12) From Rhode Island’s “Guide to Measures of Student Learning for Administrators and Teachers 2011-2012” http://www.ride.ri.gov/educatorquality/educatorevaluation/Docs/GuideSLO.pdf

23. Rhode Island student learning objectives: Administration & scoring • Administration & Scoring: • The teacher should explain how the evidence used to assess the objective will be collected and reviewed. The teacher should include detail about how assessments will be administered and scored. The teacher and evaluator should determine the most accurate, fair, and objective scoring process possible.

24. Rhode Island student learning objectives: Results • Results • At the end of the interval of instruction, the teacher should explain the results of all sources of evidence used to assess the objective. The results should be expressed numerically and in relation to the previously set targets. If any official score reports are available for the sources of evidence used (especially for common assessments) they should be submitted to the evaluator prior to the End-of-Year Conference.

25. Rhode Island student learning objectives: Scoring • Scoring • The evaluator should review all the available evidence related to Student Learning Objectives, noting the degree to which the objective was met on the form. Evaluators will informally rate each objective as Not Met, Met, or Exceeded. The evaluator may provide additional comments about the scoring. These informal ratings will serve as the basis for the holistic scoring. Using the Student Learning Objective scoring guidelines, evaluators will look at the whole body of evidence across all objectives and assign an overall Student Learning Objective rating.

26. Rhode Island student learning objectives: Rubric for final score

27. Questions to ask about each measure used • How will using this measure in the teacher evaluation system impact teaching and learning in classrooms and schools? • How will the use of this measure look different in low-capacity vs. high-capacity schools? • How will reporting on results from this measure be done (to provide actionable information to teachers, principals, schools, districts, teacher preparation programs, and the state)? • How will we know if this measure is working as we intended?

28. Resources and links • Colorado Content Collaborativeshttp://www.cde.state.co.us/ContentCollaboratives/index.asp • Indiana RISE http://www.riseindiana.org/ • New York State Evaluation http://engageny.org/administrators/ • Rhode Island Department of Education Teacher Evaluation – Student Learning Objectives http://www.ride.ri.gov/educatorquality/educatorevaluation/SLO.aspx • Tennessee Teacher Evaluation http://team-tn.org/ • Washington DC Impact Teacher Evaluation System http://www.dc.gov/DCPS/impact

29. References • Glazerman, S., D. Goldhaber, et al. (2011). Passing muster: Evaluating evaluation systems. Washington, DC, Brown Center on Education Policy at Brookings. • http://www.brookings.edu/reports/2010/1117_evaluating_teachers.aspx • Herman, J. L., Heritage, M., & Goldschmidt, P. (2011). Developing and selecting measures of student growth for use in teacher evaluation. Los Angeles, CA: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST). • http://www.aacompcenter.org/cs/aacc/view/rs/26719 • Linn, R., Bond, L., Darling-Hammond, L., Harris, D., Hess, F., & Shulman, L. (2011). Student learning, student achievement: How do teachers measure up? Arlington, VA: National Board for Professional Teaching Standards. • http://www.nbpts.org/index.cfm?t=downloader.cfm&id=1305 • Malta, C., and Williams, S. (January 27, 2010). Meaningful assessment in the music classroom. Presented at Missouri Music Educators Association Conference, Jefferson City, MO. • http://dese.mo.gov/divimprove/curriculum/fa/AssessmentintheMusicClassroom.pptx

30. Questions?

31. Laura Goe, Ph.D. 609-619-1648 lgoe@ets.org www.lauragoe.com https://twitter.com/GoeLaura National Comprehensive Center for Teacher Quality 1000 Thomas Jefferson Street, NWWashington, D.C. 20007 www.tqsource.org