1 / 29

Validity Issues in Measuring Teachers ’ Contributions to Student Learning

Validity Issues in Measuring Teachers ’ Contributions to Student Learning. Joan Herman. May 17, 2013 Association of Colorado Evaluators Boulder Colorado. Overview. Purpose Theories of Action Validity Argument What ’ s Missing? Consequences What now?. Acknowledgement.

mtieman
Download Presentation

Validity Issues in Measuring Teachers ’ Contributions to Student Learning

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Validity Issues in Measuring Teachers’ Contributions to Student Learning Joan Herman May 17, 2013 Association of Colorado Evaluators Boulder Colorado

  2. Overview • Purpose • Theories of Action • Validity Argument • What’s Missing? • Consequences • What now?

  3. Acknowledgement Developing and Selecting Assessments of Student Growth for Use in Teacher Evaluation Systems Joan L. Herman, Margaret Heritage, and Pete Goldschmidt

  4. Our Focus • Sophisticated statistical models proposed to estimate the relative value individual teachers add to students’ performance • Little attention paid to the quality of the student assessments used to estimate student growth • New addition: what about purpose? Theory of action for getting there?

  5. What is the purpose of Colorado’s Teacher Evaluation System? • Your priority? • Your district’s priority?

  6. What’s Your Vote? • Evaluation for high stakes decisions: tenure, retention, reward/punishment • Evaluation to support improvement of teaching and learning • Evaluation to support professional practice, professional standing

  7. What is theory of action for getting to that purpose?

  8. Quality of Measures Matters • Estimating student learning (growth) typically requires at least two assessments of student learning • Carefully designed and validated to provide trustworthy evidence for evaluating teacher impact on learning • Solid criterion for evaluating evidence relative to teacher quality/effectiveness

  9. VALIDITY ARGUMENT JUSTIFYING STUDENT MEASURES

  10. Validity • Validity is overarching concept that defines quality in educational measurement • Evidence: measures what it is intended to measure and • Evidence: provides sound evidence for specific decision-making purposes. • Validation involves evaluating or justifying a specific interpretation(s) or use(s) of the scores

  11. Our Validity Framework • Establishes the basic argument that justifies use of student learning (growth) measures in teacher evaluation • Lays out the essential claims within the argument that need to be justified • Suggests sources of evidence for substantiating the claims • Uses accumulated evidence to evaluate and improve score validity

  12. VALIDITY ARGUMENT AND PROPOSITIONS

  13. Propositions to Justify Use Student Assessment

  14. CLAIMS AND EVIDENCE

  15. Propositions and Claims to the Validity Evaluation

  16. Propositions and Claims to the Validity Evaluation

  17. Propositions and Claims to the Validity Evaluation

  18. Propositions and Claims to the Validity Evaluation

  19. Propositions and Claims to the Validity Evaluation

  20. Propositions and Claims to the Validity Evaluation

  21. ACCUMULATED EVIDENCE

  22. Validity: A framework for evidence-based justification and improvement • Assessment validity is a matter of degree • Framework implies long term agenda • Reciprocal process involving experts in content and teaching, special populations, assessment and measurement • Assessment and validity evidence can always be improved

  23. Obviously….. • A single assessment cannot adequately capture the multi-faceted domain of teacher effectiveness • Multiple measures are essential

  24. What’s Missing? • Teachers’ influence on student creativity, motivation, learning to learn • Consequences: Intended and unintended

  25. But What About Validity for Achieving Intended Consequences? • If the goal is improving teaching and learning, are we missing core components? • If the goal is improving teacher practice, what are other necessary components of the argument?

  26. Theories of Action Supporting On-going Improvement • What is nature of tools and processes that are likely to yield intended consequences? • What are possible unintended, negative consequences? • Can we build, monitor, and refine systems that support the former and discourage the latter? • Can we use evaluation to improve our systems?

  27. Moving Forward • Start by being clear on learning expectations • Ensure assessments developed or selected are aligned with significant, deeper learning goals • Develop theories of action to get you where you want to go: the tools and the processes • Collect and analyze evidence of theory propositions • Develop long-term agenda to improve the quality of assessment tools and processes and positive consequences

  28. For more information see: CRESST Report #823 On the Road to Assessing Deeper Learning: The Status of Smarter Balanced and PARCC Assessment Consortia herman@cse.ucla.edu

More Related