1 / 17

Inter-rater Agreement In Kansas

Inter-rater Agreement In Kansas. Summer Principals Academy July 22-24, 2014 Abilene, KS . What is Inter-Rater Agreement?. The degree 2 raters using the same scale give the same rating in identical situations. Consistent understanding Consistent agreement Consistent process

tricia
Download Presentation

Inter-rater Agreement In Kansas

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Inter-rater Agreement In Kansas Summer Principals Academy July 22-24, 2014 Abilene, KS

  2. What is Inter-Rater Agreement? The degree 2 raters using the same scale give the same rating in identical situations. • Consistent understanding • Consistent agreement • Consistent process • Consistent credibility

  3. Where is Inter-rater Agreement Useful?? Inter-Rater Agreement is useful in all areas of administrative practice including: • Kansas Educator Evaluations • Kansas Accreditation • Personnel Decisions • Family Engagement Programs • Budget Management

  4. What are the Goals of Inter-rater Agreement? • The Inter-Rater Agreement Statistical Goal is 75% - agreement with no rating more than one level apart. • The Goal is to have an administrative Common Frame of Reference for your district such as: • Agreement with no rating more than one level apart. • Consistent understanding • Consistent agreement • Consistent process • Consistent credibility

  5. How is Inter-rater Agreement Helpful?? • What is a Common Understanding? • How does Inter-rater Agreement help administrators?? • Facilitates Quality • Provide Consistent Messaging • Impacts Increased Student Growth • Enhances Trust • Facilitates Quality Feedback • Legally Defensible • Assists Accurate Data Collection • Allows for Increase Cooperation over Time • Passes Public Scrutiny

  6. District-wide Agreement about External Criteria • Agreed Upon External Criteria for: • Observations • Artifacts • Units of Study • Processes • Program Development • Time Limits • Walk-Throughs • Protocols

  7. Pragmatics/ConsiderationsPotential Landmines • Discuss with an Elbow Partner Possible Issues that may cause Rater Error • Leniency: Benefit of the Doubt • Central Tendency: Safety in the Middle • Comparing One with Another: Stick to the Rubric • Relying on Gut too Much: Stick to the Rubric • Evaluating Absent Evidence: Stick to What is Observed

  8. How to Train Administrators in using Inter-rater Agreement • Inter-Rater Agreement Training – Things to Consider: • Synchronous and Asynchronous • Experience of the Participants • How to Ensure Fidelity • Use of Video Examples • Use of Case Studies • Bias Exists • Single versus Multiple Observations • Professional Judgment • Degree of Familiarity

  9. Getting Started in YOUR District • Getting Your Bearings • Developing a Guidebook • Developing a Training Protocol

  10. I. Getting Your BearingsPilot with Administrative Team • Pilot Approach: Start with Safe Example: • Use a video from YouTube • Assess Specific Area of Teaching • Student Engagement • Teacher Activity • Classroom Environment • Review current issue/goal/vision/needs • Review current practices/instrumentation • Rubrics • Protocol • Training

  11. II. Developing a Guidebook • Paper and Electronic: Synchronous and Asynchronous • EXCELLENT Administrative Council Endeavor – Ultimately to be used during Professional Learning Times with the Staffs • Pieces and Parts • Terms/Definitions: Frame of Reference/Common Understanding • “Demonstrates” • “Frequently” • “Usually”

  12. II. Developing a Guidebook • Pieces and Parts (continued) • Examples and Rubrics • Areas to Observe • Establishes/maintains a culture of safety, respect and rapport • Knows how to make content accessible to all learners • Plans standards-based instruction and formative assessments • Uses a variety of instructional strategies to meet needs • Deepens understanding through critical thinking, problem-solving and student reflection • Analyzes student performance and provides feedback • Collaborates to support student learning

  13. II. Developing a Guidebook • Pieces and Parts (continued) • Exemplars: Video and Artifacts • Master-Coded Videos: Leads to high level of agreement • Example Artifacts • Classroom Rules • Lesson Plans • Parent Note Home • Lesson Plan for Adults: Andragogy

  14. III. Developing a Training Protocol • Step-by-Step Procedure • Initial Training: Goals/Terms • Practice Interpreting Rubrics • Differentiate between Different Performance Levels with Examples • Calibration of the Players • Video and Artifacts • Review of the System • Adjustments (If needed) • Implementation • Evaluate/Reflect/Adjust • Re-Calibrate…

  15. Take Aways – To Do List • Start Slowly • Start with “Engagement” with a Small Group • Get Agreement • Gain Confidence Over Time • Write and Refine in Teams

  16. Final Thoughts • Start with the End in Mind • Develop Common Understandings • Use Groups to Ensure Buy-In • Build Measurable Steps to Reach Each Goal

  17. Contact Information Teacher Licensure and Accreditation Team DirectorScott Myers785-296-8010smyers@ksde.org

More Related