EXPLORING INTER-RATER RELIABILITY

1. EXPLORING INTER-RATER RELIABILITY Professional Learning Team Conference June 2014

2. Desired Outcomes: Each participant will leave the session knowledgeable about inter-rater reliability What it is Why it’s important, and How to use evidence to consistently and accurately rate teachers’ performance

3. Goals for Inter-Rater Reliability 2014-2015 • Demonstrate familiarity with district rubric • Describe proficient practices based on district rubric descriptors • Provide feedback to educators based on evidence collection • Calibration of evaluators

4. GOALS: PARTICIPANTS WILL: • Establish a common vocabulary around inter-rater reliability • Be able to explain levels of performance with look-fors • Gain a deeper understanding of proficient level of performance aligned to standards

5. CHINESE PROVERB: • What I hear, I forget • 5% retention after 24 hours • What I see, I remember • 30% retention rate if use of audio-visual and demonstration • What I do, I understand • 80% retention rate with discussion and • practice by engagement

6. NORMS Share experiences to enrich others Ask questions Learn by doing Set aside any preconceived notions about evaluating educators Apply your own work

7. LEARNING TARGET: Participants will become familiar with look fors of effective practice and be able to use that knowledge to assess performance in three focused areas: • Student Engagement • Questions, Prompts & Discussion • Assessment

8. EVALUATION: NATIONAL FOCUS ON TEACHER EFFECTIVENESS AND EVALUATION However, most schools fail to evaluate teachers in a meaningful way, with 94% of teachers receiving the highest ratings. Teacher evaluations in five urban school districts, based on data taken from a report by The New Teacher Project: http://widgeteffect.org/downloads/TheWidgetEffect.pdf. EFFECTIVENESS: Teacher effectiveness is the single most important school-related factor affecting student learning. Teacher evaluations in five urban school districts, based on data taken from a report by The New Teacher Project: http://widgeteffect.org/downloads/TheWidgetEffect.pdf

9. INTER-RATER RELIABILITY: a systemic approach to calibrate observations for consistent and fair professional practice ratings

10. Learning Experience: At your table define and describe Student Engagement Capture your collective thoughts on chart paper

11. Educational author and former teacher, Michael Schmoker shares in his book, Results Now, a study that found of 1,500 classrooms visited, 85 percent of them had engaged less than 50 percent of the students In other words, only 15 percent of the classrooms had more than half of the class at least paying attention to the lesson

12. WHAT DO "ENGAGED" STUDENTS LOOK LIKE? You will see students... • Paying attention (alert, tracking with their eyes) • Taking notes (particularly Cornell) • Listening (as opposed to chatting, or sleeping) • Asking questions (content related, or in a game) • Responding to questions (whole group, small group, four corners, Socratic Seminar) • Following requests (participating, Total Physical Reponses (TPR), storytelling, Simon Says) • Reacting (laughing, crying, shouting, etc.) • Reading critically (with pen in hand • Interacting with other students Michael Schmoker

13. ENGAGING STUDENTS IN LEARNING • Activities aligned with the goals of the lesson • Student enthusiasm, interest, thinking, problem-solving… • Learning tasks that require high-level student thinking and are aligned with lesson objectives • Students are highly motivated to work on all tasks and are persistent even when the tasks are challenging • Students actively "working," rather than watching while the teacher "works“ • Suitable pacing of the lesson: neither dragging nor rushed, with time for closure and student reflection

14. STUDENT ENGAGEMENT When determining the level of performance for Engaging Students in Learning, consider the quality of activities, assignments, and resources the instructor uses to pull students into the lesson It is also important to understand how the lesson uses student grouping and pacingwhen determining the level of performance for this component

15. Learning Experience: At your table define and describe Assessment Capture your collective thoughts on chart paper

16. THE PURPOSE OF  CLASSROOM ASSESSMENT is to support student learning and to communicate that learning to others

17. ASSESSMENT: In order to support student learning classroom assessment needs to: • involve students deeply in the assessment process • provide specific, descriptive feedback during the learning, and • include evaluative feedback as required to communicate and report progress over time

18. ASSESSMENT: To ensure success of all: • students need to know what they already know • know what needs to be learned and • know what success looks like Students also need to learn how to guide their own learningthrough being involved in setting and using criteria, giving themselves feedback for learning (self-assessment), setting goals, collecting evidence and communicating that evidence of learning to others

19. INDICATORS THAT MAY BE USED AS A GUIDE FOR LOOKING FOR EVIDENCE OF ASSESSMENT • The teacher pays close attention to evidence of student understanding • The teacher poses questions specifically created to elicit evidence of student understanding • The teacher circulates to monitor student learning and to offer feedback • Students assesses their own work against established criteria • The teacher adjusts instruction in response to evidence of student understanding (or lack of it)

20. BREAK

21. Learning Experience: Work with your District partners to identify the elements in your rubric that focus on • Student Engagement • Question, Prompts & Discussion • Assessment

22. Learning Experience: While observing the video, collect evidence that you will later use to rate the teacher’s performance using your district rubric

23. Remember… You will use your district’s rubric to rate the teacher’s performance in the following areas: Student Engagement Question, Prompts & Discussion Assessment

24. Math Lesson: CC Standard 7.G.6 https://www.teachingchannel.org/videos/surface-area-lesson# Video

25. LUNCH

26. In just a moment you will be asked to review the data you’ve collected, cluster and code it for easy interpretation, and rate the teacher’s performance using your district’s rubric

27. DATA Evidence DATA Interpretation DATA Judgment

28. WHAT IT LOOKS LIKE TOCLUSTER AND CODE

29. ANOTHER METHOD OF EXAMINING YOUR EVIDENCE

30. ANOTHER METHOD OF EXAMINING YOUR EVIDENCE

31. With an elbow partner, use the verbatim transcript of the teacher’s observation to identify and organize evidence to accurately and fairly rate the teacher’s performance in the areas of • Student engagement • Questions, Prompts and Discussion • Assessment

32. Look at your evidence for each component.  Where does the preponderance of evidence align with levels of performance? • Match your data with the rubric performance level description • Assign a performance level

33. Use your clicker to rate this teacher’s performance using the numbers Proficient Unsatisfactory • Basic Distinguish

34. TESTIDENTIFY YOUR FAVORITE CANDY BAR • Snickers • Milky Way • Kit Kat • Trix :01

35. STUDENT ENGAGEMENT • Unsatisfactory • Basic • Proficient • Distinguish :01

36. DISCUSS

37. QUESTIONS, PROMPTS & DISCUSSION • Unsatisfactory • Basic • Proficient • Distinguish :01

38. DISCUSS

39. ASSESSMENT • Unsatisfactory • Basic • Proficient • Distinguish :01

40. DISCUSS

41. ONE – TWO PUNCH • One thing you heard today that you want to remember • Two things you heard that you want to act on before the next PLT Conference John Doe 1234 Paradise Lane Anywhere, OR 97890

42. QUESTIONS

43. Thank you for your active participation and cooperation 