Inter-Rater Reliability on Observation Practices

1. Inter-Rater Reliability on Observation Practices “Good observations provide good feedback which leads to support of instruction resulting in student growth” (Certification Council Teacher Evaluation, 2011).

2. Teachers are immune to feedback from a coach or administrator when they have different definitions of quality. The single most important thing that a school leader can do is reach agreement with the staff about quality. Enhancing RTI (ASCD, 2010) Fisher and Frey, 2010

3. Answer Now! Is there a shared definition of good teaching in your school/district? • We all agree on what excellent, good, mediocre, poor teaching looks like. • We agree on what good teaching looks like. • There are some disparities within the school or across the district. • There are many different opinions on what good teaching is.

4. A Shared Definition? • We all agree on what excellent, proficient, basic or does not meet looks like. • We agree on what good teaching looks like. • There are some disparities within the school or across district. • There are many different opinions on what good teaching is.

5. Goals for Inter-Rater Reliability3 Day Training 2011-2012 • To know how inter-rater reliability improves instruction definition levels of performance • To apply consistent, fair ratings across classrooms and schools using standards based outcomes with criteria. • Look fors/critical attributes • Rate proficiency • To understand how using rubrics can be a systemic approach to support professional development and a culture for learning. • Professional conversations/conferences

6. AM AGENDA • What is Inter-rater Reliability? • Why do we need it? • What is effective teaching? • Why is evidence important to inter-rater reliability? • What are the look-fors/attributes for one component of instruction? • How would we rate this teacher: Practicing! • Putting it all together: Video Observation • Next steps…

7. PM AgendaAdministrative Evaluation Processand Teacher/Principal Inter-rater Reliability • Research on Improving the Instructional Process • Background information on the rubrics • Components of the supervision/evaluation process • Challenges and support • Inter-rater reliability for supervisors of administrators

8. Quotes Pick a quote from the center of the table. Read the quote to yourself and be prepared to share its implications for our work as instructional leaders. When your table group is ready, start with the person with a birthday closest to today. Read your quote and share its implications.

9. McKinsey & Co. (2007) “And the evidence is indisputable: you can’t improve student learning without improving instruction” (p.11).

10. Marzano (2007), p. 1 Among elements such as a well-articulated curriculum and a safe and orderly environment, the one factor that surfaced as the single most influential component of an effective school is the individual teachers within that school.

11. Danielson (2009) .. When we consider the importance of ongoing teacher learning for the success of schools, then it is essential to consider how best to promote that learning (p. 3). Professional competence also, at times, requires maintaining the focus of all educators on the important principles of teaching and learning, and not permitting them to be distracted by the shifting winds of fashion (p.19)

12. Louis and Miles (1990) found that no other type of classroom change causes more conflict than attempting to change the instructional practices of teachers. • Researching and valuing instruction is one thing, implementing and collectively sustaining it systemically over time is another.

13. Leithwood, Day, Sammons, Harris, and Hopkins (2007) In a ten year study of leadership in three countries, the researchers found that the best predictor of student achievement is improving the instructional practices of teachers. The second most powerful factor is the leadership practice of the principal in facilitating student and teacher learning.

14. McKinsey & Co. (2007) “..different approaches proven to be effective all begin with what it takes to improve the quality of a single teacher, and then develop the systems to create these conditions for all teachers” (p. 32).

15. What’s happening in your school? How are teachers improving their practice? What are you doing to facilitate improvement of practice?

16. Things matter most: McKinsey & Co. Report (2007)

17. Inter-Rater Reliability

18. RESEARCH FINDINGSCINCINNATI’S USE OF A RUBRIC (2010) • Teachers have substantial effect on student achievement • Correlation between evaluation using Danielson’s rubric and student achievement • Evaluation using rubric found • Unsatisfactory and basic: students had lower gains than expected (based on test scores from previous years) • Proficient: students made expected gains • Distinguished/Exemplary: students made positive gains • Discriminating between basic and proficient is most difficult

19. Findings continued… Training was needed: • On collecting evidence from classroom observations and artifacts. • A deeper understanding of the knowledge and skills in each rubric • On conducting reflective conferences for growth. A study of 6000 teachers (MTA) found only 34% of teachers had feedback suggesting strategies or professional development for feedback. A study with New Teacher Project found 73 % of most recent observations/evaluations did not identify any areas for growth.

20. What do Effective Teachers DO!!!!!

21. A Memorable Teacher Imagine you are in a classroom of a highly effective teacher: What would you see the teacher doing, hear the teacher saying? What would you see the students saying or doing?

22. Effective teachers…. • Put a greater emphasis on meaning vs. memorization • Had coherent content and clarity of objectives • Constantly engaged the students • Assessed students frequently • Had well managed and 0rganized classrooms • Used allocated time for instruction • Used a variety of materials • Had higher expectations for students • Were enthusiastic • Exhibited support, fairness, and respect Adapted from Good (2010)

23. Effective teachers…. • Put a greater emphasis on meaning vs. memorization • Had coherent content and clarity of objectives • Constantly engaged the students • Assessed students frequently • Had well managed and organized classrooms • Used allocated time for instruction • Used a variety of materials • Had higher expectations for students • Were enthusiastic • Exhibited support, fairness, and respect Adapted from Good (2010)

25. Levels of Performance

26. Instruction Component of Rubric

27. Watch Video Collect data you could use to rate this teacher’s performance in 3C: Engaging Students. Compare your data with an elbow partner. Confirm that you have data (see/hear/count) and not opinion. Compare your data to the rubric.

28. Rating Effective Instruction Using your clicker, rate this teachers’ performance based on your rubric using the numbers 4, 3, 2, or 1.

29. Rate this teacher’s performance based on your rubric 1-4. • does not meet standard • basic • proficient • exemplary

30. Effective Rubrics are….. • Definitions of effective teaching • Based on research • Human Development Theories • Principles of Learning • Learning Theories and Brain Research • Research based • Using rubrics improves teacher practice and student achievement

31. Evidence vs.. Opinion

32. Evidence Data Data Interpretation Data Judgment

33. EXAMPLES of TYPES of EVIDENCE! • Verbatim scripting of teacher or student comments: “Bring your white boards, markers and erasers to the carpet and sit on your square.” • Non-evaluative statements of observed teacher or student behavior: Teacher presented the content from the front of room. • Numeric information about time, student participation, resource use, etc.: Two groups started on the assigned project immediately, one group talked for five minutes before starting. • An observed aspect of the environment: Desks were arranged in groups of four with room to walk between each group.

34. Evidence Reminder • Factual Information • What the teacher says/does • What the student says/does • Numbers • Time • Artifacts

36. Review Original Rating Look at specific statements for each level of performance for the component “engagement”. Review evidence collected on what teacher/student said or did Match your data with the best description for level of performance. Rate the teacher

37. Teaching…. cannot be considered proficient or distinguished if students are not thinking or doing the learning themselves.

38. How Does a Teacher Engage?Becoming More Specific • Developing a common definition of ‘engaging students’. • Why engage students? • How often should we engage students?

39. Centerpiece of Each Rubric • Cognitive Engagement • Constructivist Learning Discuss a definition of cognitive engagement. What would it look like in the classroom at the proficient level? Share an example.

40. Centerpiece: Engaging Students • Teaching is a behavior that increases the probability of learning: actively and mindfully process and practice what they are learning • Constructivist Learning • Learning is done by the learner • The active nature of learning and how to promote it • Proficient practice has evidence of learning experiences designed to facilitate students’ construction of knowledge. • Cognitive Engagement • Proficient practice has evidence of students engaged cognitively. “To meet the needs of our citizens of our future, nothing less will do.” (Danielson, 2007, p. 17)

41. Engagement at Proficient Level

42. Beyond Definitions: Critical Attributes

43. Beyond Definitions: Critical Attributes

44. More ExamplesDo these meet the criteria? Using your response cards, agree or disagree, show your response to this statement: Free trade will have a long term benefit for Canada, US, and Mexico. Be prepared to share your answer. Who can tell me how to factor an equation? • Write one question you might find on a test about today’s lesson. Trade questions with your neighbor and write a response. Then trade again and ‘grade’ the answer. • Here are some questions about the main character in the story. I’ll read one and call on volunteers to answer them.

45. Engaging Students/Activities and Assignments A B Mandatory/all Simultaneous • Discretionary • One by one

46. Engaging Students/Activities and Assignments Not engaging Engaging Mandatory/all Simultaneous 3. Throughout the lesson 2-3 min. for primary 5-7 min inter./ 7- 10 hs/adult 4. Processing the learning Does the engagement activity focus students on the learning? • Discretionary 2. One by one • Long stretches of telling without processing. More than 7 minutes

47. Observing Classroom practice Watch the video and collect data on examples of engagement using these critical attributes: Were all students engaged simultaneously? Was the engagement t/o the lesson? (based on level) Were the engagement strategies focused on the learning?

48. Using your evidence…. Compare examples with elbow partners… Reach consensus on your examples……………. Rate this teacher’s level of proficiency using the attributes and your rubric as a 4, 3, 2, or 1.

49. Rate this teacher’s performance based on your rubric 1-4. • does not meet standard • basic • proficient • exemplary

50. Putting it all Together: Another Video Watch video Collect data on what teacher/student says or does Analyze data using criteria/attributes/look-fors Rate the teacher