Student Evaluations: How Do They Influence Teacher Thinking And Behaviour?

1. Student Evaluations: How Do They Influence Teacher Thinking And Behaviour? In search of the evidence

2. Project Team Back row: Stuart Terry (OP) Sarah Stein (OU) Lynley Deaker (OU) Front row: Jo Kennedy (OU) Dorothy Spiller (WU) Trudy Harris (WU)

3. Research Question How do the current evaluation processes and practices influence teachers’ thinking and behaviours in relation to student learning at all stages of the teaching and learning cycle?

4. Overview of Project • What are teachers’ perceptions and how do these perceptions influence teacher thinking and behaviour? • Does institutional use constrain teaching innovation? • Why do teachers undertake evaluations? • Do teachers use student evaluation feedback to improve teaching? • Do they engage in dialogue with students about evaluation feedback and their responses to it? • How can we use our findings to enhance the use of evaluations to improve teaching and learning?

5. Progress to Date By the 2010 AKO Colloquium: • Literature and environmental scan around evaluations • Preparation and writing of a literature review • Questionnaire sent out to academic staff (2445 – 45% response rate). Since the 2010 AKO Colloquium: • Quantitative & qualitative data analysis of the survey data • Interviews conducted at each institution • Thematic analysis of the qualitative data (survey & interview data)

6. Quote of the Week “I’m a lapsed Irish Catholic with massive guilt and chips on my shoulders so whenever I get an evaluation faster than any computer software you have I will gravitate towards the negative comments about what I am doing and I will, my parents and grandparents have taught me very very well to ignore compliments, so any thing that is remotely complimentary I ignore and assume that it is just someone trying to be nice and just focus on the negative things, whether it’s a number or a comment. So my in my mind I think gosh I have really let someone down and them completely ignore the other 50 odd students who have good comments. That’s the other thing about how we receive them.  It’s a bit of paper, it’s a set of numbers, and we know stats are like a bikini, they show something but often hide the best bits!” (from an interview with an Otago Polytechnic staff member)

7. Initial Findings - Quantitative • Frequencies • Chi Square and cross tabulations • ANOVA • Factor analysis • Cluster analysis • Regression

8. Teachers at Otago University are more constrained by the possible negative effects on their students evaluations when looking at trying new teaching approaches than the other two intuitions.(P<0.01). 15. My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations/appraisals. A great deal 1 2 3 4 5 Not at all

9. 15. My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations/appraisals. A great deal 1 2 3 4 5 Not at all

10. Additional Quantitative Analysis – questions 15 & 26 15. My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations/appraisals. A great deal 1 2 3 4 5 Not at all 26. How long have you been teaching in the tertiary sector? 0-5 years 6-10 years 11-15 years 16-20 years 21 years or more The only statistically significant difference found was at Otago University where the 11-15 years teaching experience group identified as being more constrained than the other groups (p<0.01)

11. Initial Findings – Qualitative (Survey Comment Data) 15. My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations/appraisals. A great deal 1 2 3 4 5 Not at all 16. Please explain your answer to q15 (WU & OU) / 16. Any comments about question 15 (OP) Total of 595 comments for q16 (56% of respondents made comments)

13. Q16 - Example Quotes Code Ref1 - constrained – absolutely - “Absolutely! If my career advancement will suffer due to lower evaluations; why would I take a chance on innovative; demanding; or difficult course material?????” (OU) Code Ref2 - constrained - cautious about evaluations - “There appears to be greater monitoring teaching appraisals these days and I feel greater pressure to get good appraisal results than try new teaching approaches.” (WU) Code Ref3 - not constrained – absolutely - “No, I think that by being creative in the classroom and making changes that students actually respond to this and like it - I would not hold back” (OP) “I don't really care about getting sacked or a pay rise; if I was in the job for the money I would do a different job and earn 2* as much!” (OU) Code Ref4 - not constrained - need evaluations to learn from them - “Appraisals are good servants but bad masters. The point is that they should benefit the students, not (or only secondarily) me.” (WU)

14. Next steps Completion of the coding of the qualitative data from the survey Thematic analysis of interviews to be completed at all institutions Combination of the qualitative and quantitative data to develop a fuller understanding of teacher thinking Comparison of findings with literature Writing of the AKO research and summary reports

15. Are There AnyQuestions?

16. Are There AnyQuestions?