Student Evaluations: How Do They Influence Teacher Thinking And Behaviour?

1. Student Evaluations: How Do They Influence Teacher Thinking And Behaviour? In search of the evidence

2. Overview of Project Project team Back row: Stuart Terry (OP) Sarah Stein (OU) Lynley Deaker (OU) Front row: Jo Kennedy (OU) Dorothy Spiller (WU) Trudy Harris (WU)

3. Overview of Project • What are teachers’ perceptions and how do these perceptions influence teacher thinking and behaviour? • Does institutional use constrain teaching innovation? • Why do teachers undertake evaluations? • Do teachers use student evaluation feedback to improve teaching? • Do they engage in dialogue with students about evaluation feedback and their responses to it? • How can we use our findings to enhance the use of evaluations to improve teaching and learning?

4. Progress to Date By the 2010 AKO Colloquium: • Literature and environmental scan around evaluations • Preparation and writing of a literature review • Questionnaire sent out to academic staff Since the 2010 AKO Colloquium: • Quantitative data analysis of the survey data • Interviews conducted at each institution • Thematic analysis of the qualitative data

5. Quote of the Week “I’m a lapsed Irish Catholic with massive guilt and chips on my shoulders so whenever I get an evaluation faster than any computer software you have I will gravitate towards the negative comments about what I am doing and I will, my parents and grandparents have taught me very very well to ignore compliments, so any thing that is remotely complimentary I ignore and assume that it is just someone trying to be nice and just focus on the negative things, whether it’s a number or a comment. So my in my mind I think gosh I have really let someone down and them completely ignore the other 50 odd students who have good comments. That’s the other thing about how we receive them.  It’s a bit of paper, it’s a set of numbers, and we know stats are like a bikini, they show something but often hide the best bits!” (from an interview with an Otago Polytechnic staff member)

6. Initial Findings - Quantitative • Frequencies • Chi Square and cross tabulations • ANOVA • Factor analysis • Cluster analysis • Regression

7. There is a significant difference between Otago University and the other two institutions regarding teacher perceptions that student evaluation results constrain their willingness to try new teaching methods (P-value=0.000) 15. My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations/appraisals. A great deal 1 2 3 4 5 Not at all

8. These findings indicate that almost 10% of teaching staff are NOT prepared to take risks with teaching in case it impacts on their appraisals/evaluation results and consequently their promotion prospects. 15. My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations/appraisals. A great deal 1 2 3 4 5 Not at all

9. Initial Findings – Qualitative (Survey Comment Data) 15. My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations/appraisals. A great deal 1 2 3 4 5 Not at all 16. Please explain your answer to q15 (WU & OU) / 16. Any comments about question 15 (OP) Total of 595 comments for q16 (56% of respondents made comments)

11. Q16 - Example Quotes Code Ref1 - constrained – absolutely - “Absolutely! If my career advancement will suffer due to lower evaluations; why would I take a chance on innovative; demanding; or difficult course material?????” (OU) Code Ref2 - constrained - cautious about evaluations - “There appears to be greater monitoring teaching appraisals these days and I feel greater pressure to get good appraisal results than try new teaching approaches.” (WU) Code Ref3 - not constrained – absolutely - “No, I think that by being creative in the classroom and making changes that students actually respond to this and like it - I would not hold back” (OP) “I don't really care about getting sacked or a pay rise; if I was in the job for the money I would do a different job and earn 2* as much!” (OU) Code Ref4 - not constrained - need evaluations to learn from them - “Appraisals are good servants but bad masters. The point is that they should benefit the students, not (or only secondarily) me.” (WU)

12. Next steps Completion of the coding of the qualitative data from the survey Thematic analysis of interviews to be completed at all institutions Combination of the qualitative and quantitative data to develop a fuller understanding of teacher thinking Comparison of findings with literature Writing of the AKO research and summary reports

13. Are There AnyQuestions?

14. Are There AnyQuestions?