Impact and Evaluation

1. Impact and Evaluation •  Impact of Foundations Programs - UWA & literature •  Implementation / Transfer of Learning to Practice •  Evaluation of programs

2. Limited studies • Generally single institution / program. •  Teaching and learning improves (Coffey & Gibbs, 2000; Hall, 1996) •  Positive teaching evaluations (Andresen, 1995; Nasr, Gillet & Booth, 1996) •  Long term positive effects (Giertz, 1996; Rust, 2000)

3.  Needs a work culture that encourages change and risk taking and values teaching. (Spafford Jacob & Goody, 2002) • Need for post-program contact between educational developer and participants and their peers. (Spafford Jacob & Goody, 2002) • Need to provide systematic induction to teaching as part of graduate education. (Schulman, 1995)

4. Foundations of University Teaching and Learning Development of high quality teaching & learning at the university. • develop a professional & effective approach to teaching based on critically reflective practice; • provide support and resources for participants in their development as tertiary teachers; • a forum to discuss issues of teaching & learning; • develop networks for the continuing • dialogue & support;

5. 2.5-day workshop • seven 2-hour follow-up sessions • online module • peer observation of teaching component. • Mainly new academics but also experienced & future academics

6. UWA – Foundations of University Teaching and Learning (Spafford Jacob & Goody, 2002) • surveyed seven foundations programs • 32 responded (40% response rate) • plus information from end-of-program feedback • 45% arts, humanities, social sciences • 55% science, engineering, agriculture • 50% had no prior ‘teacher training’ • majority were relatively inexperienced teachers

7. Outcomes • Positive outcomes – increased theoretical understanding and more teaching and learning skills and methods; • More experienced teachers benefited less; • Reflection is important – although some participants say too much reflection and not enough information; • Networks were established; • Need for focus on content and process relevant to individual’s teaching / discipline context.

8. Has participation in Foundations enhanced the quality of your teaching practice?

9. Have you been able to influence the quality of teaching in your school?

10. Have you been able to overcome the barriers to implementing new teaching practice?

11. Factors impeding the transfer of learning • Impact of organisational culture. • Low value placed on teaching (perceived & real). • Lack of school support for implementation of learning. • Not enough resources (time, financial, staff) to develop and implement new teaching and learning strategies due to increasing workload and content tyranny.

12. Relevance of content and activities for individual teaching context. • Colleagues resistance to change, lack of dialogue about teaching. • Uniqueness of each individual’s engagement with the program. • Student resistance to new teaching styles. • Physical structure of teaching venues (e.g. fixed chairs and tables).

13. I am rejoicing in having the freedom and control to be able to try some of the things that I learned in Foundations. Here there is less of an established culture of the “way things have always been done”. (Comments from a Foundations participate who has left the University and works in a newer institution)

14. Facilitating Transfer of Learning • Pay attention to work context that participants will return to. • Spend time in the program on transfer / implementation of learning as well as content. • Involve participants in group discussion to identify barriers to transfer. • Include post-activity follow-up as part of program. • Need for cooperation between schools and EDU. • Involve participants in program planning.

15. Evaluation Practices in Canada (Kreber 1997) • Most EDUs made some attempt to evaluate parts of programs but comprehensive evaluation was not common; • Workshops were most frequently evaluated; • Most only evaluated goals of the programs; • Few reported assessing program impact; • Assessment of possible changes in staff’s beliefs or student achievement were the least common evaluation practice; • Most based on only one source of data;

16. Most based on informal conversations with participants at end of program – based on likes & dislikes; • Systematic analysis of data gathered from evaluations was not common; • Experience in educational evaluation seemed to make a difference in how evaluation was approached; • Lack of time, money and staff most common reasons for lack of systematic evaluation; and • More than 65% of developers said systematic evaluation was part of their work.

17. Evaluation should address the following questions: • What is the intended impact or what are the goals? • Why evaluate? • When to evaluate? • Who evaluates?; • How to evaluate? • Is the actual impact the same as the intended impact and is the actual impact desirable?; • Who should receive the results of the evaluation? • What do we expect to gain from the evaluation and what will happen as a consequence? Kreber & Brook (2001)

18. Six possible elements to evaluate: • participants' perceptions / evaluations; • participants' beliefs about teaching & learning; • participants' teaching performance; • students' perceptions of staff's teaching performance; • students' learning; and • effects on the culture of the institution. Kreber & Brook (2001)

19. Aligning evaluation strategies with intervention strategies and level of impact [From: Kreber & Brook, 2001]

20. Strongest index of effectiveness is impact on students. Weakest index of effectiveness is participants’ self reports. (Levinson-Rose & Meges, 1981) • Goal-free evaluation: rather than focus on goals, focus on outcomes (Scriven,1972) • Learning is only partially dependent on teaching which makes evaluation of impact even more difficult.