Developing Excellence across the Australian Higher Education Sector monash.edu/pats

1. Developing Excellence across the Australian Higher Education Sectormonash.edu/pats Associate Professor Angela Carbone Office of the Pro Vice-Chancellor (Learning and Teaching) Monash University Sponsored by CADAD

2. Session overview National Senior Tecahing Fellowship | Flinders University

3. Background: My journey with PATS 2011-2012 2012-current 2008-2010 2010-2011 CADAD National Senior Tecahing Fellowship | Flinders University

4. USING PATSTO DEVELOPGOOD PRACTICE Peer Assisted Teaching Scheme National Senior Tecahing Fellowship | Flinders University

5. What is Peer Assisted Teaching Scheme (PATS)? PATS is a scheme in which academics within a faculty are partnered and follow an informal process to discuss strategies to improve unit quality and develop educational innovations. MENTOR + MENTEE RECIPROCAL PARTNERSHIP • Aims • improve student satisfaction with units • improve the quality of teaching • build leadership capacity amongst teachers National Senior Tecahing Fellowship | Flinders University

6. Theoretical underpinning • Applies the principles of social-cultural theory (Vygotsky 1978) and Lave’s situated learning theory (1988) • Benefits of peer mentoring (Gratch1998; Boud 2001; Topping 2001; Arendale 2004; Hall et al. 2005) • Four lenses to view teaching practice (Brookfield 1995) • (Carbone 2012, HERD in press) Diagram sourced from Dr Maureen Bell’s Peer Review presentation as part of the PATS workshops National Senior Tecahing Fellowship | Flinders University

7. PATS Guides and Workbook • Complete Tasks • - Pre-semester (Tasks1-3) • During semester (Tasks 4-5) • Post semester (Tasks 6-7) • Explanation of tasks next ... monash.edu/pats National Senior Tecahing Fellowship | Flinders University

8. PATS Process Workshops: 1. Interactive Lecturing 2. Planning Your Teaching 3. Peer Observation of Teaching • Deliverables: • Strategy Plan • Summary of feedback • Peer Observation of Teaching • Critical Reflection National Senior Tecahing Fellowship | Flinders University

9. PATS WORKBOOKACTIVITIES Peer Assisted Teaching Scheme National Senior Tecahing Fellowship | Flinders University

10. PATS Tasks What you need to do… prior to semester starting during semester after semester is over National Senior Tecahing Fellowship | Flinders University

11. National Senior Tecahing Fellowship | Flinders University

12. Task 1 – Meet and greet • Introduce yourself - tell us a little about yourself, for example: • Your name and the unit you are teaching • What do you do? (work, study, parent) • Why are you doing PATS? . • Plan your meeting dates for the rest of the semester – following each meeting, write a one sentence summary of what was discussed • Receive coffee vouchers from your PATS co-ordinator • Organise to view previous unit evaluation qualitative comments and teaching evaluations National Senior Tecahing Fellowship | Flinders University

13. Task 2 – Break down the barriers • Academics are faced with many barriers which can often cause units to under perform — these are road blocks. • There are three types of road blocks: • 1. Internal barriers • 2. External barriers within your control and • 3. External barriers outside of your control. • List a barrier you face in teaching your unit. • Discuss barriers with your partner • Suggest a possible solution. Barriers STRATEGIES National Senior Tecahing Fellowship | Flinders University

14. Task 3 – Set goals for improvement • Identify 2-4 goals for improvement • What was the driver for your goals? • Produce a strategy plan to address the identified areas • What data will you gather to evidence whether your goal(s) have been met National Senior Tecahing Fellowship | Flinders University

15. National Senior Tecahing Fellowship | Flinders University

16. Task 4 – Gather informal student feedback • Informal Student Evaluation of Teaching and Unit • Informal student feedback helps the lecturer create a collaborative learning culture and improves both the teaching by the lecturer and the learning by the student National Senior Tecahing Fellowship | Flinders University

17. Task 5 – Peer review of teaching 1. BRIEFING SESSION 2. OBSERVATION SESSION 4. PRODUCTION OF JOINT STATEMENT 3. POST-OBSERVATION SESSION Peer review instruments (Templates available: HERDSA Guide, Macquarie University, Monash University, Griffith University) National Senior Tecahing Fellowship | Flinders University

18. Task 6 – Critical reflection • Reflect on your unit and teaching evaluations • Did you achieve your goals? • Use instrument to identify areas that are meeting students’ expectations and needs, as well as areas that can be improved National Senior Tecahing Fellowship | Flinders University

19. Task 7 – Performance planning and strategies • Discuss with partner about completing your teaching improvement, educational standing or education leadership in your Personal Development Plan • Enter improvements into your Personal Development Plan • Reference your teaching and unit results • Consider a promotion application • Create a teaching portfolio which contains all of your achievements (eg. awards, certificates, commendations, emails from peers or students) • Consider a publication that reports on your innovation/ change in practice/ improvement National Senior Tecahing Fellowship | Flinders University

20. Opportunities to explore • About the process – theoretical framework – evaluation • Understanding student concerns – developed course quality attributes • Understanding how to embed innovations in higher education institutions • HERSDA showcases • Barriers academics face: motivation, knowledge, skills, and practicalities • Goal setting: need for a Smarter PATS • PATS: Creating collegial places to close the feedback loop • Creating a smarter observation instrument: Focusing PATS peers on Development Goals National Senior Tecahing Fellowship | Flinders University

21. Opportunities to publish Avenues to publish • Journal of Teacher Education • Teachers and Teaching • Higher Education Research Development • British Journal of Educational Technology • Australasian Journal of Educational Technology • Higher Education • Assessment and Evaluation in HE • Studies in Higher Education National Senior Tecahing Fellowship | Flinders University

22. PATS Senior Management decision points Who will participate? All staff – new staff – sessional staff – tutors – staff with low UEs – ECDF- other What will be the policy regarding participation? Optional – strongly encouraged – mandatory What form will the process take? Mentor/mentee – reciprocal partnership Who will act as mentors? Teachers with education focus – outstanding teachers – colleagues within same organisational unit – colleagues within same discipline- other How will program participation be supported? coffee vouchers – conditional/unconditional funding – time relief – credit for other academic development programs What type of reporting will take place? submission of workbook – changes to UE – other Who is notified when the scheme is complete? PDP supervisor- HoS- ADE - Dean –Director Education Quality What type of follow up will occur after completion of PATS? focus group – performance development review Carbone et al 2013 (under review SiHE) National Senior Tecahing Fellowship | Flinders University

23. OLT: Good PRACticeGuIDES Peer Assisted Teaching Scheme National Senior Tecahing Fellowship | Flinders University

24. OLT Good Practice & Final Reports OLT Good practice reports: • Blended learning • Technology-enhanced learning and teaching • Curriculum renewal; Assuring Graduate Outcomes; Work integrated learning Final Reports • Peer Review in online and blended learning environments • Peer Review of Teaching in Australian Higher Education • Supporting student peer assessment and review in large groupworkprojects • Curriculum development in studio teaching; (Volume 1) • Improving learning in undergraduate physics using integrated 'studio' environments to replace traditional lectures, laboratories and tutorials Articles • Flipped classroom National Senior Tecahing Fellowship | Flinders University

25. Exercise • Work with the Good Practice Report nearest to you, to answer the following: • What does the report tell us about the state of play? • What research opportunity does the report flag? • How could this report support what you are about to change? (10-15 mins) • Each team reports back to the group (10-15 mins) National Senior Tecahing Fellowship | Flinders University

26. SUCCESSFUL MENTORING Peer Assisted Teaching Scheme National Senior Tecahing Fellowship | Flinders University

27. What does the literature suggest about successful mentoring programs? • Clarity of purpose • Methods of matching mentor and mentee • Mentor training • Holding regular meetings to nurture mentor-peer relationship building • Contract of/clear expectations • Visible & tangible support from senior leadership • Evaluation National Senior Tecahing Fellowship | Flinders University

28. Mentee role • Regular meetings – face-to-face, Google Hangouts • Clear agenda • Give feedback • Gain feedback • Learn from your peers National Senior Tecahing Fellowship | Flinders University

29. PATSRESULTS Peer Assisted Teaching Scheme National Senior Tecahing Fellowship | Flinders University

30. PATS trials/results • Data gathered: • Changes in unit evaluation results • Surveys with ADEs, HoS • Focus group sessions with Mentors and Mentees • Results: • Positive changes in unit evaluation results/student satisfaction • Experiences of engaging academics PATS • Overall faculty rankings in unit evaluations • Advances in career progression • Better understanding of student concerns (course quality attributes) • Decision points for senior managers • Improvements in teaching practice – barriers, goals set National Senior Tecahing Fellowship | Flinders University

31. Positive changes to unit evaluation results Unit Evaluation Questions Item 1: The unit enabled me to achieve its learning objectives Item 2: I found the unit to be intellectually stimulating Item 3: The learning resources in this unit supported my studies Item 4: The feedback I received in this unit was helpful Item 5: Overall I was satisfied with the quality of this unit Two open-ended questions: What were the best aspects of the unit? What aspects of this unit are most in need of improvement? Table 1: PATS Pilot in FIT unit evaluation results 2008-2009 All of the units improved by at least 0.5. • In a 4 year period (2009-2012) • 12/15 units improved (+0.3 to +1.8) • 1/15 was a new unit scored 4.4, 64%rr • 2/15 decreased (-0.1 & -0.3) Carbone, Wong and Ceddia2011; Carbone, Ross & Ceddia2013 National Senior Tecahing Fellowship | Flinders University

32. Focus group sessions – PATS Experience reflection Relationship-building constructive accountability fostering discussions Learning and Teaching Effectiveness encouraging collaborative supportive Sharing ideas review & plan Interpretative guidance communication (Carbone 2011) National Senior Tecahing Fellowship | Flinders University

33. Opportunities and Challenges • Partnering process • Time commitment • Feeling stigmatised • Requires total dedication from both partners to be successful • Work balance research vs. teaching Challenges Opportunities • Building leadership/mentoring capacity • Coffee vouchers to build supportive, collegial relationship with colleagues within faculty • Places more priority on teaching can lead to improved teaching practice • Improving quality of higher education, better student experience • (Carbone, 2012) National Senior Tecahing Fellowship | Flinders University

34. Career Advances /Overall faculty performance • Mentees went on to become mentors • Several mentors nominated for a national award • Mentors granted promotion • Mentees and mentors publishing innovations: eg: “conversation sim” • Mentees and mentors submitting grant proposals • Mentors have applied for positions like Associate Deans (Learning and Teaching) Unsolicited Email Hi Ange, ... I just wanted to let you know that all your hard work has paid off. Cheers, Academic Staff member Faculty of Information Technology 12/5/2012 The Faculty was ranked fifth in the university – the highest ranking achieved in the last 5 years; Our unit evaluations have been improving constantly, and we have the lowest percentage of units now in the “needing improvement” category since 2009. Additionally, we have the highest proportion of units considered “outstanding” since 2009. We have the highest response rate (45%) across the University. This should mean our performance data is quite reliable. For many semesters a significant focus of our development efforts have been units that are, in university terminology, "in the red zone". Such units have been given additional support in the form of programs such as PATS. These efforts have paid off and together we have successfully managed to move many units out of this range.... National Senior Tecahing Fellowship | Flinders University

35. CADAD TrIAL Peer Assisted Teaching Scheme National Senior Tecahing Fellowship | Flinders University

36. Multi- institutional Trial Focus : on re-invigorating or challenging all aspects of course (not just low UEs) Trials • Monash University – Angela Carbone • University of Newcastle – Liam Phelan and Katherine Lindsay • Griffith University – Steve Drew • Edith Cowan University – Susan Stoney • University of the Sunshine Coast – Kylie Readman and Caroline Cottman Data Gathered • Reflections/case studies from PATS co-ordinators • Focus group sessions with Mentors and Mentees • Workbook activities • Unit evaluation results Findings • Understanding institutional uptake of innovations • Analysis of PATS Activities – HERDSA showcase 1-4 July New Zealand • Experiences of academics (PATS coordinators & participants) engaged with PATS • Course focus areas and changes in course evaluation results National Senior Tecahing Fellowship | Flinders University

37. Reflections/case studies from PATS co-ordinators- Understanding institutional uptake PATS coordinators involvement Recruiting/Engaging Participants Support from senior management as a strategy to improve L&T (Newcastle (Faculty of Business & Law), Griffith, Monash University) Staff volunteering (University Sunshine Coast, Newcastle (GradSchool) and Monash University) Integrating PATS into an existing and complimentary or aligned process (Griffith, Edith Cowan and Monash University) • PATS is consistent with the mission of Directors of central units and Associate Deans Teaching & Learning to improve the quality of learning and teaching • Awareness of positive responses and results from past PATS participants • Researcher-teachers had experimented with, or were trialling, peer assistance with teaching and wanted to introduce a scheme in order to enhance unit and teaching quality • Experiences engaging academics with the Peer Assisted Program to enhance teaching submitted to the Journal of Higher Education • Carbone et al, 2013 National Senior Tecahing Fellowship | Flinders University

38. Task analysis (in progress) • Task 2 – Understanding barriers to teaching improvement • What are inhibiting or prohibiting factors preventing academics from initiating improvements to unit’s content, design and/or delivery • Themes: Curricula, cultural, environmental, financial (DaRosa et al 2011) • Awareness and Knowledge, Skills, Acceptance and beliefs, Motivation, Practicalities (NICHE, 2007) • Task 3 – Developing a framework for effective educational goal setting • Goals are formulated in a broad or vague manner, need to be more focused • Focus areas of Goals | Goal setting frameworks | Role of Reflection in goal setting (Day & Tosey 2011; Dilts & DeLozier 2000; Zimmerman 2008) • Task 4 – Using Informal Student Feedback to close the feedback loop • Using ‘outputs’ as ‘inputs’: Benefits of creating feedback loops • Lecturer’s interpretations of students’ perspectives on their learning experiences, and lecturers’ decisions in response to vary or not vary teaching strategies and course management (Brookfield 1995) • Task 5 – A smarter peer observation instrument • Observers did not provide steps to develop teachers’ stated goals or to assist in addressing students’ feedback and unit development ideas (HERDSA, 2013 Showcase presentation) National Senior Tecahing Fellowship | Flinders University

39. Focus areas / Changes in course evaluations • Monash University • 9 courses participated (2 Education; 5 ICT; 2 Arts) • Response Rates (31-68%) *Indicates Early Career Development Fellow (ECDF) program participant Focus areas: course, lecturer, lecture, assessment, resources and increasing student motivation and participation National Senior Tecahing Fellowship | Flinders University

40. Focus areas / Changes in course evaluations • The University of Newcastle • 9 courses (7 Business/Law; 1 Environmental; 1 Education) • 5 low UE (FBL 1-5); 5 voluntary (FBL 6-7; EDU, ENV) National Senior Tecahing Fellowship | Flinders University

41. Focus areas / Changes in course evaluations • Griffith University • 12 courses participated (4 GBS; 2 Health; 6 SEET) • 6 low UE, 6 voluntary • Response rates (19-58%) National Senior Tecahing Fellowship | Flinders University

42. Focus areas / Changes in course evaluations • University of Sunshine Coast • 3 courses participated (1 Nursing; 1 Public Health; 1 Paramedic Clinic Founds) • All voluntary Focus Areas: course, students, teacher, tasks, assessment National Senior Tecahing Fellowship | Flinders University

43. Focus areas / Changes in course evaluations • Edith Cowan • 7 courses participated (3 Business-Law, 3 Education-Arts, 1 IT-Health Sciences) • All voluntary (as part of GCHE) • Response rates (26-80%) [still waiting on final data] Focus Areas: course, students, resources, administrative processes, professional, assessment National Senior Tecahing Fellowship | Flinders University

44. Reflection Peer Assisted Teaching Scheme National Senior Tecahing Fellowship | Flinders University

45. Minute of silence • Think of: • a new idea • a confirmation • a challenge • an observation National Senior Tecahing Fellowship | Flinders University

46. References • Brookfield, S. (1995). "Becoming a critically reflective teacher". San Francisco: Jossey-Bass. • Carbone, A. (2011). Building peer assistance capacity in faculties to improve student satisfaction of units. Proceedings of the 34th Annual International Conference, Higher Education Research Development Society Australasia (HERDSA), Gold Coast, Queensland 4-7 July 2011.  • Carbone, A., Wong, J., Ceddia, J., (2011). A scheme for improving ICT units with critically low student satisfaction. Proceedings of 16th Annual Innovation and Technology in Computer Science Education (ITiCSE'11) Darmstadt, Germany, 27-29 June, 2011. • Carbone, A., and Ceddia, J. (2012) Common Areas for Improvement in ICT Units that have Critically Low Student Satisfaction, Proceedings of the Fourteenth Australasian Computing Education Conference (ACE2012), Melbourne, Australia, 31 January - 3 February, 2012. • Carbone, A. (2012) A Peer Assist Teaching Scheme – A Way of creating, sustaining and developing new connections. Proceedings of the Annual International Conference of the Higher Education Research and Development Society of Australasia (HERDSA 2012), University of Tasmania, Hobart, Tasmania, (in press). • Carbone, A. & Ceddia, J. (2013) Common Areas for Improvement in Physical Science Units that have Critically Low Student Satisfaction, Learning and Teaching in Computing and Engineering, (LaTiCE 2013 ), Macau, 21-24 March 2013. • Carbone, A., Ceddia, J., Simon, D'Souza, D., Mason, R. (2013) Common Concerns in Introductory Programming Courses, Proceeding of the Fifteenth Australasian Computing Education Conference (ACE2013), Adelaide, South Australia, 29 January - 1 February, 2013 • Carbone, A. (2012). Opportunities and challenges faced in attempting to introduce a Peer Assisted Teaching Scheme to improve units with critically low student satisfaction, Higher Education Research Development (in press). • Carbone, A. Phelan, L., Ross, B., Cottman, C., Drew, S., Stoney, S., Lindsay, K., Readman, K. (2013) From taps on the shoulder to PATS on the back: Experiences engaging academics with the Peer Assisted Teaching Scheme, Studies in Higher Education (submitted for review) National Senior Tecahing Fellowship | Flinders University

47. References • Galbraith, C., Merrill, G., Kline, D. (2012) Are Student Evaluations of Teaching Effectiveness Valid for Measuring Student Learning Outcomes in Business Related Classes? A Neural Network and Bayesian Analyses. Research in Higher Education 53:353–374 • Kember, D. and Leung, D. Y. P. (2011). Disciplinary Differences in Student Ratings of Teaching Quality, Research in Higher Education, 52, 3, 278-299. • Lave, J. & Wegner, E. (1991) Situated Learning: Legitimate peripheral participation. Cambridge: Cambridge University Press • Ramsden, P. (1991). A performance indicator of teaching quality: The Course Experience Questionnaire, Studies in Higher Education, 16, 2, 129-150. • Vygotsky, L.S. (1978). Mind in Society. Cambridge, MA: Harvard University Press. National Senior Tecahing Fellowship | Flinders University

48. Questions Associate Professor Angela CarboneAssociate DirectorOffice of the Pro Vice-Chancellor (Learning & Teaching)angela.carbone@monash.edu (03) 9903 4481 PATS Program Officer Ms Joanne Rae joanne.rae@monash.edu(03) 9903 4494 Research Officers Dan Tout Dan.tout@monash.edu Bella Ross Bella.ross@monash PATS Website: monash.edu.au/pats National Senior Tecahing Fellowship | Flinders University