What works? Student retention and success c hange programme Progress Meeting

1. What works? Student retention and success change programmeProgress Meeting • MacDonald Hotel Manchester • 17 September 2013

2. Welcome and housekeeping • Fire procedure. • Cloakrooms. • Refreshments. • Catering. • Administration and staffing. • Wifi code.

3. Introductions • Professor Liz Thomas, HEA Associate • Dr Helen May, HEA Academic Lead • Steve Outram, HEA Academic Lead • Dr Joan O’ Mahony, HEA Academic Development Officer • Professor Patricia Broadfoot, HEA Associate and Chair of the Advisory Group • Denise Barrows, Paul Hamlyn Foundation (PHF) • Professor MantzYorke, PHF and HEA Associate • Michael Hill, Action on Access and HEA Associate • Professor Chris Hockings, HEA Associate

4. Objectives of the progress meeting • To work with and learn from other institutions to reflect on progress and share plans, evaluation strategies and experiences towards enhancing student engagement, belonging, retention and success. • To reflect upon approaches to engaging and influencing different stakeholder groups (especially staff) in implementing sustainable change. • To work as a team to revise and update your institutional, discipline and collective approach to implementing and evaluating change.

5. ‘Rules’ of engagement • Mutual respect, trust, support and encouragement • Remain open minded and non-judgemental. • Confidentiality • Work within ‘Chatham House Rule’. • Consent • Obtain permission prior to disclosure of data, materials or other information.

6. Initial feedback from your reports • Summary of the review process. • Strong commitment to the What works? Programme, informed by institutional context and priorities and shaped by institutional approaches. • Alternative approaches or models are emerging in terms of the process of change, particularly in relation to types of institutional level changes, top-down and bottom-up approaches and nature of the relationship between core and disciplinary teams. • A broad range of disciplines working across the thematic areas.

7. Disciplines and themes

8. Sustainability • Institutional teams given thought to the issue of sustainability, identifying the following approaches: • It is a strategic priority and will be embedded into future strategy/policy documents. • Changes will be embedded into institutional services. • Involve additional discipline teams, incentivised through KPIs, development funding. • Staff development. • Dissemination, e.g.through L&T conference.

9. Evaluation Always challenging in education in general and change programmes more specifically. Good progress has been made, but we would like to build on this today.

10. Revision of reports • Following the review of reports, the Advisory Group recommends that some aspects of the reports are revised. • One member of the Change team or Advisory group will provide you with specific verbal feedback today. This will be followed up with a summary of the key points via email by the end of the week. • Institutional teams should provide a written response (1-2 pages only) about how they have addressed concerns or suggestions made. Only re-submit your report if there are significant revisions. • Commentary needs to be submitted to retentionandsucces@heacademy.ac.uk by 11 October 2013.

11. Requests for today • Update from other teams, reflection on first year, sharing practice, time with similar teams (2) • 1-1 feedback (x2) • Engaging staff, overcoming resistance, creating staff belonging • More detail on evaluation (x5) • Team building / leadership • Data, evidence and research • Structured time for project managers • Roundtable discussion time can pick up other issues.

12. Overview of the day • Looking forward, looking back: Sharing and reflecting with other teams. • Evaluation in context: covering range of evaluation and process issues, project manager time, and sharing with others. • Making (more) friends and influencing (more) people: engaging staff and developing belonging, using evidence, embedding retention and success. • Team time: working together, includes feedback on your report and suggestions for revisions. • Next steps: clarifying process issues.

13. Looking forward, looking back: 1045-11.15 • Steve Outram

14. Looking Back • Think of what has happened with your initiative. • What has worked? Where has there been impact? • What issues have you resolved? • What lessons have you learned? • What issue or challenge still • remains? • Share your thoughts with the table

15. Looking Forward • Turn your challenge into a goal • What options do you have? • Which is the best one? • So what WILL you do? • Share your thinking with • your table 16

16. Evaluation in context: 11.15 – 12.15 • Liz Thomas and Mike Hill

17. Anticipated causal relationship 18

18. Evaluation indicators and methods Unintended consequences 19

19. Translation into the real world

20. SMART indicators • Specific – Key indicators need to be specific and to relate to the conditions the activity seeks to change • Measurable – Quantifiable indicators are preferred because they are precise, can be aggregated and allow further statistical analysis of the data. However, process indicators may be difficult to quantify, and qualitative indicators should also be used. • Attainable – The indicator (or information) must be attainable at reasonable cost using an appropriate collection method. • Relevant – Indicators should be relevant to the management information needs of the people who will use the data • Timely – Information on an indicator needs to be collected and reported at the right time to influence many management decisions. • ) 21

21. Some examples of indicators • Module Leader on-line toolkit: 80% pass rate for the toolkit’s online quiz: data collection built into toolkit functionality. Salford University • New Student Review (PDP) Scheme: That 70% students will complete on-line form in year 1; 85% in year 2. St Mary’s University College • Co-curricular: Focus groups will be conducted with students – possibly also facilitated by students in conjunction with the SU – to evaluate the co-curricular provision. University of South Wales – Music Technology • Induction: Majority of students report feeling engaged and belonging. Post – event (induction) short and on-line questionnaires. University of Chester – Computer Science

22. Areas for development • More specific indicators. Many of the reports currently describe the type of changes that are expected but do not yet quantify them. • Measuring the expected change. Some of the indicators are output or take up indicators, rather than outcome indicators. • Methods of data collection. Some indicators do not have information about how the data will be collected to provide evidence of the expected change.

23. Activity • There will be six groups. You should join the group most closely aligned to the type of changes you are implementing: • Induction – MantzYorke • Active learning and teaching – Chris Hockings • Personal tutoring – Joan O’ Mahony • Peer mentoring – Helen May • Changes to policies and strategies – Patricia Broadfoot • Changes to services to support change – staff development, data, etc. – Steve Outram

24. Activity • Work in pairs. Look at your activity and evaluation plan (either section 7 & 8, or 10 &11). Spend 30 minutes critiquing your evaluation plans, using the following prompt questions: • Are there lines in the evaluation plan relating to each of the planned activities? • Is the logic underpinning the evaluation plan for each activity clear and convincing? • Do the change indicators adequately measure the expected change in behavior or attitudes? Can you think of any amendments or additional indicators that could be useful? • Are the methods of collect the evidence specified in sufficient detail, and are they appropriate and realistic? • Is the baseline data provided and/or is additional baseline data likely to be required?

25. Activity • As a whole group discuss: • The challenges involved in evaluating impact in this area. • Solutions and good examples of ways of measuring impact in this area. • The facilitator should make a note of the challenges and good examples, to be shared with all teams following the meeting.

26. Making (more) friends, influencing (more) people: 13.15 – 14.00

27. Roundtable sessions • Embracing resistance, Steve Outram • Incentivizing students, Helen May • Influencing staff, Liz Thomas • Making use of evidence to change, Mike Hill • Embedding student engagement, belonging, retention and success into institutional structures, Patricia Broadfoot • Promoting ownership and responsibility amongst academic staff , Chris Hockings • How to encourage students to respond to the belongingness survey,MantzYorke • Your call. An opportunity to discuss issues not identified above. Denise Barrows

28. Team time: 14.00 – 15.30Time for you to work together, receive feedback on your report and update plans based on what you have participated in today.

29. Feedback arrangements

30. Next Steps: 15.30 – 15.45 • Liz Thomas

31. Revision of reports • You will receive a written summary of the verbal feedback you received on your report. • Please provide a written response (1-2 pages only) about how they have addressed concerns or suggestions made. Only re-submit your report if there are significant revisions. • Commentary needs to be submitted to retentionandsucces@heacademy.ac.uk by 11 October 2013. • The reports and commentaries will be reviewed by the Advisory Group and then 22 October approved by the PHF Board 5 November 2013.

32. Contextual evaluation • In 2013/14 we will undertake two aspects of the contextual evaluation. • In October 2013 we will arrange a phone interview with the Team leader and any other members of the core team s/he things appropriate to explore the process of change. • In May/June 2014 we will ask discipline leads to provide us with details of the interventions they have put in place, reflections on the process of change, and outcomes of the interventions. • During 2013-14 each core team and/or discipline team needs to collect student views on interventions, and any other evaluation evidence as detailed in your evaluation plan. Qualitative evidence will need to be included in interim reports (July 2014 and 2015) and final case study (December 2015).

33. Impact evaluation clarification • Pre-entry survey (optional, by end of October each year) • Student engagement and belonging survey • Cohort AY2013: by mid-Nov 2013 and by mid-June 2014 & 2015) • Cohort AY2014: by mid-Nov 2014 and by mid-June 2015 & 2016) • Institutional data (supplied by 1st February each year). • NB: the Institutional data 2011-12 row of the Key Dates table should be disregarded.

34. Discipline teams • Teams must implement a (pilot) intervention informed by What works findings in one of the areas of induction, active learning and teaching and co-curricular support. • Discipline teams should work with their core team to ensure they are supported with any they encounter, and ensure institutional level changes are working to support academic teams to implement change effectively. • There will be two residentials for discipline teams 3-4 April, Telford Conference Centre and 16-17 April, Weetwood Hall, Leeds. The team should sign up for the residential that is most convenient for them to attend. • Core teams and discipline teams should work together to ensure the evaluation plan is implemented to be able to report in July 2014.

35. Residentials 3-4 April, Telford Conference Centre 16-17 April, Weetwood Hall, Leeds

36. Core team • During 2013/14 you should be continuing to implement institutional level changes and revising them in response to formal and informal feedback. • You should also be working with your discipline teams to address any other ‘strategic challenges’ that arise. • You should contribute to impact and contextual evaluation, and undertake institutional and discipline level evaluation. • You should attend the follow up meeting which will take place in June 2014. • Interim report will need to be submitted by 31st July 2014. • Further activity, evaluation, meetings and reporting will take place in 2014/15. • Final case study and institutional data by December 2015.

37. Change of HEA contact details From this point on the What works? Student retention and success change programme will be co-ordinated by the Retention and Success team at the HEA rather than the Change team. The generic email is now: retentionandsuccess@heacademy.ac.uk Thank you. Have a safe journey home.