Paired Learning/Vanguard Programme Evaluation Report 26 April 2013

1. Paired Learning/Vanguard ProgrammeEvaluation Report26 April 2013

2. Contents Section 1. Introduction Section 2. Evaluation findings Section 3. Service improvement projects and outcomes Section 4. Programme improvement suggestions Appendices Appendix 1 - Participants in the evaluation/feedback process Appendix 2 - Interview questions Appendix 3 - On line survey questions

3. Section 1 - introduction

4. Objectives, processes and data sources for this evaluation Objectives The objectives of this evaluation were set out in the original brief as follows: ‘The aim of the evaluation is to develop an understanding of the benefits and challenges of participating in the programme for the participants, sponsors and their organisations; to learn lessons, celebrate success and to inform future leadership development programmes in the Region’. Evaluation process/data The evaluation process and the findings presented in this report take into account the following elements: • NHS Institute for Innovation and Improvement evaluation data • Semi-structured telephone interviews at the end of the programme with: • The Programme Sponsors (x2) • The Programme Manager • Programme facilitators from the Region (x3) • Participants (all invited; x19 conducted) • Project sponsors (x6) • An on-line survey to all participants at the end of the programme (all invited; x14 completed) • Review of participant assessment presentations and posters (x22) • Review of the assessment process and outcomes Notes: • A full list of names of those interviewed and completing the on-line survey, together with the questions used are shown at the Appendices. • A document containing all the participant posters is attached separately with this Report • Full interview notes and survey data have been provided to the Project Manager

5. Background and programme aspirations • The Paired Learning programme was commissioned within the overall frame of the Emerging Leaders initiative under the sponsorship of Chris Bain, Chief Executive, Rotherham &Doncaster and South Humber NHS Foundation Trust David Wilkinson, Postgraduate Dean, Yorkshire and Humber Deanery. Fiona Sullivan was the Project Manager • It was organised and delivered in collaboration with the Institute for Innovation and Improvement who designed and delivered the ‘NHS Vanguard Programme’ elements. Vanguard had run successfully in 2011/12 , with six participants from Yorkshire and the Humber involved. The programme was offered again this year and interest was expressed by both Yorkshire and Humber and the Southern region • The integration with the ‘Paired Learning’ focus was unique to the Yorkshire and Humber programme • It arose from reflections on Emerging Leaders and was aimed at strengthening relationships and networks between clinicians and mangers in the Region and to build leadership sills for service improvement. The intention was that a manager and clinician ‘pair’ would work closely together on the programme - supporting each other in their project work and overall learning • Publicity literature targeted applicants as follows: • ‘The Yorkshire and the Humber Deanery and the Yorkshire and Humber SHA have teamed up to develop a new six month Paired Learning Scheme to help leaders deliver service improvements. We will pair up ‘emerging leaders’ , circa Bands 7 and 8 manager and clinicians (circa ST5 and 6) as ‘buddies’ in order to support leadership development and encourage collaborative work on service and quality improvement activities for the benefit of patients’. • I believe passionately in this type of • development • Chris Bain I was very committed to the principle of clinicians and managers working together – as early as possible David Wilkinson The aim was to build on the Vanguard foundations – and really to strengthen our clinician-manager relationships and networks Fiona Sullivan

6. Overview of the programme and its elements • The programme ran over a six month period from September 2012 to April 2013 • There was a heavy emphasis on ‘virtual’ delivery – with webinars (covering service improvement themes and tools) and Virtual Action Learning Sets delivered entirely on line • These were organised at fixed times and participants were expected to achieve a 70% attendance level • Participants met face to face at launch, assessment and final celebration events (alongside Sothern Region participants) • Participants were individually responsible for organising their project work – in conjunction with their project sponsor. Sponsors attended an on line coaching webinar and had access to all the learning materials and sessions • Participant pairs were responsible for organising the most effective ways of working together through the programme. One to one preparatory discussions Programme Elements

7. Profile of programme participants Recruitment • 25 participants from Yorkshire and Humber were selected for the programme drawn from a wide range of disciplines and localities • Participants were paired in advance of the programme (a clinician with a manager) • Some pairs were proposed through the application process • Most were matched centrally after application – with efforts made to reflect potential common/shared interests for project work • Most pairs did not working in the same organisation • A full list of participants and pairings is shown at Appendix 2. Completed • 3 participants dropped out before the final assessment process • The remaining 22 participants successfully completed the programme and were awarded the innovation practitioner certificate Awards • Alastair Mew, Senior Commissioning Manager, Sheffield and Laura Wakely, Vitreoretinal Fellow, York were awarded Best in Region.

8. Section 2 - Evaluation findings

9. Overall feedback from participants Would you recommend this programme? YES This evaluation aims to look at the programme in detail but it is perhaps useful to begin with the final question in both interviews and the on-line survey asking participants it they would recommend the programme to colleagues. Of 19 participants interviewed the overwhelming majority answered that they would recommend it – and many would do this strongly . The on-line survey data (with a smaller response size) reflects this majority view. Many expressed a hope that the learning from the programme can be amplified and shared widely - and a number offered support and suggestions for doing this. But within this very positive feedback there were also improvement suggestions which are provided at Section 4. • It was very useful.. very powerful…should be commonplace…part of the fabric…and I hope that managers and clinicians who have been on it take it forward…that we don’t lose the talent developed here

10. Recommending the programme - comments from participant and sponsors Participants (x19) • This programme would be beneficial to people from any area; I have already recommended it to others. I hope it continues to run • I would recommend this programme to middle grade, operational people. Am I glad I did it? I think so – it helped me to debate some ideas in my own mind • I would definitely recommend it. A good programme. I am enthusiastic about it. • It should be highly recommended… I would definitely say to doctors yes its worth doing… a lot of common sense but worth doing…I feel very, very grateful • I would recommend it – you can see how much effort has been put in; there is a body of learning about NHS improvement… science behind it…methodology…you can do it in a meaningful way…make more of improvement projects; a very rewarding experience; be good if it could continue • This type of training needs to be more widely available in the NHS; more about being a leader (not another ‘management’ course) - participative leadership, engaging people; Band 6 and 7 should be exposed to this kind of course; I would absolutely recommend it • I would recommend it; it does all come together… a journey; I don’t know whether Public Health will have access to this sort of programme in future? • I would definitely recommend it.. thought it very worthwhile.. increased my skills and confidence…good for junior doctors to have on your CV…; I will definitely use the improvement tools to help implement change; made me aware of what motivates people in work etc • I would recommend the programme – but just be aware of the time commitment • I have already written an article about the programme for the British Society of Rheumatologists; I think every new NHS consultant should have to go through this. .. it is bread and butter for the role • I would recommend it strongly to clinicians – without hesitation – think they found it fantastic; maybe reframe for manager like me with better matching of experience between managers and clinicians. However overall my impression has been very positive; I have learned a lot – particularly from the challenge of my VALS facilitator • I would recommend this programme; I’ve done a few management programmes and thought what was that all about? This time I’ve come away with models I’m using… I’m actually quoting things …. readings etc • I would certainly recommend it to others – but make sure they understand the time commitment – was a bit of a struggle juggling childcare to make sure I could attend

11. Recommending the programme - comments from participants and sponsors (cont) • I would recommend the programme but make sure you understand the time commitment; I would really like there to be a VALS in this region – keep something going • I would recommend it to others; in fact we are going to try and run a type of programme with junior doctors in my area – got the backing of the Trust Improvement Team; running short lunch time improvement sessions to give people tools and approaches; then going to give them projects and I will help and guide them with these; if this works well then I’m sure other Directorate Managers will be interested to do something like it too • I would recommend it – with the caveats for improvement • I would definitely recommend it; I’m the trainee rep from Yorkshire on the Royal College of Physicians training committee; happy to help in any way I can; I hope it happens again • I absolutely would recommend the programme; I think the pitching is pretty good; do people know it exists? how to keep the profile going ? • I would still have opted to go on it (even though did not complete) …very well pitched in terms of what the NHS needs to do ….value for money, improvement, quality Sponsors (x4) • I would love for it to continue ; it would be a real shame if it falls by the wayside… it has been really helpful for her and for the team • I think it has been well received • I think this programme is a wonderful gift for clinical trainees and I would recommend it . If you do it (centrally) that would be fantastic… but if not we would do it locally … I am willing to help push it forward as a ‘link person’ … there may be ways the Deanery can channel funding to this? • I would recommend it – definitely there is value in the tools and techniques it exposes them too ; but be careful with expectations of delivering an ‘earth shattering’ project in this time frame • I really think this should be pushed more for consultants …they come into the role with very little idea of how things work or how to change things…. They are used to working in clinical teams but don’t know the intricacies of how to get other people motivated.. how managers work • Yes I would recommend this programme – it should have a future; I hope it is not another NHS ‘good idea’ then we never see it again; I think it is really valuable for someone who is newly appointed to a senior manager role – I would certainly recommend the Vanguard model

12. Overall usefulness and benefits of the programme The online survey indicated that the majority rated the programme as very useful or useful. No one thought it had not been useful. The interviews explored the specific benefits which participants felt the programme had brought. These typically fell under three major headings: • Tools to use on this project and beyond: • Helped to push projects forward • Relevant to day to day work • Sustainable learning because putting it to use • Practical and useful resources to go back to • Understanding the people issues of projects • ‘My project developed more in the last 6 months than it had in the last 3years’ • Personal development: • Building confidence • Stepping back and reflecting • Space to explore issues • Being challenged in the VALS • Tough managing the time commitment • The amount covered in 6 months • ‘Glad I did it’ • Relationships and networks: • Access to other perspectives • Sharing ideas • Hearing about other people’s projects • Great support from the VALS • Understanding the NHS structure better • Having a sponsor to give access to the system

13. Impact on areas of work • Given the focus of the programme on building relationships, working in teams and across boundaries, the on-line survey invited ratings of the programme’s impact in a number of specific areas. Responses on the chart below show that areas of greatest impact were: • Clinician-manager partnerships • Working in a team • Initiating projects • Project managing • Areas of lowest impact were understanding hierarchies and commissioning which may reflect the organisational turbulence at the time of the programme.

14. Feedback on the webinars • I was sceptical about the whole management thing - overdoing common sense; as the programme went on I became more and more convinced of the importance • the teaching was inspirational • The interviews, on-line survey and Institute evaluation information show that the majority of participants valued the webinars but varied in their views on the most useful content areas - with needs, interests and relevance to project work varying across the group. • Some participants (particularly the more experienced managers) were familiar with the NHS Change Model and associated innovation approaches and tools and saw the webinars as an opportunity to refresh their knowledge. By contrast, all the material was new to some of the clinician participants . • A number of interviewees commented on the value of hearing Helen Bevan at seminar 1 . The Institute’s evaluation survey data on this seminar completed by 11 Yorkshire and Humber participants appears to reinforce this (chart above); Julia Taylor on Creating Contagious Commitment was also commended by a number of interviewees • Participants were very impressed with the quality of the resources made available to them - ‘great reading materials’; ‘access to experts’; ‘a privilege’ • A small minority of interviewees thought the webinars were too long (1.5 hours) with duplication/repetition of pre-reading • One participant valued the passion of the presenters but was wary of over-deference to a single approach to service improvement

15. Feedback on the VALS • Almost all of those interviewed commented that the VALS was a key strength of the programme – well facilitated and offering fresh perspectives, support and encouragement. • I thought the VALS were the best aspect – very supportive • I was very encouraged by my group - brought a range of perspectives – like a breath of fresh air The VALS was really beneficial – everyone had ‘air time’ but using a structured questioning process I would state my challenges and they would give an impartial view; give ideas – pity but we are not continuing to meet • The VALS was brilliant ; I’m a bit gung hoso the VALS was really good at giving other perspectives.. sometimes better to hold back a bit…a nice diverse mix of clinicians and commissioners • Seeing clearly some of the hostility between doctors and managers A really good VALS facilitator she made me do stuff; challenged my approach; some deep learning; made me reflect on how I approach things • Sometimes I wanted ideas to make my project more effective; but more importantly I got to hear about other projects – and understand them There was an absolutely wonderful moment towards the end when a girl who was sceptical at the outset had done a fantastic project… came out of herself… there was real excitement in the group… a highlight VALS group rallied round if you were struggling with your project; used good questioning; also offered suggestions and tools from their experience • VALS – a great bunch of people; • great facilitation – could have been a nightmare! • VALS worked well; a very good facilitator

16. Feedback on the paired learning element • The paired learning element divided opinion amongst participants • A few saw it is one of the strengths of the programme offering a source of support for project ideas and general encouragement • However many felt the time commitment from other elements of the programme was already intense and because the VALS gave regular access to a clinician-manager network and project challenge/support many participants could not see the added-value in the paired learning element for the further time pressure it created • For those paired with someone in another organisation the practicalities of finding time to meet were an inhibitor – although most made contact by phone and e mail • There was also felt to be less guidance and structure than for other aspects of the programme; and some participants were confused because members of their VALS (from Southern region) were not paired • Higher satisfaction with the paired learning element came where there was a good ‘bond’ of shared interest within the pair - eg working in the same organisation or on a similar project. Here it was felt that strong relationships were established. • Definitely paired learning was a useful part of the programme – but we both work part time so hard to carve out time together • Pairs are critical to the programme … can’t have doctors who do everything alone…can’t have managers who do everything alone… • Maybe the success of pairing depends on having shared ground with your partner • Idea was great but in reality I got most help from my sponsor ; you are so busy with your day job, the programme and the project that you just don’t have time to help someone else out • He helped me when I was having a problem with commissioners on my project • I suppose you get out of it what you put in and we might have made more effort • We never really found the time to work together as a pair; • we met in the VALS and got on well but not outside it • The paired element was a bit of a funny one - not really sure what we were supposed to do

17. Assessment process and outcomes • The final assessment of the programme was based on a number of elements: • a 10 minute powerpoint presentation to demonstrate programme learning linked to the service improvement project. (assessment criteria were set out in detail as shown on the table below) • a poster summarising the learning from the programme and its application • 70% attendance at webinars and VALS sessions • All 22 participants successfully passed (with one required to re-present) and were awarded the innovation practitioner certificate • Most of interviewees felt that the assessment process was clear and fair • One felt the process was weighted against those without previous experience of presenting in this way, that 10 minutes was a very short time to present on hundreds of hours of project activityand that the pass/fail approach was inappropriate for a programme that does not have external accreditation • A manager who passed the first presentation thought the process could have been more rigorous I can’t say I enjoyed it but it was good to be clear what was expected

18. Feedback on programme practicalities and quality • Most participants welcomed the virtual nature of much of the programme and the travel time this saved • Where technical problems were encountered support was available to sort things quickly • The chat facility on the webinars was not widely used although it was noted by a number of participants that this gave the opportunity to raise issues live if you wished to • Participants were generally very complimentary about the quality of resources, support and administration • However a number of participants commented on the volume of e mails and big attachments which were sent out • One interviewee felt it was ‘over-administered’ • One project sponsor questioned the cost of providing all the materials to him in hard copy • I would like to say the resources available – slides, seminars, information were fantastic, amazing I loved the virtual part of it … very cost effective… you can develop a relationship using a webcam! • You could do all this without leaving your office

19. Section 3 - Service improvement projects and outcomes

20. Overview of the project work and sponsor role Project progress • Many interviewees commented on the value that the programme brought to their service improvement project by: • bringing focus and attention to the way they were approaching their project work • offering tools and theories for practical application • bringing new ideas and challenge from the members of the VALS group and pairings • strengthening relationships with the sponsor • For a number of participants this meant that their project progressed faster and more effectively than they believe would have been the case without the programme The role of the sponsor (participant perspective) • The majority of interviewees commented positively on the support they received from their sponsor – although the level of their input varied some meeting weekly whilst others less frequently • For many the sponsor was already the line manager working closely on the project - but the programme brought additional focus to the project review and learning activity The role of the sponsor (sponsor perspective) • The sponsors valued the structure offered by the programme, the tools offered and relevance to the project work • They generally agreed that that the programme had significantly increased the confidence and skills of the participant in their project activity and increased their understanding of the system as a whole • As sponsors they valued the development offer available: • For their coaching role • Giving them access to all the webinar materials • But In reality sponsors did not have time to attend many of these The following pages give a summary of the projects undertaken and what has been achieved to date. All of the project posters are provided as an attachment to this report. • The programme allowed more reflective time together than we would have in our normal line management relationship. I used to go behind her to meetings. But after the first project meeting she said she couldn’t make the next one … she wanted me to go on my own… the programme allowed this to happen

21. Improvement projects and outcomes to date (1)

22. Improvement projects and outcomes to date (2)

23. Improvement projects and outcomes to date (3)

24. Improvement projects and outcomes to date (4)

25. Participant projects and outcomes to date (5)

26. Section 4 - Programme Improvement suggestions

27. Improvement suggestions/future programmes Communication and targeting: • Communicate more clearly about the actual time commitment (an overwhelming view) so that participants realise that whilst it is largely ‘virtual’ the timings of session are fixed • Allow a longer lead time for the application process to help people plan the time commitment • Target clinician/manager pairs from the same organisation/project area to improve the effectiveness of this element • Encourage early career participation for clinicians Delivery: • Timing of webinars mid-Wednesday not ideal for clinicians – general preference to run things early or late in the day • Webinars 5 and 6 both on Creating Contagious Commitment could be combined • Reduce the number of VALS • The above time reductions could increase time available for the paired learning element Content: • Tailor content to the different learning needs of clinicians who may never have come across the webinar material before v experienced managers who may be very familiar • Include More input from practising NHS Trust leaders/senior managers • Include stronger review of the paired learning element at the final assessment Administration: • Reduce the number of e mails Offers of support for future programmes: • A number of interviewees (participants and sponsors) made offers of support to keep a programme of this kind running • Some are already sharing learning locally - or are considering ways of doing this

28. What future for the programme?

29. appendices

30. Appendix 1 – Participants in the evaluation/feedback process

31. Appendix 2 – Participant pairings

32. Appendix 3 - Interview questions Overall benefits/impact of the programme • What do you see as the main benefits of this programme • For you? • For your organisation? • How have you changed your practice as a result of the programme? • How has the programme enabled this change? • What are the key pieces of learning you will take away from the programme? • How will you continue to apply these now the progamme is over? Approach/ format of the programme • (NB You are asked on the on line questionnaire to rate the effectiveness of each element of the programme but I would like to explore in more depth the overall strengths, weaknesses and suggestions for improvement) • What are the challenges of participating in this programme for • The individual? • Their organisation? • What do you see as the particular strengths of this programme? • And the weaknesses? • How effective have the various elements of the programme been? • Most useful? Why • Least useful? Why? • What could be done in the future to improve the progamme/minimise the weaknesses? Relationships – Paired learning; mixed-group learning; sponsorship • What have been the benefits of participating in a programme with a group of professionals from mixed disciplines? • What have been the benefits of being paired with a professional from a different discipline to your own? • What have been the benefits of being on a programme with someone involved in your project? • Any suggestions for making the paired learning relationship more effective? • How has your sponsor been involved? • What have been the benefits of having a sponsor? Any other comments/suggestions for improvement

33. Appendix 4 – On line survey questions Reflecting on your experiences of the programme, please rate the impact it has had on how prepared you feel in each of the areas of work below (1=No impact; 2=Some impact; 3=Significant impact; 4=Outstanding impact) • Working alongside senior management colleagues • Working alongside clinical colleagues • Working in clinician-manager partnerships • Working in a team to deliver a service • Communicating with all members of my department • Communicating with all stakeholders • Leading multi-professional teams to improve services • Developing a business case to support a development plan • Initiating a project (gaining commitment) to improve local services • Project managing a quality improvement initiative • Measuring outcomes of projects • Understanding how management decisions are made • Understanding the professional hierarchies within clinical teams • Understanding how services are commissioned and funded Please tell us the most useful/essential webinar topics of the programme Please tell us any improvement suggestions you have for any element of the programme