Julie Repper

1. Assessing the Impact of Policy on Practice: Using Organisational Case Studies to Assess the Implementation of Mental Health Nursing Policy Julie Repper

2. Today ... • To consider the value of organisational case studies as a means of evaluating the implementation and impact of mental health policy • To draw on our experience of evaluating the impact of the CNO’s review of mental health nursing: From Values to Action (DoH 2006)1 • To reflect on the involvement of service users and carers in conducting the research and in assessing the impact of policy 1(see full report at http://www.nottingham.ac.uk/nursing/cno-review)

3. Shift in Mental Health Policy • From requiring action to develop services with implementation guidance and measurable fidelity criteria (eg NSF for MH, 1999 and associated ‘PIG’s) • Towards values that underpin practice within those services (New Horizons - equality & justice, reaching potential, being in control of our lives, valuing relationships) • The implementations of such values is much more difficult to assess, attribution is hard to ascertain and measurement of policy impact is even more challenging!

4. From Values to Action (DoH, 2006a) • 15 years since last Review and much has changed in the healthcare environment. • Review of Mental Health Nursing commissioned by CNO to assess the profession’s fit in the rapidly changing healthcare environment. • Makes 17 recommendations for current and future practice and education. • Supported by good practice guidance for education and a ‘self-assessment tool-kit’ for Trusts. • Evaluation of implementation was an integral requirement of the report and funding for this was announced in 2007 and undertaken 2007-9.

5. Recommendations in Review

6. Aims of Study To evaluate implementation and impact of CNO review Objectives • To establish progress and strategies for implementing recommendations • To identify facilitators and barriers to implementation • To examine impact of implementation on user/carer experiences • To explore the relationship between organisational ownership, implementation progress and impact • To highlight areas of good practice and positive outcomes including effective strategies used to facilitate implementation     

7. Design Phase 1: Survey of Trusts and HEIs Phase 2: Case studies of 6 Trusts Phase 3: Follow-up survey of Trusts & HEIs

8. Case Study “... A strategy (not a method) for doing research which involves an empirical investigation of a particular contemporary phenomenon within its real life context.” (Robson, 1993) • Allows complex issues to be examined in detail (Platt, 1992) • Retains holistic characteristics of real life events (Stake, 1995) • More variables than data points (Yin, 1994) • Multiple sources of evidence (Robson, 1993) • Data collection and analysis guided by theoretical propositions (Yin, 1994)

9. Organisational Case Studies In-depth instrumental case studies (Stake 1995) of organisations to: • Understand diverse approaches and processes from different perspectives and levels • Identify key components of successful (an unsuccessful) processes and strategies • Suggest wider organisational features that support such processes • Generate theory by identifying key features within individual cases and building on this through cross case analysis (Eisenhardt, 2002 ) Case studies do not promote generalisability per se (Sandelowski, 1988), but they do: • generate insights which can be ‘transferred’ or ‘recontextualised’ (Morse 1994) by testing their application in diverse settings • therefore purposively selected distinctively different Trusts (high and low implementers; urban/rural; north, south & London; PCT/MH & Partnership).

10. Data Collected • Interviews with most appropriate personnel working at various levels within the organisation from Executive Directors to team leaders and front line workers (snowballing from Lead Nurse) - topics: response to review, strategy for implementation, monitoring progress, barriers & levers. • Policy and procedure documents and other relevant sources of documentary evidence identified during interviews (eg records of meetings, audits, and reports) • Interviews with staff working in related organisations (voluntary and/or private sector, education) • Interviews and/or focus groups with people using the service and with family members (‘carers’).

11. Process • Each Trust visited by a team of four researchers for several days to gain a clear picture of implementation at that site. • Framework Approach used to guide analysis - starting with a meeting following each Trust visit to identify key themes and areas for follow up on subsequent visits and/or in subsequent case studies (familiarisiation). • Recorded interviews and commentaries were transcribed and sent to all researchers who met to compile a coding index that was systematically used with each site data to produce a chart of relevant evidence for each Trust and develop site reports. • Cross case analysis began to identify over-arching themes, categories and patterns in relationships and events. All worked together on this to improve reliability and reduce bias in interpretation.

12. Service User and Carer Involvement • In good position to judge impact of review on practice • Contributed to meaningful research design (questions asked, involvement of external organisations etc) • Elicit more detailed and accurate responses from interviewees with shared experiences .... So • Four people with personal experience on Advisory Group • All SU interviews (n=32) conducted by SU and carer interviews (n=28) by carer (who had experience and training in research) • All staff interviews done with a service user/carer • All analysis done as a team • Writing up also done as a team.

13. Findings • No systematic differences between ‘high’ and ‘low implementers’. • At executive level there was general acceptance of Review’s recommendations but very little evidence of action linked directly to Review (no dedicated strategy group in any Trusts): “We were doing it all anyway. Maybe it put a bit of wind in the sail but it certainly did not build the boat” “We decided to check out that we are doing what it says but not to make a big deal about the document” “The Review box could be ticked because of work we were already doing on other policies”. • The research objectives were therefore amended to add what are the factors that influenced the implementation of the Review?

14. Factors influencing Response to the Review • Involvement in CNO consultation • Local Context Structural changes ( all recent or imminent FT status) Crises demanding a response (eg SUIs, poor HCC review) Leadership (3 Trusts had Acting Lead Nurses, two new executive nurses) • National Context Most recommendations reflected other extant policy requirements (eg Recovery, Social Inclusion, equality, EBD, Psychological therapies) Nursing voice on Executive Boards felt to be relatively powerless • Nature of the Review Recommendations are not part of accountability structures, not part of commissioning guidance, bring no linked funding, not worded as specific targets or requirements. Recommendations are not radical or challenging but reflect good practice (“motherhood and apple pie”)

15. Implementation of Recommendations in Practice (Practitioner interviews) • Few respondents other than exec team were familiar with the Recommendations so interviews focussed on the action that was being taken in these areas (whether or not this could be attributed to the review) • Most recognised the principles underpinning recommendations and could comment on progress made in their Trust (this was verified with evidence and compiled into matrices for each site). • It was more difficult to assess progress with ‘Values’ implementation such as Recovery and Equality: respondents had differing ideas about what these meant and how they should be implemented. No evidence of damaging interpretations, a lot of enthusiasm, much name changing but very little action. • Many would have liked to know more about the Review as they felt it would be a good guide to service development and improvement.

16. Impact on Service Users’ & Carers’ Experiences • Despite rhetoric of involvement, all Trusts found it difficult to identify service users & carers for us to interview; none had an involvement strategy; move to FT status was felt to have sidelined the voices of service users in generic groups/councils etc. • Most interviewees had not heard of review but most could cite examples of their involvement in small one-off initiatives and in interviews, induction, training, on PPI forum. • Voluntary sector groups had not heard of Review and had not noticed any additional involvement or funding to support recommendations. • When asked about changes in nursing services/service provision there was little perception of change other than increasing workloads and increasing levels of disturbance on acute wards. • Limited understanding of Recovery • Limited opportunities for social inclusion

17. Comparison of nurses with SU & Carers on barriers to implementation

18. Some Conclusions • Organisational case studies do elicit an understanding of the process of responding to and implementing policy but a three day visit is not sufficient to assess and attribute changes in culture • Different data sources expose gulf between rhetoric and practice and explain reasons for this • Different levels of interviewees demonstrate competing pressures in all areas and facilitate understanding of progress (or lack of) with implementation • But a week long visit is not enough to meaningfully • Similarities and differences between cases test the validity of developing theories • But the long distance between a policy recommendation and person receiving care makes it difficult to attribute any aspect of that care to a particular policy • ...made more difficult when the policy is based on values rather than definable action • ...and complicated if the policy does not say anything distinctive/different from other policies. • Service users & carers’ voices are an essential part of the research – not only in determining design but in keeping interviews on track (as interviewers) and in telling their stories (as interviewees).

19. So ... What about Recovery? • Conceptual confusion (clinical vs personal vs individual ) • No benchmarks for what ‘it’ looks like in practice • No approved Recovery training available • How can you assess the values underpinning a culture? CMH recently published ‘Making Recovery a Reality: A Methodology for Organisational Change’ which is a project to run in 6 MH Trusts working towards meeting 10 ‘organisational Challenges. These have been developed to guide commissioners – a move away from standardised outcome measures towards transforming a culture:

20. SCMH Implementing Recovery (2009 )10 key organisational challenges • Changing day-to-day interactions and the quality of experience • Delivering comprehensive, user-led education and training • Establishing a ‘Recovery Education Unit’ for peers to provide education and support for staff and for service users • Ensuring organisational commitment, creating the ‘culture’ • Increasing ‘personalisation’ and choice • Changing the way we approach risk assessment – negotiating agreed safety plans • Redefining user involvement into more equal partnership • Transforming the workforce – peer support workers • Supporting staff in their recovery journey • Increasing opportunities for building a life ‘beyond illness’.

21. Thank you Julie.repper@nottingham.ac.uk