A Pragmatic approach to Evaluation in Utilisation Research

1. A Pragmatic approach to Evaluation in Utilisation Research Professor Brendan McCormack (with Rob Garbett)

2. Current Challenges in Evaluating Utilisation • Little conceptual clarity and murky language • Dominant focus on evaluating technical interventions (instrumental utilisation) • Linear and logical approaches to evaluation that don’t always take account of complex organisations • Reliance on input and output models of evaluation

3. Current Challenges in Evaluating Utilisation • Complexity of decision-making processes, meaning that no single evaluation mechanism can capture this complexity • The balance of process and outcome and the privileging of one over the other • `Evidence-based person-centred practice’ – what are we doing to evaluate the person-centredness? • What is it we are evaluating?

4. Where is the evaluation of person-centeredness in the current utilisation agenda? • Receiving the right intervention, at the right time etc based on an assessment of a need/problem • Person as a disconnected entity from the context in which they receive healthcare • Practitioners as disconnected entities from the contexts in which they provide healthcare

5. A Problem - ‘Unclean’ endoscopes • Major public concern and media profile • RGH – 3000+ patients recalled • 2 departments in RGH involved – DPU and main theatre suite (Same unit) • DPU = no cases of contaminated scopes • Main theatres = many cases of contaminated scopes • DPU use guidelines developed by the ‘Association of Endoscopic Surgeons’ • Main theatres use guidelines developed by the ‘National Association of Theatre Nurses’ • Similar but differently worded guidelines leading to differing interpretations

6. What questions does this example raise for us? • Why the use of two different sets of guidelines in the same unit? • What interpretations led to safe and unsafe practices? • What changes would have led to the agreement of ‘best practice’? • What contextual factors acted as barriers to the sharing of knowledge?

7. Contextual Factors as Barriers • Person-centred perioperative nursing project (Kerr, McCormack et al, in progress) • 150 hours of non-participant observation • Appreciative inquiry groups • Stakeholder analysis with steering group

8. Barriers Identified • Culture of busyness ‘call for the next one!’ • Inconsistent/inappropriate approaches and styles of leadership • Ineffective use of the nursing resource available • Poor delineation of roles • Reactive training rather than learning through practice • Factions, Cliques and Tribes • Lack of person-centredness, i.e. • Patient as product • Nurse as producer • Doctors as product controllers • Managers as quality controllers

9. Context • PARiHS Framework: Culture, Leadership and Evaluation • Evaluation [weak]: • Absence of feedback • Narrow use of performance information • Reliance on single methods

10. Being person centred • Creating the conditions that help us to: • Know another as a unique individual • Understand and acknowledge individuals beliefs, values, wants and needs • Provide care characterised by flexibility, mutuality, respect and care • Create environments that are flexible, respectful and caring of peoplebbbb

11. The person centered nursing project • 8 clinical areas in the intervention group, 3 in the control group • Includes critical care, acute areas, peri-operative care, rehabilitation and clinics

12. Methodology Quasi-experimental design Pre-post test; Instruments to measure particular dependent variables via patient and staff questionnaires at 4 monthly intervals Intervention Phase Practice Development (PD) Framework Mapping the journey Qualitative data sources

13. Quantitative measures Two questionnaires developed and administered Nursing Context Index (NCI) 21 Constructs/ Caring Dimensions Index Perceptions of Nursing Index (PNI) Satisfaction and Experience of being nursed

14. Qualitative sources • Taping of interactions • Practically challenging • Field notes • Meetings, working with individuals • Thoughts and feelings • Contextual issues: staffing, competing priorities etc.

15. Using the data • ‘Problematising’ • Workload scores and ‘well, what do we do with our time?’ • Questioning and reflecting • Communicating constructs and ‘how do we get through to people?’ • Confirming

16. ‘Food for thought’ ‘The project is giving us food for thought, especially the information from the nurse questionnaires, it gives us an idea of how the staff are really thinking. Also what (our facilitator) has told us what she is hearing on the tapes. ‘This project has spurred us to make changes which we never had thought we needed to make, for example how EENT and recovery are actually managed staff wise. It has identified that we need to perhaps have more senior staff nurse grades. And the noise level in the unit has been reduced after feedback.  ‘I particularly enjoy being involved. It has certainly motivated me to make and accept changes and differences to the department to ensure that it is more patient centred and person centred. ‘I know that over time when staff do see the benefits to patients, the department and to themselves that they will be less timid about the project. This will take time and more upbeat promotion from the team …’

17. Mapping the journey • Evidence of reduced stress in all intervention areas over 1 year • Greater sense of feeling supported and of workload becoming more manageable

18. Some examples • ‘Not enough time to complete all nursing tasks’ • October ’03: 40% said often, frequently or always • April ’04: 21% said often, frequently or always • ‘Not enough staff to adequately cover the unit’ • October ’03: 80% said often, frequently or always • April ’04: 53% said often, frequently or always • ‘Lack of opportunities to talk openly about problems on the ward’ • October ’03: 40% said often, frequently or always • April ’04: 5% said often, frequently or always

19. Tape recorded data: Interaction between nurses and patients (2) Getting work done Sharing decision making Opportunities for interaction occurred while getting ‘tasks’ done: for example drug rounds Interaction could be mainly task focused There was evidence that these could be used to work with patients, for example making choices about whether to take analgesia

20. Reduction in stress • Evidence that staff feel: • More appreciated • Clearer about their own development • More supported in the workplace

21. Some examples • Work in the clinical areas has provided opportunities for people to talk, and the emphasis of person centeredness is resulting in more openness • ‘People are saying things to each other that wouldn’t do at the beginning …’ • Working on changes provides people with recognition and affirmation • ‘This project has given me a new interest …’

22. Feeling more in control • The journey in intensive care: • Taking control over continuity of care • Becoming more reflective, learning from practice

23. Tape recorded data: Interaction between nurses and patients (1) Showing ‘sympathetic presence’ ‘Chatting’ For example, picking up on a person’s comments to find more out about them and their home circumstances For example, talk that appears warm and friendly but without obvious therapeutic intent

24. Evaluation as praxis • Praxis: • Doing action. • Concerned with an ‘ethical end’ that cannot always be predetermined in advance and is context dependent • Quality of the end product is inseparable from the process of getting there. • Not rule-following behaviour but based on ‘practical; wisdom’, i.e. combined perception, reasoning, virtue and technical competence.

25. A Programmatic Approach Mechanism Context Stakeholder, concerns, claims and issues Multi-method Stakeholder, concerns, claims and issues Multi-method Interactions Process Evaluation/Outcomes Multi-method Outcomes [after McCormack, 2000]

26. Conclusions • Using multiple sources of data allows us to demonstrate the impact of a practice development programme over time • Placing a quasi-experimental design within an overarching philosophy of ‘praxis’ enables ownership, participation and changes in culture whilst ensuring rigour. • A programmatic approach to evaluation may offer us a way forward in the development of our understandings of knowledge use