Jonathan Benn Centre for Patient Safety and Service Quality Imperial College London Glenn Arnold

1. Enhanced feedback from perioperative quality indicators: Studying the impact of a complex QI intervention Jonathan Benn • Centre for Patient Safety and Service Quality • Imperial College London • Glenn Arnold • Imperial College Healthcare NHS Trust • Research Group: • Danielle D’Lima • Joanna Moore • Igor Wei • Alan Poots • Alex Bottle • Stephen Brett

2. Declaration of funding and conflicts of interest Project funding: NIHR CLAHRC Northwest London NIHR HS&DR Research Programme Conference attendance funded by: NIHR & Imperial College London Conflicts of Interest: None: No payment received for presentation

3. QI concept:Provision of real-time feedback on quality of anaesthetic care (for anaesthetists) • Anaesthetists rarely receive systematic, routine feedback on the quality of anaesthetic care delivered (and as experienced by the patient) in post-operative recovery

4. Review of quality indicators in anaesthesia (2009) • Perioperative morbidity and mortality data lacks the sensitivity and specificity necessary for analysis of variation in quality of anaesthesia. • Few validated indicators incorporating the patient's perspective on quality of anaesthetic care.

5. Survey of use of quality indicators in perioperative units (2012) Local data collection driven by theatre productivity and external reporting requirements Patient satisfaction with anaesthesia infrequently monitored Post-op patient temperature, pain and nausea data is not reliably monitored and utilised at local level, in the majority of perioperative units

6. Concept for a quality monitoring and feedback initiative • A continuous control loop representing learning at the individual and micro-system levels:

7. Concept for a quality monitoring and feedback initiative • A continuous control loop representing learning at the individual and micro-system levels: • Research questions for improvement science: • Can we conceptualise “data feedback” as the core of a quality improvement intervention? • Under what conditions are “data feedback initiatives” effective in improving care?

8. Contributions from improvement science Continuous process monitoring - an industrial model: • Provides a continuous signal, representing variation over time, rather than a snapshot view of standards at one point in time • Emphasises reliability rather than the extent of specific deviations • Supports open and objective discussion about variations in performance and learning from best practice examples • Supports rapid detection and correction of problems in near real-time • Effects of QI interventions are observable, iterations are systematic and guided by empirical evidence • Disaggregates data onto a level that is meaningful for users • Fosters local ownership of data and responsibility for improvement • Data collection is integrated within routine operations • Metrics are stable and reliable

9. Research basis for data feedback interventions • Systematic reviews of the effects of feedback on professional practice typically show small to moderate positive effects (e.g. Jamdtvedt, 2005) • Adding elements (such as education & quality improvement methods) to basic data feedback reports enhance their effectiveness (van der Veer, 2010; de Vos, 2009) • Qualitative research suggests that effective data feedback for quality improvement has a number of characteristics (Bradley, 2004) • Timeliness • Specific to the local context • Originates from credible/respected sources • Is non-punitive • Is sustained over time

10. IMPAQT (CLAHRC project): Anaesthetic quality monitoring & feedback at St Mary’s, London CLAHRC improvement model: • Iterative change (PDSA) • Focus upon local multidisciplinary engagement • Supported by continuous measurement and evaluation (SPC) Quality monitoring in PACU: • Temperature on arrival in recovery (NICE Guideline) • Quality of recovery/anaesthetic: • Patient reported Quality of Recovery (QoR) score (Myles, 1999) • Post Operative Nausea and Vomiting (PONV) (Categorical) • Pain scales (Categorical and continuous scales) • Patient transfer efficiency (Ward Wait Time) • Additional data is routinely compiled from the theatre and patient administration systems.

11. St Mary’s Main Theatres: Data process Intra-operative care pathway Ward feedback report Feedback anaesthetic quality indicators (personal level data) Pre & Intra- operative care Quality of Recovery, PONV, Pain, Temp, Patient transfer delays Anaes. feedback report Data validation & cleansing PACU Excel templates Database Feedback quality of recovery and transfer efficiency metrics PACU data posting Feedback patient transfer efficiency metrics (ward level data) Surgical wards

12. Monthly PACU & Ward Feedback Data posted in recovery Surgical ward reports

13. Personalised feedback for anaesthetists(Version 1: Sep 2010)

14. Enhanced feedback reports (Version 3: Feb-July 2012) • Developed based on interviews with end-users • Programme of active, trust-wide engagement and work with specialty sub-groups • Enhanced monthly report features: • Inclusion of multi-site data • Comparative perspective: individual vs peer group • Longitudinal view on variation in personal and group practice • Identification and description of statistical outlying cases to support case-based learning • Specialty-specific reporting of Pain scores (to better account for case mix)

15. Mixed-methods evaluation of anaesthetics QI initiative (NIHR HS&DR) • Evaluation of effects upon perioperative process and outcome indicators • Interrupted time series analysis of quality indicators dataset merged at case level with hospital administrative data • Semi-structured investigation of implementation context and perceived acceptability of the initiative • Theoretically-informed qualitative research interviews with consultant anaesthetists and perioperative service leads • 2 rounds of interviews: 1) formative, 2) evaluative • End-user evaluation • Survey data collected at multiple time points • Baseline (pre-feedback) • Multiple post-implementation follow-ups at three hospital sites

16. Effects of implementation of feedback on perioperative warming No Feedback Basic feedback Enhanced feedback Main anaesthetist cohort, all St Mary’s surgical cases Mar 2010 - Sep 2013

17. Effect of introduction of enhanced feedback(multi-site data) • Proportion of patients with temp below 36 degrees: • Stepwise decrease of 9% with introduction of enhanced feedback (p<0.01) • Proportion of patients reporting no pain or mild pain (compared to moderate or severe): • Stepwise increase of 8% with introduction of enhanced feedback (p<0.01) • Proportion of patients free from nausea: • Small improvement in rate of change over time following introduction of enhanced feedback (p<0.01) • No significant effect of feedback on Surgical Site Infection rate • No significant effect of feedback on 30 day mortality

18. Qualitative investigation: Anaesthetists’ views on feedback “I know that I’m able to immediately affect the outcome of these measures, so I can do things to make these measures different.” “I thought: ‘My goodness, I do quite a lot of patients’; ‘my goodness, oh, some of them are in more pain than I thought they would be in’. So I did some things to change it.” “For me to improve my practice I would need to first have my own data over a month or over a year.....and also how does my data compare to other anaesthetists that do exactly the same thing” “I think having departmental level data is important, data for the department that identifies areas where the department as a whole needs to improve or is performing adequately.” “I don’t think we’re particularly adversarial here, and I think we generally, discuss things and we’re quite open with each other about our data and about how we do things.”

19. Longitudinal survey evaluation: Usefulness of locally available data for QI Scale: 1 “Completely inadequate” to 8 “Excellent” Item descriptions • Level of analysis: Relevance of data to personal practice • Timeliness: Adequate frequency for monitoring variation • Communication: Effectiveness of channel and method of dissemination • Data presentation: Clarity and usefulness of graphical formats • Credibility: Perception of trustworthiness and freedom from bias Comparison of pre and post feedback implementation

20. Complexity & challenges in evaluation

21. j.benn@imperial.ac.uk www.cpssq.org