Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care

1. Developing Quality Indicators When There is Limited Evidence: The Example of Injury Care H. Thomas Stelfox, MD, PhD University of Calgary February 9, 2012

2. Objectives • Describe a quality indicator development process • Review strategies for determining the need for quality indicators • Review the value of environmental scans • Discuss the role of consensus methodologies • Review lessons learned from the process

3. Objectives • Describe a quality indicator development process • Review strategies for determining the need for quality indicators • Are there opportunities to improve care?

4. 16 yo Post Motor Vehicle Collision

5. Decompressive Craniectomy • Patient scheduled for OR • Pre-op blood work ordered • aPTT > 5 X normal

6. Ordered Administered

7. Problem – Quality of Care • 98,000 die in US hospitals from error each year • Canadian study of adverse events: • 7.5% of hospitalized patients • 1/3 of adverse events judged preventable

8. Patient Safety Publications Before & After IOM Report Stelfox et al. IQSH 2006

9. Personal Experience with Error Blendon et al. NEJM 2002

10. Quality of Trauma Care Medicine’s quality problem includes trauma: Half of all patients do not receive recommended care Medical errors common in critically ill trauma patients 2.5% - 14% trauma deaths in hospital are preventable

11. What is the Quality of Trauma Care?

12. The Challenge – Quality Measurement & Improvement “If you can’t measure it you can’t manage it” Peter Drucker

13. To Address This Challenge Research program to develop population-based & evidence-based indicators of quality of care in critically injured adult patients

14. Research Program International Audit of QI Practices Research Synthesis Years 1-2 Potential Quality Indicators Multi-Step QI Development Process Year 3 Quality Indicators for Evaluation of Implementation Evaluation of Implementation Years 4-5 Final Quality Indicators

15. Objectives • Describe a quality indicator development process • Review strategies for determining the need for quality indicators • Are there opportunities to improve care? • What quality indicators currently exist?

16. Research Synthesis To systematically review the literature about quality indicators (QI) for evaluating trauma care

17. Scoping Review Quality Indicators Stelfox et al. Arch Surg 2010

18. Pediatric Patients Deficiencies in care 8%-45% pts. 6%-32% deaths judged preventable Need for pediatric specific measures No evaluations of validity or reliability Stelfox et al. Crit Care Med 2010

19. Adult Patients Best Indicators Potential Indicators Complications Non-fixation of femur fracture Scene time ↓ LOC & airway management Unplanned return to OR Time to emergency laparotomy ↓ LOC & time to CT head Readmission to hospital Time to basic diagnostics Reintubation <48 hrs Preventable morbidity Missed injury Statistically unexpected death • Preventable death Indicators To Avoid • Time to craniotomy in TBI Stelfox et al. Crit Care Med 2011

20. Impact on Quality of Care *p<0.05 Stelfox et al. Crit Care Med 2011

21. What does the Evidence Tell Us? QI literature for evaluating trauma care: Adults more than children Acute care more than post-acute care Supported by limited scientific evidence A few promising indicators May be associated with improved care

22. Objectives • Describe a quality indicator development process • Review strategies for determining the need for quality indicators • Review the value of environmental scans • How are quality indicators used?

23. Environmental Scan of Trauma Centres • Goal – describe real world trauma centre performance improvement activities • U.S., Canada, Australia & New Zealand • Electronic surveys to 328 trauma centres • 249 centres responded (76%) • Follow up interviews of 76 centres

24. Quality Indicators * * * * p<0.05 Stelfox et al. Ann of Surg 2012 in press

25. Performance Improvement Practices * * * * p<0.05

26. Phase of Care Evaluated by QIs * Structure Process Outcome

27. IOM Aims Evaluated by QIs

28. Do Trauma Centres use the Same Indicators? Santana et al J Trauma 2012 in press

29. 10 Most Common Quality Indicators Santana et al J Trauma 2012 in press

30. How did you decide on your QIs? • Accreditation “A lot of it is driven by formal regulations as what is expected of us as a trauma center either by our State authority or the ACS.” • Local Issues “We had cases with certain issues that were recurrently popping up, and these issues were chosen for monitoring.”

31. How have PI activities impacted your trauma program? • Culture of quality “The whole program is driven by safety and quality.” • Standardized care “Set standards so we have clear expectations on how trauma patients should be care for.” • Improved processes & outcomes of care “It has improved it tremendously, we keep getting better at what we are doing, it helps us drive down our morbidity and mortalities.”

32. How has your trauma program evaluated its PI activities? • No Evaluation “No it’s hard – you can take forever – you can evaluate the evaluation of the evaluation” • Informal Evaluation “Nothing formally, but we meet once a week and go through what’s working, what’s not working, what we want to change.”

33. What can a program do to improve its quality of trauma care? • Better measures: “It’s a vicious circle. If we had the stats to prove we make a difference it would be easier to get funding. But how do you get there?” • Better benchmarking: “In an ideal world I would love to benchmark outcomes, but people don’t use the same data dictionary and it is hard to benchmark when not comparing apples to apples” • More concurrent: “My dream would be that we would track our indicators in real time, review charts within a day or two of them being flagged and fix problems while still active.”

34. Summary of Environmental Scan Observations • Trauma centres spend a lot of energy and time on quality measurement & improvement • Significant variation exists in how trauma centres measure & manage the quality of care they deliver • Significant gaps exist within the observed quality improvement processes

35. What Next?

36. Objectives • Describe a quality indicator development process • Review strategies for determining the need for quality indicators • Review the value of environmental scans • Discuss the role of consensus methodologies • How to bridge the gap between the evidence base and the need for developing applied measures?

37. Quality Indicator Development Potential Quality Indicators Expert Panel Review Round 1 Expert Panel Review Round 2 Expert Panel Workshop Quality Indicators for Evaluation of Implementation

38. Expert Panel

39. Sample Quality Indicator

40. Rating Scale

41. Quality Indicator Development Process 32

42. Research Program International Audit of QI Practices Research Synthesis Years 1-2 Potential Quality Indicators Multi-Step QI Development Process Year 3 Quality Indicators for Evaluation of Implementation Evaluation of Implementation Years 4-5 Final Quality Indicators

43. Stepping BackAre we on target? Gruen et al. BJS 2012

44. Indicators Developed * 1 to 2 indicators per cell of our conceptual model

45. Objectives • Describe a quality indicator development process • Review strategies for determining the need for quality indicators • Review the value of environmental scans • Discuss the role of consensus methodologies • Review lessons learned from the process

46. Lesson #1 Clear Purpose & Goals for the Quality Indicator “...I would submit that the end that we seek here is to try and draw the line between the indicator in question and its measurability and whether or not the patients are going to have better outcomes as a consequence”

47. Lesson #2 Incorporating Evidence, Expertise & Patient Perspectives “The timing I’m not sure about, there’s no class one evidence to support it … At the same time I think there is some value added to early management … prevention of certain secondary complications… that’s why I advocate for it.”

48. Lesson #3 Contextual Considerations & Variation “…again this is going to be a local guidelines issue because different hospitals and systems will triage to their resuscitation room differently depending on their volumes, …[this difference is] just going to influence the wording [of the indicator ]…”  

49. Lesson #4 Data Collection & Management “… anything that encourages the whole trauma system to … improve the data that is submitted … if we’re going to have a measure that forces the issue that we need better data collection, I’m all for it.”

50. One Additional Lesson Take Advantage of Potential Gaps / Opportunities Identified During the Process