Quality Improvement Pitfalls and Opportunities

1. Quality ImprovementPitfalls and Opportunities James Hallenbeck, MD ACOS/Extended Care VA Palo Alto HCS

2. Objectives Be able to……. • Compare top-down and grassroots strategies for quality improvement in terms of advantages and disadvantages • Discuss different uses of quality monitors • Comparisons across groups • Quality improvement • Better respond to queries regarding monitors, especially performance measures, especially MDS

3. How this might be useful to you… • Self defense - protecting you and your program when data does not accurately reflect the quality of care you deliver • Promote better use of monitors (MDS and others) for quality improvement

4. Purposes of Quality Improvement Monitors • Comparisons • VISNs, facilities, wards, clinicians • Stimulus for quality improvement efforts • Benchmarking and ongoing evaluation of quality improvement efforts

5. What Makes a Good Monitor?(Top-down or grassroots) • Desired outcome strongly linked to what is measured (i.e. minimally influenced by factors outside the health system’s control) • System change, informed by monitor, would likely result in better outcomes • Accuracy • Minimal false negatives & false positives

6. Enables comparisons across large systems Establishes organizational priorities Buy-in from top management If successful, results in broad, measurable improvement Blind to local issues By attending to certain priorities, others may be ignored Lack of buy-in from front-line staff Need large numbers Disempowerment at local levels QI INDICATORSTOP TO BOTTOM Advantages: Disadvantages:

7. What Makes a Good Top-Down Monitor? • Simple, discrete data • Counts of things • Minimal variation in local systems and populations • Comparing apples to apples • Large data numbers – in numerator and denominator • Data gathering, measurement and comparison automatic • Encourages GOOD clinical practice (not just designed to identify bad practice)

8. Examples Good Top-Down Monitors • Flu vaccine administration • Appointment waiting time

9. Tailored to local circumstances Better buy-in from front-line staff Can improve morale through empowerment Comparing outcomes across systems difficult Potential lack of buy-in from top management Harder to disseminate best practices Politically, invisible QIGrassroots Disadvantages Advantages

10. Examples Grassroots QI • Use of out-of-hospital DNR forms • Addressing options of autopsy/organ donation

11. Apples to Apples Problem… • Case-mix issues • Severity of illness (Problematic: pressure ulcers, weight loss, dehydration in Hospice) • Gender (Problematic: UTI rates) • Depression/atypical neuroleptics (Problematic: geropsych units, hospice)

12. Vertical Hierarchy Problem • What is “true” at one level of a vertical hierarchy is not necessarily “true” at another (higher or lower) • Example: Newtonian physics versus quantum mechanics • 30,000 feet view: reveals certain patterns, otherwise invisible, but obscures other

14. Vertical Hierarchy Nation VISN 1 VISN 2 VISN 3 Facility 2 Facility 1 Facility 3 Facility 4 Division/Ward Division/Ward Clinician + Patient Clinician + Patient Clinician + Patient

15. Connection Between What is Measured and Desired Outcome Problem • Many outcomes (good and bad) minimally dependent on what we as clinicians do • Examples: • UTI rates • Fall rates Important to tease out those aspects of the outcome that are dependent on the healthcare system. Ideally, these aspects should be measured and worked on.

16. Environment Luck Patient Variables Healthcare System Clinician Patient Outcomes Good Bad Ugly

17. Example: Falls 1 month data: 47 falls • Falls related to transfers/ambulation---18 • Falls related to toileting—13 • Falls related to reaching/picking/bending—7 • Claimed did not fall/eased self to floor--5 • Slipped/tripped-2 • Falls related to medical condition---2 • Falls related to inappropriate footwear-2 Can you tell from this data how these falls might be related to the healthcare system?

18. Numerator/Denominator Problem • Percentages meaningless in isolation (numerator/denominator not given) • Small numbers – 1 – 10 always suspicious • Numerator problem: rare events • Denominator problem: • Restricted population based on: • Geography (single ward) • Time interval (monthly tracking)

19. Oct ‘04 - Jun ‘05 Total FALLS REPORT

20. Coin-Toss QI • 10 Facilities toss a coin 10 times • Heads is Good, Tails is Bad • True chance (within infinite tosses) = 50% • Let’s compare facilities to “national” data

21. Coin Toss “Problem” Facilities “Exceptional” Facilities Of 100 tosses 56/100 Heads = “National” average

22. Implications Unadjusted percentages create a bias in which small facilities/programs with small numbers to report for a given indicator are more likely to be classified at extremes (failing or exceptional)

23. Data Validity Problem • Ideal: • Automated • Unambiguous results • Problematic: • Multiple people/services responsible for data entry • Results dependent on subjective judgments • Questionable inclusion/exclusion criteria

24. MDS Examples • Dehydration monitor: • Little or no activity monitor • Bedfast Status monitor

25. Blue = sub acute and hospice wards, Red = national

27. DehydrationSo, What’s the Problem Here?

28. MDS Quality Indicators:What does the literature show? • MDS tends to under-report problems • Example: depression: • Differences in rates often reflect either differences in reporting or case mix – NOT quality • Example: High/Low Pressure Ulcer NH did not differ significantly in care processes (Bates-Jensen) • Little evidence that MDS indicators per se useful in improving processes of care (Rantz)

30. Comparison to Published Rates • Percentage of NH depressed on MDS: ~ 10-36% • Gold standard comparisons: ~ 46-55% (Schnelle, Simmons) “The MDS depression quality indicator underestimates the prevalence of depressive symptoms in all homes, but in particular, among those reporting low or nonexistent rates” • Percentage of depressed patients without antidepressant treatment: 45% (Brown) Note: not validated against pharmacy database

31. Suggestions for Using MDS (and similar) Monitors for QI • Consider the good intent of the monitor • It would be bad to have patients starving or thirsty, unnecessarily incontinent, subject to polypharmcy, inappropriately treated with antipsychotics, who never get out of bed and are bored to tears • Step Back from the individual and the percentages and ask, “what part of this outcome might we have some influence on? • Brainstorm with staff

32. Suggestions for Using MDS (and similar) Monitors for QI • Consider feasibility of possible interventions • Benchmark interventions • Note changes following intervention • Revise intervention

33. Example: Falls • Transfers/ambulation: • Wheelchair breaks working and routinely inspected on patients’ wheelchairs? • Toileting • Nightlights available, sleepers, patient instructions to call for assistance • Slipped/tripped • Inspect rugs, walkways especially during rainy/snowy seasons • Falls related to medical condition • Review drug therapy for high-risk patients • Inappropriate footwear • Inspect patients’ footwear, none-slip socks and slippers

34. Summary • Top-down monitors by themselves WILL NOT result in improved care delivery • Top-down monitors work best when combined with grass-roots efforts to improve care • Where monitors work poorly for a given purpose, it is our obligation to say so – and then work to make things better