1 / 15

Alex Bottle robert.bottle@imperial.ac.uk Imperial College London Dr Foster Unit

A method for estimating the cost of reducing the false alarm rate in multi-institution performance monitoring using CUSUM charts. Alex Bottle robert.bottle@imperial.ac.uk Imperial College London Dr Foster Unit. Overview.

Download Presentation

Alex Bottle robert.bottle@imperial.ac.uk Imperial College London Dr Foster Unit

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. A method for estimating the cost of reducing the false alarm rate in multi-institution performance monitoring using CUSUM charts Alex Bottle robert.bottle@imperial.ac.uk Imperial College London Dr Foster Unit

  2. Overview • Background: cumulative sum charts and nationwide NHS mortality monitoring tool • Extent of multiple testing • Factors affecting false alarm rate • Simulation for false alarm and successful detection rates • Estimation of ‘cost’: worked example for AMI • Summary

  3. CUSUM chart essentials • Plots one patient at a time • Chart statistic (log-likelihood ratio) goes up if patient dies and down if patient survives • Chart rises faster if low-risk patient dies • If crosses preset threshold, chart ‘signals’ • Threshold choice involves consideration of type I and type II error rates

  4. CUSUM charts

  5. Mortality monitoring tool • In use in ~100 acute hospitals in England • Compares each hospital’s case-mix adjusted mortality rate with national average • Tests for an odds ratio of at least 2 • Displayed using cumulative sum charts • Data are updated monthly

  6. Mortality monitoring tool: opening screen

  7. Extent of multiple testing Over time: threshold handles this element But… At each hospital trust each month: • 78 diagnosis groups • >100 procedure groups National monitoring incurs further ‘cost’: • ~150 acute hospital trusts • Consultant-level monitoring?

  8. Factors affecting the false alarm rate • Threshold: the higher this is set, the lower the false alarm rate • Length of monitoring: number of patients varies by hospital and diagnosis • Expected mortality rate: e.g. 5% rates will have high FAR than 1% rates • Size of increase (OR) to be detected (not considered here)

  9. Research question • A higher chart threshold -> lower FAR but slower detection of high mortality rates • Compared with the conventional 5% false alarm rate, what is the ‘cost’ of having a lower false alarm rate (1% or 0.1%) to deal with all the multiple testing?

  10. Simulation: FAR and SDR • For FAR, generate 5,000 artificial hospitals with mortality rate p • Do this for various p, p=0.1% to 30% • Calculate FAR after t patients, t in steps of 5 from 5 to 20,000 • Do this for different thresholds h, h=0.5 to 15 • For SDR, generate hospitals with rate 2p/(1-p)

  11. Using the simulation to estimate ‘cost’ • For each dx, work out the threshold h needed for FAR of 5% at average hosp • Find number of monitored patients t needed for SDR of 80% using threshold h • Knowing the dx’s expected death rate and OR to be detected, convert t into a number of deaths • Repeat for FAR of 1% and 0.1% • Find the difference in number of deaths between the pairs of FAR values

  12. ‘Cost’ calculation for AMI in England (1) • National death rate=11.8%. Average number of AMIs per hospital=467 • For FAR=5%, h=5.2 -> t=185 for SDR=80% • At rate p, this means 21.8 deaths • At rate 2p/(1-p), this means 39.0 deaths • ‘Excess’ deaths: 39.0 – 21.8 = 17.2

  13. ‘Cost’ calculation for AMI in England (2) • For FAR=0.1%, h=8.6 -> t=305 for SDR=80% • At rate p, this means 36.0 deaths • At rate 2p/(1-p), this means 64.4 deaths • ‘Excess’ deaths: 64.4 – 36.0 = 28.4 • ‘Cost’ of lowering FAR to 0.1% = 28.4 – 17.2 = 11.2 extra deaths at average hosp

  14. Findings for AAA repair and CABG • AAA repair: less common but high risk • ‘Cost’=6.3 for FAR=0.1%, 2.4 for FAR=1% • CABG: common but low risk • ‘Cost’= 6.8 for FAR=0.1%, 2.8 for FAR=1% • These are all figures for an average hospital

  15. Summary • Multiple testing can be addressed by lowering the false alarm rate: raise the threshold for CUSUM charts • Other approaches include minimising ‘loss function’ or maximising ‘desirability function’ • The proposed measure of ‘cost’ depends on mortality rate and hospital volume • The ‘cost’ can be derived from simulation and is intuitive to less-technical users

More Related