1 / 24

Heterogeneity is not always noise

Heterogeneity is not always noise. Frank Davidoff 29 March 2012. Heterogeneity. Composition from diverse elements or parts; multifarious composition Oxford English Dictionary. The Heterogeneity Problem. Heterogeneity: You can’t live with it, and you can’t live without it.

sherronf
Download Presentation

Heterogeneity is not always noise

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Heterogeneity is not always noise Frank Davidoff 29 March 2012

  2. Heterogeneity is not always noise Heterogeneity • Composition from diverse elements or parts; multifarious composition • Oxford English Dictionary

  3. Heterogeneity is not always noise The Heterogeneity Problem • Heterogeneity: • You can’t live with it, and you can’t live • without it

  4. Heterogeneity is not always noise Today’s territory • How heterogeneity interferes with causal inference in clinical science • How heterogeneity also deepens our knowledge • How the effects of heterogeneity play out differently in improvement science • How we can begin to manage the effects of heterogeneity

  5. Heterogeneity is not always noise Benefit from Drug X: treated populationResults from a standard clinical trial in “ICA” patients Rx benefit: ARR 2 percentage points (pp) RCT

  6. Heterogeneity is not always noise Heterogeneity of treatment effect: main sources • Variation in outcome risk when the primary disease is untreated (mainly biological and behavioral variation) • Treatment-related harm • Competing risk • Direct treatment-effect modification

  7. Heterogeneity is not always noise How summary results of trials can be misleadingHypothetical example Modified from Kent et al, Trials 2010;11:85

  8. Heterogeneity is not always noise Major differences in therapeutic benefitLow- vs. high-risk subgroups in risk stratified analysis • Surgery for carotid stenosis • Anticoagulation in non-valvular atrial fibrillation • CABG for coronary artery disease • Statin therapy as primary prevention in coronary disease • Invasive and non-invasive therapies for acute coronary syndromes • tPA and PCI in ST-elevation myocardial infarction • Drotrecogin in severe sepsis • Kent et al, Trials, 2010:11:85

  9. Heterogeneity is not always noise Benefit from Drug X: high-risk patient subgroupRisk-stratification of results from clinical trial in “ICA” patients RCT ARR 2 pp Risk stratification Rx benefit: ARR 20 pp

  10. Heterogeneity is not always noise Benefit from Drug X: individual high-risk patientReal world results in a “usual” local care system General Hospital - Admin rate 40% Risk stratification ARR 20 pp Rx benefit: ARR 8 pp RCT ARR 2 pp

  11. Heterogeneity is not always noise Benefit from Drug X: individual high-risk patientReal world results in a local care system that successfully supports changes QI Program ??? Rx benefit: 19 pp Risk stratification ARR 20 pp Community Hospital – Admin rate 95% RCT ARR 2 pp

  12. Heterogeneity is not always noise Benefit from Drug X: individual high-risk patientReal world results in a local care system that has trouble supporting changes QI program ??? Risk stratification ARR 20 pp Net benefit: 12 pp RCT ARR 2 pp Proprietary Hospital – Admin rate 60%

  13. Heterogeneity is not always noise Heterogeneity of improvement effect: main sources • Improvement interventions: • Consist of multiple components: hard to standardize; easily mixed and matched

  14. Heterogeneity is not always noise A multi-component improvement intervention: The Michigan ICU central line infection control study • In addition to introducing checklists, prep carts, new skin antiseptic, organizers and leaders: • Recruited advocates within the organization • Kept the team focused on goals • Created alliances with central administration to secure resources • Shifted power relations (particularly with nurses) • Developed social and reputational incentives for cooperating • Opened channels of communication with units that face the same challenges • Used audit and feedback

  15. Heterogeneity is not always noise Heterogeneity of improvement effect: main sources • Improvement interventions: • Consist of multiple components: hard to standardize; easily mixed and matched • Must first be absorbed and adapted: they change in the process (also easily shared, spread)

  16. Heterogeneity is not always noise Heterogeneity of improvement effect: main sources • Improvement interventions: • Consist of multiple components: hard to standardize; easily mixed and matched • Must first be absorbed and adapted: change in the process (also easily shared, spread) • Are context-dependent: context can’t be “controlled out”

  17. Heterogeneity is not always noise Heterogeneity of improvement effect: main sources • Improvement interventions: • Consist of multiple components: hard to standardize; easily mixed and matched • Must first be absorbed and adapted: they change in the process (also easily shared, spread) • Are context-dependent: context can’t be “controlled out” • Are unstable by design: refined over time in response to feedback (“reflexiveness”)

  18. Heterogeneity is not always noise Change factor analysis: detail-level An “ex post” theory of a quality improvement program: Michigan study • Isomorphic (peer) pressure applied to join the project • Networked community formed with strong horizontal links • Bloodstream infections reframed as a social problem • Interventions used to shape a “culture of commitment” • Data harnessed as a disciplinary force • “Hard edges” used • Dixon-Woods et al, Milbank Quarterly, 2011;89:167-205

  19. Heterogeneity is not always noise Change factor analysis: mid-levelImproving survival after acute myocardial infarction: (AMI) • Organizational values and goals • Senior management involvement • Broad staff presence and expertise in AMI care • Communication and coordination among staff groups • Support for staff problem solving and learning • Curry LA, et al. Ann Intern Med 2011;154:384-90

  20. Heterogeneity is not always noise Change factor analysis: high-levelIn-depth field studies in 9 US/UK hospitals • Six universal challenges: structural, cultural, political, educational, emotional, physical & technological • Single factor (even “dominant” set) rarely explains “heterogeneity of improvement effect” • Answers lie in interactions among a multiplicity of factors • Quality a multi-level phenomenon • “Universal but variable” thesis: six challenges same everywhere, but specifics vary within them – the “cityscape phenomenon” • Bate P, et al. Organizing for Quality, 2008

  21. Heterogeneity is not always noise SUMMARY • Heterogeneity is everywhere in medicine • Interferes with detection of causal relationships  noise • BUT • Also key source of information regarding individual risk and outcome  signal

  22. Heterogeneity is not always noise CONCLUSIONS • In order to use heterogeneity as a source of knowledge • In clinical science • Need better techniques for understanding effects of biological and behavioral variation on clinical outcomes • In improvement science • Need better techniques for understanding effects of social factor variation on performance change outcomes • Everyday challenge for everyone • Observe, record, reflect, model, share: you might just come up with the techniques we need

  23. Heterogeneity is not always noise REFERENCES • Davidoff F. Heterogeneity is not always noise. JAMA 2009;302:2580-6. • Kent DM, et al. Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials 2010;11:85 • Provost L. Analytical studies: a framework for quality improvement design and analysis. BMJ Qual Saf 2011;20 [Suppl 1]:i-92-i96. • Dixon-Woods M, et al. Explaining Michigan: developing an ex post theory of a quality improvement program. Milbank Q 2011;89:167-205. • Kaplan HC et al. The Model for Understanding Success In Quality (MUSIQ): building a theory of context in healthcare quality improvement. BMJ Qual Saf 2012;21:13-20. • Curry LA., et al. What distinguishes top-performing hospitals in acute myocardial infarction mortality rates. Ann Intern Med 2011;154:384-90. • Bate P, et al. Organizing for Quality. 2008; New York: Radcliffe Publishing

  24. Heterogeneity is not always noise ACKNOWLEDGMENTSFor helpful comments on this presentation • Yale-New Haven Hospital medical directors leadership council • SQUIRE development group: • David Stevens, Paul Batalden, Greg Ogrinc • Mary Dixon-Woods • Jane Roessner • Jules Hirsch

More Related