Heterogeneity is not always noise

1. Heterogeneity is not always noise Frank Davidoff 29 March 2012

2. Heterogeneity is not always noise Heterogeneity • Composition from diverse elements or parts; multifarious composition • Oxford English Dictionary

3. Heterogeneity is not always noise The Heterogeneity Problem • Heterogeneity: • You can’t live with it, and you can’t live • without it

4. Heterogeneity is not always noise Today’s territory • How heterogeneity interferes with causal inference in clinical science • How heterogeneity also deepens our knowledge • How the effects of heterogeneity play out differently in improvement science • How we can begin to manage the effects of heterogeneity

5. Heterogeneity is not always noise Benefit from Drug X: treated populationResults from a standard clinical trial in “ICA” patients Rx benefit: ARR 2 percentage points (pp) RCT

6. Heterogeneity is not always noise Heterogeneity of treatment effect: main sources • Variation in outcome risk when the primary disease is untreated (mainly biological and behavioral variation) • Treatment-related harm • Competing risk • Direct treatment-effect modification

7. Heterogeneity is not always noise How summary results of trials can be misleadingHypothetical example Modified from Kent et al, Trials 2010;11:85

8. Heterogeneity is not always noise Major differences in therapeutic benefitLow- vs. high-risk subgroups in risk stratified analysis • Surgery for carotid stenosis • Anticoagulation in non-valvular atrial fibrillation • CABG for coronary artery disease • Statin therapy as primary prevention in coronary disease • Invasive and non-invasive therapies for acute coronary syndromes • tPA and PCI in ST-elevation myocardial infarction • Drotrecogin in severe sepsis • Kent et al, Trials, 2010:11:85

9. Heterogeneity is not always noise Benefit from Drug X: high-risk patient subgroupRisk-stratification of results from clinical trial in “ICA” patients RCT ARR 2 pp Risk stratification Rx benefit: ARR 20 pp

10. Heterogeneity is not always noise Benefit from Drug X: individual high-risk patientReal world results in a “usual” local care system General Hospital - Admin rate 40% Risk stratification ARR 20 pp Rx benefit: ARR 8 pp RCT ARR 2 pp

11. Heterogeneity is not always noise Benefit from Drug X: individual high-risk patientReal world results in a local care system that successfully supports changes QI Program ??? Rx benefit: 19 pp Risk stratification ARR 20 pp Community Hospital – Admin rate 95% RCT ARR 2 pp

12. Heterogeneity is not always noise Benefit from Drug X: individual high-risk patientReal world results in a local care system that has trouble supporting changes QI program ??? Risk stratification ARR 20 pp Net benefit: 12 pp RCT ARR 2 pp Proprietary Hospital – Admin rate 60%

13. Heterogeneity is not always noise Heterogeneity of improvement effect: main sources • Improvement interventions: • Consist of multiple components: hard to standardize; easily mixed and matched

14. Heterogeneity is not always noise A multi-component improvement intervention: The Michigan ICU central line infection control study • In addition to introducing checklists, prep carts, new skin antiseptic, organizers and leaders: • Recruited advocates within the organization • Kept the team focused on goals • Created alliances with central administration to secure resources • Shifted power relations (particularly with nurses) • Developed social and reputational incentives for cooperating • Opened channels of communication with units that face the same challenges • Used audit and feedback

15. Heterogeneity is not always noise Heterogeneity of improvement effect: main sources • Improvement interventions: • Consist of multiple components: hard to standardize; easily mixed and matched • Must first be absorbed and adapted: they change in the process (also easily shared, spread)

16. Heterogeneity is not always noise Heterogeneity of improvement effect: main sources • Improvement interventions: • Consist of multiple components: hard to standardize; easily mixed and matched • Must first be absorbed and adapted: change in the process (also easily shared, spread) • Are context-dependent: context can’t be “controlled out”

17. Heterogeneity is not always noise Heterogeneity of improvement effect: main sources • Improvement interventions: • Consist of multiple components: hard to standardize; easily mixed and matched • Must first be absorbed and adapted: they change in the process (also easily shared, spread) • Are context-dependent: context can’t be “controlled out” • Are unstable by design: refined over time in response to feedback (“reflexiveness”)

18. Heterogeneity is not always noise Change factor analysis: detail-level An “ex post” theory of a quality improvement program: Michigan study • Isomorphic (peer) pressure applied to join the project • Networked community formed with strong horizontal links • Bloodstream infections reframed as a social problem • Interventions used to shape a “culture of commitment” • Data harnessed as a disciplinary force • “Hard edges” used • Dixon-Woods et al, Milbank Quarterly, 2011;89:167-205

19. Heterogeneity is not always noise Change factor analysis: mid-levelImproving survival after acute myocardial infarction: (AMI) • Organizational values and goals • Senior management involvement • Broad staff presence and expertise in AMI care • Communication and coordination among staff groups • Support for staff problem solving and learning • Curry LA, et al. Ann Intern Med 2011;154:384-90

20. Heterogeneity is not always noise Change factor analysis: high-levelIn-depth field studies in 9 US/UK hospitals • Six universal challenges: structural, cultural, political, educational, emotional, physical & technological • Single factor (even “dominant” set) rarely explains “heterogeneity of improvement effect” • Answers lie in interactions among a multiplicity of factors • Quality a multi-level phenomenon • “Universal but variable” thesis: six challenges same everywhere, but specifics vary within them – the “cityscape phenomenon” • Bate P, et al. Organizing for Quality, 2008

21. Heterogeneity is not always noise SUMMARY • Heterogeneity is everywhere in medicine • Interferes with detection of causal relationships  noise • BUT • Also key source of information regarding individual risk and outcome  signal

22. Heterogeneity is not always noise CONCLUSIONS • In order to use heterogeneity as a source of knowledge • In clinical science • Need better techniques for understanding effects of biological and behavioral variation on clinical outcomes • In improvement science • Need better techniques for understanding effects of social factor variation on performance change outcomes • Everyday challenge for everyone • Observe, record, reflect, model, share: you might just come up with the techniques we need

23. Heterogeneity is not always noise REFERENCES • Davidoff F. Heterogeneity is not always noise. JAMA 2009;302:2580-6. • Kent DM, et al. Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials 2010;11:85 • Provost L. Analytical studies: a framework for quality improvement design and analysis. BMJ Qual Saf 2011;20 [Suppl 1]:i-92-i96. • Dixon-Woods M, et al. Explaining Michigan: developing an ex post theory of a quality improvement program. Milbank Q 2011;89:167-205. • Kaplan HC et al. The Model for Understanding Success In Quality (MUSIQ): building a theory of context in healthcare quality improvement. BMJ Qual Saf 2012;21:13-20. • Curry LA., et al. What distinguishes top-performing hospitals in acute myocardial infarction mortality rates. Ann Intern Med 2011;154:384-90. • Bate P, et al. Organizing for Quality. 2008; New York: Radcliffe Publishing

24. Heterogeneity is not always noise ACKNOWLEDGMENTSFor helpful comments on this presentation • Yale-New Haven Hospital medical directors leadership council • SQUIRE development group: • David Stevens, Paul Batalden, Greg Ogrinc • Mary Dixon-Woods • Jane Roessner • Jules Hirsch