1 / 56

From Clinical Trial Evidence to Practice Guidelines Lost in Translation

From Clinical Trial Evidence to Practice Guidelines Lost in Translation. Sanjay Kaul, MD, FACC George A. Diamond MD, FACC Division of Cardiology Cedars-Sinai Medical Center and Geffen School of Medicine at UCLA Los Angeles, California. Complexity of American Strategy in Afghanistan .

stew
Download Presentation

From Clinical Trial Evidence to Practice Guidelines Lost in Translation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. From Clinical Trial Evidence to Practice Guidelines Lost in Translation Sanjay Kaul, MD, FACCGeorge A. Diamond MD, FACCDivision of CardiologyCedars-Sinai Medical Center and Geffen School of Medicine at UCLALos Angeles, California

  2. Complexity of American Strategy in Afghanistan “When we understand that slide, we’ll have won the war” - General McChrystal

  3. Complexity of Evidence-Based Medicine Lost in a jungle of evidence, we need a compass

  4. Evidence to Guidelines Lost in TranslationKey Issues for Discussion • Establish the scientific evidence • - Appraise and synthesize the evidence • Elucidate the clinical context • - Clinical importance vs. statistical significance • - Clinically relevant weighted outcomes • Encourage optimal processes of care • - Quality initiatives • - Reimbursement initiatives

  5.  Quality Quality • Important limitations • - Study design or execution (bias)-Lack of randomization • - Lack of concealment • - ITT principle violated • - Inadequate blinding • - Loss to follow-up • - Early stopping for benefit • - Inconsistency of results • - Indirectness of results • - Imprecision • - Publication bias • Special strengths • - Randomized, controlled, prospective, double-blind trials • - Large, consistent, and precise treatment effect- RR<0.5 or >2.0 (large) - RR<0.2 or >5.0 (very large) • - Minimal confounding & bias Appraisal of Evidence Design and Methods

  6. Synthesis of Evidence ACC/AHA Clinical Practice Guidelines

  7. Evidence to Guidelines Lost in TranslationSelf-evident Truths • Does empirical evidence trump expert opinion?

  8. AF 11.7 Heart failure 26.4 PAD 15.3 STEMI 13.5 Perioperative 12 Secondary prevention 22.9 Stable angina 6.4 SV arrhythmia 6.1 UA/NSTEMI 23.6 Valvular disease 0.3 (1/320) VA/SCD 9.7 PCI 11 CABG 19 Pacemaker 4.9 Radionuclide imaging 4.9 0 10 20 30 Scientific Evidence Underlying The ACC/AHA Clinical Practice Guidelines Level of Evidence A Tricoci P et al. JAMA 2009

  9. Scientific Evidence Underlying The ACC/AHA Clinical Practice Guidelines Level of Evidence C Tricoci P et al. JAMA 2009

  10. ACC/AHA Clinical Practice Guidelines Paucity of High-Quality Evidence Recommendations are largely developed from lower levels of evidence or expert opinion. “Exercise caution when considering recommendations not supported by solid evidence” Tricoci P et al. JAMA 2009

  11. Scientific Evidence Underlying The ACC/AHA Clinical Practice Guidelines Caveat Emptor, Caveat Lector “…it seems unlikely that substantial change will occur because many guideline developers seem set in their ways. If all that can be produced are biased, minimally applicable consensus statements, perhaps guidelines should be avoided completely. Unless there is evidence of appropriate changes in the guideline process, clinicians and policy makers must reject calls for adherence to guidelines. Physicians would be better off making clinical decisions based on valid primary data”Shaneyfelt and Centor, JAMA 2009 Guidelines that are driven by scientifically documented, high-quality evidence are more likely to be accepted by the stakeholders, thereby reducing the variability in care and improving the quality and cost of care

  12. 2009 ACC/AHA Focused Updates for STEMI/PCI Paucity of High-Quality Evidence Kushner FG, Hand M et al. 2009 Focused Updates, JACC/Circulation 2009

  13. The Laws of Diminishing Objectivity in the Interpretation of Evidence • vehemence  evidence-1 • vehemence  eminence2 Peter McCulloch The Lancet, 2004;363;9004

  14. Evidence to Guidelines Lost in TranslationKey Issues for Discussion • Establish the scientific evidence • - Appraise and synthesize the evidence • Elucidate the clinical context • - Clinical importance vs. statistical significance • - Clinically relevant weighted outcomes • Optimal processes of care • - Quality initiatives • - Reimbursement initiatives

  15. ACC/AHA Clinical Practice Guidelines Metrics for Assessing Strength of Evidence • Effect size- Absolute risk difference (NNT or NNH) • - Relative risk differenceRisk ratio Odds ratio • Hazard ratio • Statistical certainty/precision • - Hypothesis testing (P value) • - Estimation (confidence interval) • ? Clinical importance Little or no explicit guidance

  16. Disconnect Between Statistical Significance and Clinical Importance 1 P value  Effect Size  Sample Size Statistical significance  Clinical importance!

  17. Death / MI at 30 days Risk Ratio & 95% CI 2b/3a Placebo Trial (IIb/IIIa) N (%) (%) GUSTO IV PRISM PRISM-Plus PURSUIT PARAGON A PARAGON B POOLED 7800 3232 1570 9461 1513 5169 28,745 8.7 5.7 8.7 14.2 10.3 10.5 11.3 8.0 7.0 11.9 15.7 11.7 11.4 12.5 0.91 (0.86, 0.99) P=0.015 ARR = 1.2% RRR = 9% P=0.339 Breslow-Day Homogeneity 0.1 1 Better Worse 10 Boersma et al, Lancet 2002;359:189-1198. Statistical Significance vs. Clinical Importance GP IIb/IIIa Inhibitors in UA/NSTEMI

  18. What Does a P(ee) Value of 0.05 Mean? • ‘Fisherian’ P value of 0.05 • is arbitrary and originally based on n=30! • Always demand a P value of <0.001 for a sample size> 200 as strong evidence against the null hypothesis of zero differenceAl Feinstein The plain fact is that in 1925 Ronald Fisher gave scientists a mathematical machine for turning “baloney into breakthroughs”, and “flukes into funding”. Robert Matthews

  19. Disconnect Between Statistical Significance and Clinical Importance 1 P value  Effect Size  Sample Size Lack of statistical significance  lack of clinical importance!

  20. Death/MI Trial N Risk Ratio & 95% CI ASA+UFH ASA 1.6% 3.3% 1.4% 3.7% 3.8% 8.3% 27.3% 30.5% 0% 3% 5.7% 9.6% Theroux 243 RISC 399 ATACS 214 Holdright 285 Cohen 1990 69 Gurfinkel 143 0.67 (0.44-1.02) P=0.06 7.9% 10.4% Overall 1335 0.1 1.0 10 ASA+UFH Better ASA Better ARR = 2.5% RRR = 33% Statistical Significance vs. Clinical Importance Unfractionated Heparin in UA/NSTEMI Oler A et al, JAMA 1996;276:811-15

  21. Statistical Significance vs. Clinical Importance • MDD (minimum detectable difference, “d”) - The “minimum difference” the study is powered to detect- Utilized for sample size estimation - May or may not reflect a clinically important difference • MCID (minimum clinically important difference) The “minimum acceptable difference” to change the behavior of the clinician, patient, payer or policy maker, given the side effects, costs and inconveniences of therapeutic interventions

  22. Small Large 0% 50% MCID (RRR) Harm Very low Low Moderate High Outcomeseverity Mortality Surrogate Endpoint Reversible Morbidity Irreversible morbidity Cost Very low Low Moderate High Guideline Criteria for Clinical Importance Impact of Outcome, Harm, and Cost on MCID

  23. Statistical Significance vs. Clinical Importance MCID Threshold for UA/NSTEMI ACS “In ACS, a relative reduction of 15% in recurrent clinical events has recently been considered clinically important (GUSTO I); this level is far below the perceived threshold that drove the sample size calculations for clinical trials just a decade ago. As we develop more incrementally beneficial therapies, it is likely that the minimally important clinical difference will become even smaller.” Califf and DeMets Circulation. 2002;106:1015

  24. Statistical Significance vs. Clinical Importance Strength of Evidence MCID Statistically not significant, clinically not important A Statistically not significant, may be clinically important B Statistically significant, not clinically important C Statistically significant, may be clinically important D Statistically significant, clinically important E MCID = minimal clinically important difference= 15% RRD 1.0 0.85 Risk Ratio (95% CI) Sackett, D

  25. Statistical Significance vs. Clinical Importance Class I, LOE A Recommendations for UA/NSTEMI Impact on Death or MI Aspirin is the only intervention listed as a performance measure!

  26. Evidence to Guidelines Lost in TranslationKey Issues for Discussion • Establish the scientific evidence • - Appraise and synthesize the evidence • Elucidate the clinical context • - Clinical importance vs. statistical significance • - Clinically relevant weighted outcomes • Encourage optimal processes of care • - Quality initiatives • - Reimbursement initiatives

  27. Major Adverse Cardiac Events Minor Inconvenient Cardiac Events “Soft” but prevalent “Hard” but infrequent Silent CK/Tn Release Restenosis Reintervention Recurrent angina Rehospitalization Groin hematoma Death Cardiac arrestLarge MIDisabling Stroke Emergency CABG Endpoints in Cardiovascular Clinical Trials MACE vs MICE

  28. Placebo Antihistamine p < 0.05 % patients Death Recurrent MI Itching Cardioprotective Effects of Antihistamines Means to an End or an End to Means

  29. D/MI/Stroke/TVR Individual Components 25 Stent (N=452) P < 0.005 20.9 PTCA (N=448) 20 P < 0.0005 15 % 10.7 10 P = 0.07 P = 0.7 5.6 P = 0.83 5 3.1 2.9 2.5 0.5 0.5 0 Death Stent (N=449) MI Stroke TVR PTCA (N=444) Cardioprotective Effects of StentingClinical Outcomes at 1 Year in Stent PAMI Benefit driven by the “least robust” but the “most prevalent” component

  30. Validity of the Composite Endpoint • Components should be of comparable frequency • Components should be of comparable clinical importance • Components should be comparably responsive to therapy Montori VM et al. Br Med J 2005; 330:594-596

  31. OR (95% CI) Stent PTCA Death 1.81 (0.93-3.53) 5.6% 3.1% MI 1.17 (0.52-2.65) 2.9% 2.5% Stroke 0.99 (0.14-7.05) 0.5% 0.5% TVR 0.45 (0.31-0.66) 10.7% 20.9% Composite analysis 16.9% 24.8% 0.62 (0.45-0.86) 0.00 1.00 2.00 3.00 4.00 Odds ratio Cardioprotective Effects of Stenting Validity of the Composite Endpoint in Stent PAMI Cochran’s Q = 14.64 Hetero P = 0.002 I2 = 80% (46-92%) Composite: Variable gradient in clinical importance, frequency and treatment effect across components

  32. Cardioprotective Effects of Stenting Weighted Analysis of Composite Endpoint 1.00 Weights Death = 1 MI = 1 Stroke = 1 0.90 0.80 0.70 0.60 Global P value 0.50 0.40 0.30 0.20 0.10 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 Weight of TVR Composite endpoint becomes significant at a TVR weight of >0.7!

  33. ACC/AHA Guideline Recommendations Prasugrel During Primary PCI for STEMI Kushner FG, Hand M et al. 2009 Focused Updates, JACC/Circulation 2009

  34. Benefit-Risk Balance in TRITON (All ACS Cohort) 1000 Patients Treated with prasugrel instead of clopidogrel Prasugrel vs Clopidogrel Benefit Risk • 24 endpoints prevented • - 3 CV deaths • - 0 strokes • - 21 nonfatal MIs - 4 PPMIs - 17 MI events- 13 clinically relevant MIs • 30 excess TIMI bleeds • - 2 bleeding deaths- 3 TIMI Major bleeds - 5 TIMI Minor bleeds • - 20 TIMI Minimal bleeds • or • 29 excess moderate/severe bleeds • - 2 bleeding deaths - 9 transfusions - 6 nonfatal serious bleeds • - 12 nonfatal moderate bleeds • or • 17 excess serious bleeds • ? 3-6 excess cancer (1 cancer death)

  35. Judgments about Strength of RecommendationPrasugrel for Patients with ACS Undergoing PCI Does the evidence favor Class I (benefit >>> risk)recommendation for prasugrel?

  36. Evidence to Guidelines Lost in TranslationKey Issues for Discussion • Establish the scientific evidence • - Appraise and synthesize the evidence • Elucidate the clinical context • - Clinical importance vs. statistical significance • - Clinically relevant weighted outcomes • Encourage optimal processes of care • - Quality initiatives • - Reimbursement initiatives

  37. Quality MattersLinking Guidelines Adherence and Mortality Every 10%  in guidelines adherence  10%  in mortality (OR=0.90, 95% CI: 0.84-0.97) Peterson et al, JAMA 2006;295:1863-1912

  38. GRACE: Outcome Measures over TimeNSTE ACS Changes in Clinical Outcomes for NSTE ACS Patients • Risk-adjusted hospital deaths declined by 0.7 percentage points (95% CI, • -1.7 to 0.3) in NSTE ACS patients. • The rate of congestive heart failure and pulmonary edema decreased by 6.5% (95% CI, -8.4 to -4.7). p <.001 p = .02 for linear trend n =1566 n = 2228 n = 1564 n = 2213 Fox et al. JAMA. 2007; 297:1892-1900

  39. ACC Improvement InitiativesContinuous Quality Improvement PLAN Education and Training DO ACT STUDY Translating Science into Practice

  40. Fuel Boost The Role of Evidence-Based Guidelinesin Improving Clinical Practice Turbocharging the Guidelines High-quality evidence Implementation Design, process evaluation

  41. 2007 ACC/AHA Guideline Recommendations Acute Coronary Syndromes • Number of recommendations: >250 • Number of pages: 157 • Number of figures: 21 • Number of tables: 26 • Last update: 2002 • Writing committee members: 15 • Reviewers: 40 (6 different layers from evaluation to publication) • Conflict-of-interest disclosure - Writing committee members: 14/15 - Reviewers: 30/40 J Am Coll Cardiol 2007; DOI:10.1016/j.jacc.2007.02.028

  42. Evidence to Guidelines Framework for Refinement • Quality- Rigorous and standardized methodology (GRADE) • - Emphasize clinical importance over statistical significance - Transparent and explicit benefit-risk assessment • Efficiency • - User-friendly and parsimonious (avoid the 160 page report) • Timeliness • - Keep pace with advances (annual updates) • Dissemination- Direct clinical relevance (at point of care via EMR) • - Guide and inform clinical practice (performance measures)- Financial incentives (evidence-based reimbursement)

  43. Evidence to Guidelines Framework for Refinement • Firewall between systematic review & guideline development • Multidisciplinary guideline developers: methodologists, clinical content experts, patient representatives • Avoid LOE C recommendations (best suited as “advisories”) • Minimize conflicts of interest (COI) for writers/reviewers • “Zero tolerance” COI policy for chairs • PIs of guideline-relevant trials should only serve as advisors

  44. Evidence-Based MedicineACC Improvement Initiatives • Turbocharging guidelines (18 currently available, 9 in development, 6 being updated) • Transform and transfer guidelines at the point of care - Just in time strategies (Vivisimo, Cardio Compass) • Appropriate use criteria (Noninvasive imaging, CABG/PCI) • Quality initiatives (D2B, H2H, FOCUS) • Registries - NCDR (CathPCI, ICD, CARE, ACTION-GWG, IMPACT, PINNACLE) • Physician incentives (PQRI, ACO) • Patient involvement (CardioSmart)

  45. Framework for Increased Adherence to Clinical Practice Guidelines and to EBM • Treat as “guides”, not “rules” • Patient-specific, not disease-specific • Pragmatic/assistive, not prescriptive/directive • Flexible and adapted to local practice • Based on empirical high-quality evidence, not “codified” or “filtered” expert clinical opinion • Drive the standard of care, not be driven by them • Inform clinical judgment, not replace it

  46. “Evidence-Based” Not “Evidence-Bound” Three Key Dimensions Scientific evidence Patient preference Clinical Judgment

  47. Complexity of Evidence-Based Guidelines Illusion of understanding? Illusion of control?

  48. "Yes, I have tricks in my pocket, I have things up my sleeve. But I am the opposite of a stage magician. He gives you illusion that has the appearance of truth. I give you truth in the pleasant disguise of illusion." Tennessee Williams (The Glass Menagerie)

  49. Caveats in Interpretation of Meta-analysis “Although it challenges logic that one could obtain new accurate information from the quantitative integration of a number of very diverse studies, the numerous meta-analyses published speak for themselves. Used in the proper setting, I think they can make a valuable contribution. The job of the Journal will be to ensure that those published are in this setting and are methodologically sound.” Anthony N DeMaria, MD Editor-in-Chief, JACC Has the Journal lived up to its ideals? J Am Coll Cardiol, 2008; 52:237-238

More Related