1 / 19

Study Design and Outcomes:

Study Design and Outcomes:. Questioning the Quality and Nature of Evidence. Summary. Context given current Health Reform Concato et al Benson et al Implications for the future. Context. Evidence-Based Medicine as a Basis for Comparative Effectiveness

katina
Download Presentation

Study Design and Outcomes:

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Study Design and Outcomes: Questioning the Quality and Nature of Evidence

  2. Summary Context given current Health Reform Concato et al Benson et al Implications for the future

  3. Context Evidence-Based Medicine as a Basis for Comparative Effectiveness Stimulus (Feb 2009) includes $1.1 B for Comparative Effectiveness Peter Orzag1 (Director of the Office of Management and Budget): We are "woefully underinvested in research” Evidence-based medicine will hold down US healthcare costs A guide for reimbursement (CMS) decisions2 Medical device industry Sources: (1) Commonwealth Fund, May 2007 (2) Dhruva et al, New England Journal of Medicine, 2009

  4. Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs Concato et al, New England Journal of Medicine, 2000

  5. Background Sacks et al, 1982 Landmark study comparing published RCTs with observational studies 6 different therapies evaluated: 50 RCTs 56 trials with historical controls Treatment groups had similar rates of outcome Control subjects in observational studies had worse outcomes than RCT controls Efficacy of treatment intervention differed: Observational studies: 79% RCTs: 20% Sacks et al concluded that selection bias skewed results to favor new interventions

  6. Methods Meta-analyses from five major journals: Annals of Internal Medicine, British Medical Journal, Journal of the American Medical Association, The Lancet, New England Journal of Medicine Time period: 1991-1995 Studies classified into (1) RCT only (2) Observational only (3) Both DerSimonian and Laird analytic method used (Fleiss, 1993) Exclusion Criteria: Cohort studies with historical controls Clinical trials with non-random assignment Results not reported as point estimates (RR or OR) or with CI’s 102 meta-analyses: 72 RCT only 24 Observational only 6 Both

  7. Methods 9 clinical topics: 5 topics met eligibility criteria (99 articles, 1,871,681 study subjects) 4 topics were excluded Clinical topics selected: BCG Vaccine and Tuberculosis Screening Mammography and Breast Cancer Mortality Cholesterol Levels and Death due to Trauma Treatment of Hypertension and Stroke Treatment of Hypertension and Coronary Heart Disease

  8. Results: Figure 1

  9. A Comparison of Observational Studies and Randomized, Controlled Trials Benson & Hartz, New England Journal of Medicine, 2000

  10. Background Observational studies are thought to be (Feinstein, 1989): Less Expensive More timely Cover a broader range of patients Several studies done in the 1970-1980s suggest that observational studies exaggerate treatment effects 56% of observational studies had favorable treatment effects vs. 30% in RCT’s (Chalmers et al, 1983)

  11. Methods Observational study search: Medline Cochrane Database of Systematic Reviews Years included: 1985-1998 Journals limited to those cited in the Abridged Index Medicus 3868 articles identified Inclusion criteria for Observational Studies: (1) Non-experimental study (2) Assessed difference between two treatments or one treatment and non-intervention (3) Treatments administered by physicians (4) Study included a control group Articles that met all criteria were searched through Medline (1966-1998) Mantel-Haenszel analytic method (Fidalgo et al, 2008) 19 treatment comparisons (53 observational, 83 RCTs, 2 pseudo-randomized): 7 Cardiologic Treatments 11 Non-Cardiologic Treatments 2 Pseudo-randomized

  12. Results: Figure 1

  13. Results: Figure 2

  14. Conclusions Authors interpreted their findings as different from previous published studies Observational studies not as susceptible to confounding bias as previously thought Despite subjectivity inherent to clinical condition and study selection, in sum data supports the conclusion that observational data can provide high quality data to use as a benchmark

  15. Implications "Half of what you'll learn in medical school will be shown to be either dead wrong or out of date within five years of your graduation; the trouble is that nobody can tell you which half -- so the most important thing to learn is how to learn on your own" -Dr. David Sackett, a pioneer of evidence-based medicine RCTs: Patient selection and cherry-picking Subgroup-analyses Personalized medicine

  16. Guidelines for Care Bhattacharyya et al, Health Affairs, March/April 2009 Medicare pay-for-performance data, hip and knee replacement by orthopedic surgeons 260 hospitals in 38 states Finding: Conforming to or deviating from expert quality metrics had no relationship to actual complications or clinical outcomes Glickman et al, JAMA, 2007 5,000 heart failure patients at 91 hospitals Finding: Application of most federal quality process measures did not change mortality from heart failure

  17. The Practice of Medicine “Human beings are not uniform in their biology. A disease with many effects on multiple organs, like diabetes, acts differently in different people. Information evolves and changes. Rather than rigidity, flexibility is appropriate in applying evidence from clinical trials. To that end, a good doctor exercises sound clinical judgment by consulting expert guidelines and assessing ongoing research, but then decides what is quality care for the individual patient. And what is best sometimes deviates from the norms.” -Jerome Groopman & Pamela Hartzband, Wall Street Journal, 2009

  18. References Commonwealth Fund, May 22nd 2007 Retrieved from: http://www.commonwealthfund.org/Content/Newsletters/Washington-Health-Policy-in-Review/2007/May/Washington-Health-Policy-Week-in-Review---May-29--2007/Orszag-Discusses-New-Ways-of-Alleviating-Soaring-Health-Care-Costs.aspx JL Fleiss Review papers : The statistical basis of meta-analysis Statistical Methods in Medical Research, August 1993; 2: 121 - 145. Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000 Jun 22;342(25):1887-92. Benson K, Hartz AJ A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000 Jun 22;342(25):1878-86. Feinstein AR. Current problems and future challenges in randomized clinical trials. Circulation 1984;70:767-774.  Chalmers TC, Celano P, Sacks HS, Smith H Jr. Bias in treatment assignment in controlled clinical trials. N Engl J Med 1983;309:1358-1361 Fidalgo, Angel M., Madeira, Jaqueline M. Generalized Mantel-Haenszel Methods for Differential Item Functioning Detection Educational and Psychological Measurement 2008 Timothy Bhattacharyya, Andrew A. Freiberg, Priyesh Mehta, Jeffrey Neil Katz, and Timothy Ferris Measuring The Report Card: The Validity Of Pay-For-Performance Metrics In Orthopedic Surgery Health Affairs March/April 2009 - Volume 28, Number 2 Seth W. Glickman; Fang-Shu Ou; Elizabeth R. DeLong; Matthew T. Roe; Barbara L. Lytle; Jyotsna Mulgund; John S. Rumsfeld; W. Brian Gibler; E. Magnus Ohman; Kevin A. Schulman; Eric D. Peterson Pay for Performance, Quality of Care, and Outcomes in Acute Myocardial Infarction JAMA. 2007;297(21):2373-2380. Groopman J and Harzband P. Apr 2008. Why 'Quality' Care Is Dangerous. Wall Street Journal.

More Related