1 / 22

Erik Fernandez y Garcia , MD MPH University of California, Davis

Subgroup Reporting in the General Medical Literature: Do Investigators Misinterpret Their Own Findings?. Erik Fernandez y Garcia , MD MPH University of California, Davis

prunella-ronny
Download Presentation

Erik Fernandez y Garcia , MD MPH University of California, Davis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Subgroup Reporting in the General Medical Literature: Do Investigators Misinterpret Their Own Findings? Erik Fernandez y Garcia, MD MPH University of California, Davis Co-authors: Hien Nguyen, MD (UCD); Naihua Duan, PhD (Columbia University); Nicole Bloser Gabler, MHA MPH (UCD); Diana Liao MPH(UCLA); Richard L. Kravitz, MD MSPH (UCD) Supported by a grant from Pfizer.

  2. RATIONALE • Randomized-controlled trials (RCTs) yield an average treatment effect. • In a trial, treatments may have different net benefits and harms for different patients (heterogeneity of treatment effects, orHTE). • Examination of the HTE is important for optimizing treatment for individual patients. • Especially critical in era of increasing population diversity and health disparities.

  3. RATIONALE • Subgroup analysis (SGA) in RCTs is one way of investigating HTE. • The usefulness of SGA is hampered by the problems of • insufficient power (false negative) • multiple testing (false positive)

  4. RATIONALE • Under-use: SGA not performed in studies with sufficient power and theoretical rationale to anticipate helpful results. • Over-use: SGA performed in studies which were underpowered or lacked theoretical rationale to anticipate helpful results. • Misuse: SGA was performed in the appropriate setting but with inappropriate methodology and/or interpretation.

  5. STUDY QUESTIONS We sought to specifically investigate the potential misuse of SGA by asking: • How often are HTE analyses and corresponding covariates prespecified and were the reasons (if any) primarily substantive or statistical? • What was the objective evidence for or against the presence of HTE? • How did authorsinterpret their own HTE-related findings, and to what extent did their interpretations match the objective evidence?

  6. METHODS • Design: Systematic Review • Population: JAMA, BMJ, Lancet, NEJM, Annals • Probability Sample: • Odd months in 1994, 1999, 2004 • Initial search: N = 4,863 articles • After additional random sampling and exclusions, N = 319 clinical trials • Final sample: 87 of 319 trials (27%) reporting test for HTE

  7. Coding of Covariates Covariates Examined in HTE analyses Prespecified Not Prespecified Number of Covariates with Rationale All covariates Some None Types of Reasons Substantive Statistical

  8. Defining Clinicostatistical Divergence Clinicostatistical Divergence: Clinically meaningful and statistically significant differences between subgroup effects and average effect (coded as “none” “weak” “moderate” “strong”) • Clinical Divergence (CD): Was the ratio measure of effect in any subgroup at least 25% greater or smaller than in the sample as a whole? • Statistical Significance (SS): Was a test for interaction associated with a p value of less than or equal to 0.10?

  9. Coding Clinicostatistical Divergence * Denotes absence of data

  10. Coding Authors’ Interpretations • Evidence for HTE sufficient to support different treatment recommendations in one or more subgroups • Evidence for HTE insufficient to support different treatment recommendations but sufficient to warrant further systematic research • Evidence for HTE was possibly present but insufficient to warrant further research • Definite evidence against HTE • No interpretation of HTE results

  11. RESULTS 87 RCTs Prespecified Not Prespecified 53 (61%) 34 (39%) Trials by Number of Covariates with Rationale All covariates Some None 17 (32%) 12 (23%) 24 (45%) Types of Reasons Substantive Statistical 22 (76%) 7 (24%)

  12. 29/87 = 33%

  13. 17/58 = 29%

  14. 25/87 = 29%

  15. 31/62 = 50% 10/62 = 16% 31/62 = 50%

  16. Overstated = 27% Understated = 9%

  17. LIMITATIONS • Limited number of journals, years, trials reviewed • Data potentially incomplete • HTE analyses performed but not published • HTE analyses performed and published in secondary journals

  18. CONCLUSIONS • Analysis and reporting of HTE incomplete • Prespecification inconsistent • Rationale incomplete • Effect measures and p-values (or CIs) incompletely reported • When reported, objective evidence for clinicostatistical divergence found in approximately 1/3 of trials • Authors frequently misinterpret own findings (in both directions)

  19. IMPLICATIONS • Researchers: • Ensure that SGA are prespecified with a priori rationales for covariate inclusion, or clearly labeled as exploratory • Include a statistical test for interaction or heterogeneity in the analyses • Report all results from SGA (including p values for HTE tests and effect measures with confidence intervals), even if not significant

  20. IMPLICATIONS • Journal Editors: • Ensure authors report SGA-associated data when SGA is performed • Ensure that authors’ discussion includes interpretation of SGA performed and limitations of such analyses • Readers/Clinicians: • Weigh the authors’ interpretations and recommendations in light of the objective evidence presented prior to changing practice or implementing recommendations

  21. Thank you

More Related