The Quality of Randomized Controlled Trials in the Rheumatologic Literature from 1998-2018

1. Michael Putman, MD Rheumatology Fellow RWCS 2019 The Quality of Randomized Controlled Trials in the Rheumatologic Literature from 1998-2018

2. Background-imumab Rheumatology has been inundated by new therapies…

3. Background-failimab? …yet somehow the majority of our guideline recommendations are level C JAMA Intern Med. 2018 Jan 1;178(1):146-148.

4. Background How can we have so many therapies yet so few high level recommendations? There are a LOT of these too There are a LOT of these (I’m not one of them)

5. Background Let’s focus on the randomized controlled trials (RCTs) How good are they? Are they getting better? …or worse? RCTs How strong are their findings? Who funds them? What kinds of questions are they asking?

6. Background How do we know what’s a good trial and what’s not?

7. Background How do we know what’s a good trial and what’s not? • Blinding • Identification of primary outcomes • Patient reported outcome measures (PROMs) • Power calculation • Sensitivity analysis • Adjustment for multiple hypothesis testing • Intention to treat analysis (ITT)

8. Methods Overview • We identified the top 3 clinical rheumatology journals by h5-index, Scimago Journal and Country Rank, and Emeunet 2016 Top Journals assessment

9. Methods Overview • All clinical RCTs reporting primary analysis of a pharmacologic intervention against a comparator in 1998, 2008, and 2018 from selected journals were included • Intervention, metrics of RCT quality, absolute risk estimates, p-values, and funding were recorded • A “quality scale” was constructed from the quality metric variables and normalized to a value from 1-10 • Bivariate associations between variables were assessed using Chi2 testing for categorical variables and either independent samples t-test or one way ANOVA for continuous

10. Methods Biting off more than I meant to chew 3,338 Titles Identified 3,338 Titles Identified 3,338 Titles Identified 3,338 Titles Identified 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 85 Papers included 85 Papers included 85 Papers included 85 Papers included

11. Methods Journals and years 3,338 Titles Identified 3,338 Titles Identified 3,338 Titles Identified 3,338 Titles Identified 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 85 Papers included 85 Papers included 85 Papers included 85 Papers included

12. Results Has the quality of journals changed over time? 3,338 Titles Identified 3,338 Titles Identified 3,338 Titles Identified 3,338 Titles Identified 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 85 Papers included 85 Papers included 85 Papers included 85 Papers included

13. Results Does industry involvement impact the quality of a trial? 3,338 Titles Identified 3,338 Titles Identified 3,338 Titles Identified 3,338 Titles Identified 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 85 Papers included 85 Papers included 85 Papers included 85 Papers included

14. Results If the quality hasn’t changed much over time, what has? * * * * * * 3,338 Titles Identified 3,338 Titles Identified 3,338 Titles Identified 3,338 Titles Identified 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 85 Papers included 85 Papers included 85 Papers included 85 Papers included

15. Results Bigger is better? Rheumatology rocks? Publication bias? * * * * * * 3,338 Titles Identified 3,338 Titles Identified 3,338 Titles Identified 3,338 Titles Identified 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 85 Papers included 85 Papers included 85 Papers included 85 Papers included

16. Limitations “Project does not currently meet our standards for publication” • Only assessed 3 years out of 20, may not be generalizable • Review by co-author has not been completed • Not powered to detect small differences between groups • Interpretation of quality metrics may be subjective • Limited by reporting from studies; actual study quality may not reflect that described in study methods • Publication patterns may vary over time and high impact general interest publications were not assessed * * * * * * 3,338 Titles Identified 3,338 Titles Identified 3,338 Titles Identified 3,338 Titles Identified 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 85 Papers included 85 Papers included 85 Papers included 85 Papers included

17. Conclusions There are a whole lot of RCTs out there • Intention to treat (ITT) analysis has become more common, but RCT quality by other metrics has remained constant. • Industry funds the vast majority of studies. Industry studies are significantly more likely to be appropriately blinded, report PROMs, use ITT, and have a higher overall quality. • Over time, fewer studies have included an active comparator and studies with significant primary outcomes (p < 0.05) have become significantly less common. • Smaller studies were associated with larger effect sizes. Overall, rheumatologic interventions have a high ARR. * * * * * * 3,338 Titles Identified 3,338 Titles Identified 3,338 Titles Identified 3,338 Titles Identified 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 186 Did Not Meet Criteria After Full Review 85 Papers included 85 Papers included 85 Papers included 85 Papers included

18. Thank You