Guidelines for Reporting Network Meta-Analysis: PRISMA Extension

1. Reporting Guidelines and the PRISMA Extension for NMA April 10, 2014 Brian Hutton, MSc, PhD Associate Scientist, Knowledge Synthesis, OHRI & DSEN New Investigator Assistant Prof, Dept. Of Epidemiology & Community Medicine, University of Ottawa Steering Committee Members: David Moher, Georgia Salanti, Debbi Caldwell, Ed Mills, Kristian Thorlund, Chris Schmid, Anna Chaimani, Doug Altman

2. Context: the DSEN NETMAN team • Led by Drs. David Moher, myself, in collaboration with other scientists at OHRI and in Toronto, Vancouver, etc • Duty: Perform systematic reviews incorporating Network Meta-Analyses (NMAs) to address queries from CIHR/DSEN posed by knowledge users (e.g. Health Canada) • Queries commonly deal with emerging safety concerns related to pharmaceuticals, often in the context of comparing multiple agents or classes in the context of a treatment network • We’ll briefly review networks in a moment...

3. Relative effectiveness of classes of pharmacotherapies in non-diabetics with hypertension Serotonin affinity of antidepressants and their impact on risk of falls and fractures

4. Outline for this Session • Network Meta-Analysis • What is it? • Why do we need special guidance? • Overview: Developing Reporting Guidance • Methods: Moher et al, 2010. Steps to follow? • How did PRISMA NMA adapt this process? • Status: Progress to Completion of PRISMA NMA • Where are we, what are our timelines? • What core findings have emerged from our efforts? • How we will produce uptake of PRISMA NMA

5. What is Network Meta-Analysis?

6. Current “standard” approach to SR / MA: X? A limitation in some situations: • How often does “A versus B” address true question of interest? • Many diagnoses with >2 therapies available. • ‘Traditional’ approach compares just two trts; so do we just pick a certain pair to look at? “short-sighted view…” • To get an answer, could try and “synthesize” 3, 4, 5 or more traditional systematic reviews depending on # of treatments to consider. Then ‘rank’ treatment effects in our mind…. • Cumbersome; something more rigorous would be helpful

7. Biologics for rheumatoid arthritis • Etanercept, infliximab, anakinra, abatacept, rituximab, certolizumab, golimumab, tocilizumab, adalimumab • Chemotherapy for breast cancer • Large number of complex multi-agent regimens • Treatments for depression • Sertraline, paroxetine, escitalopram, fluoxetine, duloxetine, etc…

8. Another Example: Generalised Anxiety Disorder Baldwin D, Woods R et al. Efficacy of drug treatments for generalised anxiety disorder: systematic review and meta-analysis. BMJ. 2011;342:d1199.

9. Brief terminology note... “Indirect comparison” “Direct comparison” Sertraline Sertraline Paroxetine 0 RCTs 1> RCTs Estimates from RCTs, pairwise meta-analysis can be estimated Can compare A, B via ‘common comparator’ C

10. To be ‘clinically relevant’, 9 active agents available which are used in practice and should be considered. Librarian plugs away on a search to find all our relevant RCTs... Placebo controlled (“indirect”) data: 4 RCTs 2 RCTs Escitalopram Placebo Head-to-head (“direct”) data: 5 RCTs Duloxetine Placebo 1 RCT 1 RCT Fluoxetine Placebo 1 RCT Escitalopram Paroxetine 3 RCTs Lorazepam Placebo 2 RCTs Duloxetine Venlafaxine 3 RCTs Paroxetine Placebo 1 RCT Fluoxetine Venlafaxine 5 RCTs Pregabalin Placebo 1 RCT Lorazepam Pregabalin 2 RCTs Sertraline Placebo Pregabalin Venlafaxine 2 RCTs With more relevant treatments to consider and existing evidence, we can create a ‘network’ of the relevant direct and indirect evidence... Tiagabine Placebo Paroxetine Sertraline 8 RCTs Venlafaxine Placebo

11. The GAD treatment network Paroxetine Pregabalin Lorazepam Fluoxetine Sertraline Tiagabine Duloxetine Escitalopram Venlafaxine Placebo

12. Benefits of Using Network Meta-Analysis • Answer more clinically relevant questions • All therapies of interest versus a subset of them • Try to give ‘the full picture’ for clinicians • Gain precision by considering all available evidence • E.g. One head-head trial of A vs B alone, or supplemented with 10 studies of A vs placebo and 10 studies of B vs placebo • Potential to more explicitly ‘rank’ treatments using summary outputs from NMA • Many more introductory topics we could discuss, but focus for today will be on PRISMA development

13. How Often are NMAs Reported? • Geometric (i.e. Large!) increase in the publication of papers that present systematic reviews where meta-analyses compare many agents together simultaneously • Nikolakopoulou et al reported that 40 such papers were published between 1997 and the end of 2008 (i.e. 12 year span) • Since then: 25 in 2009, 30 in 2010, 42 in 2011, and 50 in 2012. • Published in high impact journals and also increasingly used by key organizations: Cochrane, CADTH in Canada, NICE, etc... • Clearly a design which is here to stay

14. Why does NMA Need ‘Special’ Reporting Guidance beyond PRISMA?

15. What is the PRISMA Statement? • Reporting guidance tool to help researchers maximize the transparency and completeness of systematic reviews published in the literature • Provides a checklist of items recommended for researchers to include in SR/MA which can be used to maximize reporting rigor • Preceded by QUOROM; both have had important impacts on SR/MA literature

16. Summarizing the extent of included evidence... • Traditional reviews: typically will see # of studies and patients reported in main text of the SR/MA • Networks: most commonly see ‘network’ diagrams; nodes/links sometimes sized to reflect #’s of studies and patients, sometimes just reported with additional text Sutton et al. Mixed comparison of stroke prevention treatments in individuals with nonrheumatic atrial fibrillation. Arch Int Med 2006. Cipriani et al. Comparative efficacy and acceptability of 12 new generation antidepressants: a multiple treatments meta-analysis. Lancet (2009).

17. Summarizing treatment effects from data analysis... • Traditional reviews: most commonly see forest plots with summary estimates and confidence intervals of trial level and summary level findings • Networks: no longer see the study level data given mass of trials, but forest plots still often used to present summary estimates... Mills E, Druyts E, Ghement I, Thorlund K. Pharmaco-therapies for COPD: a multiple treatment comparison meta-analysis. Clinical Epidemiology (2011).

18. Authors may also become creative with tables to present estimates more concisely.... Cipriani et al. Comparative efficacy and acceptability of 12 new generation antidepressants: a multiple treatments meta-analysis. Lancet (2009).

19. Summarizing additional parameters of potential interest (e.g. Treatment rankings and probabilities...) Graphical approaches such as “rankograms” are used... Cipriani et al. Comparative efficacy and acceptability of 12 new generation antidepressants: a multiple treatments meta-analysis. Lancet (2009).

20. Lots of novel elements; many are based in advanced epidemiologic concepts and statistical methods • Challenges also relate to the challenge of dealing with a larger abundance of both data “in” and results “out” • These are considerations in reporting of research studies which are susceptible to variations in reporting, not all of which may be optimal or transparent • A need to “help” authors in reporting?

21. Generating reporting guidance:What model / process do we follow?

22. Applying its suggested process for PRISMA NMA in collaboration with other experts:Georgia Salanti, Debbi Caldwell, Chris Schmid, Anna Chaimani, Kristian Thorlund, Ed Mills, Doug Altman

23. Our PRISMA NMA extension plan • Step 1 - Systematic Review performed • Reporting quality of network meta-analyses in the published literature; guidance from existing overviews for conduct and interpretation to extract ‘themes’. • Summary of key aspects identified for survey incorporation. • Step 2 - Questions about reporting guidance framed through teleconference and several drafts of survey. Implement online Delphi survey of relevant experts. • Step 3 - Face to face meeting. • Key items lacking consensus up for discussion; endorsement • Step 4 – Writing sub-committee efforts. • Prep of checklist/Elaborations document as per PRISMA

24. Step 1:Systematic Review

25. Methods • Review of existing literature commenting on reporting quality issues; not a primary QA • Search.IS developed a strategy for Medline, EMBASE, Cochrane. Conducted September 2012. Update performed recently, results will be included in final version to be submitted for publication. • Screening.Two individuals independently. Two stages to work through abstracts and then full text publications. Summary of selection via PRISMA recommendations. • Inclusion Criteria.Full texts/abstracts dealing with reporting quality of NMAs/ITCs related to search, stats, assumptions, evidence base, etc. Also sought key guidance docs (from search and committee recs). • Data Extraction and Synthesis.Double extraction. Narrative summary of findings.

26. Study Selection • 684 citations reviewed stage 1; 32 considered potentially relevant and screened in full text. • Total of 24 included for synthesis: • 8 reports related in some manner to reporting quality • AHRQ/Coleman et al 2012, Donegan et al 2010, Song et al 2009; Brooks Renney et al, Bending et al, Buckley et al; Tan et al 2013; Bafeta et al 2013 • 16 guidance documents reviewed for additional key themes • Examples include ISPOR guidance, Mills et al JAMA 2012, Caldwell 2005, Ioannidis 2009, Salanti 2012, etc • Findings. For brevity will not review details from individual manuscripts, however worth highlighting the themes which emerged and considered for the survey...

27. Completed systematic review; published March 26, 2014 in PLOS ONE

28. Summary of Considerations for Methodologic Reporting Literature Search: • Transparent specification of efforts to search for both direct and indirect evidence of relevance • Presentation of search terms and the full search strategy (or strategies if separate searches undertaken for each comparison) for one electronic database • Provideinfo regarding involvement of primary lit searching versus use of existing systematic reviews • If existing reviews used, specify how these were located , describe their related inclusion criteria Specification of Assumptions for Network Meta-Analysis: • Clear specification of the assumptions of homogeneity, similarity and consistency • Clear specification of efforts taken by reviewers to empirically assess assumption appropriateness • Clear specification of what information is being provided to readers to allow them to consider the validity of the assumptions Statistical Approach to Analysis: • Specification of key details of the approach to statistical analysis taken: Frequentist or Bayesian approach, fixed or random effects model, homogeneous or heterogeneous between-trial variance structure, mean or median summary values reported, all clinical outcomes planned a priori for investigation • Specification of methods used to assess statistical heterogeneity and potential for publication bias within the treatment network • Specification of methods used to evaluate for inconsistency between direct and indirect evidence in the treatment network • Specification of approach to assess model fit (e.g. as appropriate, how model convergence was evaluated, criteria used to compare the fit of different but related models, etc) • Description of statistical methods used to address clinical and methodologic homogeneity in the analyses (subgroups, meta-regression including adjustments for baseline risk and the impact of risk of bias variations, etc) Formulation of Network Structure for Planned Analysis: • Specification of all treatments included in the planned meta-analysis. • Specification of how different doses of the same agent are to be handled with associated rat ionale, as well as related but different control groups (i.e. address ‘lumping and splitting’ of treatments) • Provide justification for synthesis data across different patient populations where undertaken if it may be perceived that such an analysis may introduce important clinical heterogeneity (e.g. across disease types, severity of illness, etc)

29. Summary of Considerations for Reporting of Results and Subsequent Interpretation Presentation of Evidence and Its Characteristics in the Treatment Network • Presentation of network diagram to summarize identified evidence; optimal characteristics for visual representation of available evidence in such figures (e.g. node size, connection size, etc). • Means of reporting sample sizes, numbers of studies, geometry considerations ,implications for potential bias • Optimal information and approach to presentation of information allowing readers to assess clinical and methodologic heterogeneity within the treatment network (e.g. evidence tables, risk of bias findings, min/max of important features such as years of study/publication, follow-up duration, etc) Empirical Assessment of Assumptions for Network Meta-Analysis • Optimal information to summarize evaluations of statistical heterogeneity within the treatment network • Optimal information ato summarize analyses to assess consistency of direct and indirect evidence Presentation of Summary Treatment Effect Estimates • What estimates to report: all possible pairwise comparisons or only those for particular subset of relevance? • Findings from traditional pairwise analyses also be provided? • Presentation of summary estimates and corresponding uncertainty (i.e. credible/confidence intervals) • Presentation of summary estimates from sensitivity and subgroup analyses Presentation of Additional Results from Analysis • Presentation of treatment rankings and corresponding probability estimates • Optimal use of tables and figures to best present treatment ranking and probability information Discussion and Conclusions • Commentary on the clinical and biologic plausibility of the observed findings • Commentary relevant to any important inconsistency and heterogeneity concerns identified in analyses that may play an important role in strength of interpretations drawn • Commentary relevant to any perceived concerns of the treatment network geometry which may play an important role in the results observed and the strength of interpretations drawn

30. Component 2:Delphi Survey - Methods

31. Survey Implementation Question Development: • Several rounds of review of draft questions amongst the steering committee that were generated based on the themes from our review Survey Invitees: • corresponding authors of all published NMAs; members from the 30-member group responsible for PRISMA guidance; editors from journals that have published 2 or more NMAs. • 103/215=48% had replied prior to analysis; >90% of respondents provided complete data. Question structure: • combination of multiple choice; ranking (1-10 where 1=strong disagreement, 10=strong agreement); and free text answers. • 41 questions; www.fluidsurveys.com. Mailout and 4 reminders sent.

33. Approach To Summary of Findings • Multiple choice answers summarized by calculating proportion of responses for each possible choice. Free text questions reviewed to extract additional unaddressed themes. • Ranking questions (i.e. Value from 1-10) summarized by calculating % in strong agreement/disagreement: • If % of responses with a score of 8 or higher >75%, denotes % in high agreement; if % of responses with a score of 3 or less >75%, denotes in high disagreement. • Those in high agreement; to be included in checklist. Those in high disagreement; not further considered. • Those without a clear answer; subject for discussion at today’s meeting.

34. Results: Survey Items Where Consensus Achieved ABSTRACT/INTRO • Abstract/introduction rationale for NMA • Clear statement of rationale for including multiple treatments METHODS • Rationale for inclusion of the chosen treatments in the network ‘defense’ of network structure (trts, admin, lump vs split, etc) • Describe methods to verify ‘likeness’ of studies (i.e. Similarity or other terms***) • Details of statistical model used and steps taken to explore heterogeneity (meta-regressions, etc. but to what extent beyond core PRISMA?) • Describe efforts taken to explore agreement of direct, indirect evidence RESULTS • Network diagram to summarize evidence • Rankings/probabilities should not be reported independent of effects DISCUSSION • Comment on appropriateness of assumptions, which estimates are most reliable, which could use more research; discordance if findings from direct and indirect evidence

35. Results: Items Remaining for Discussion • Reporting all possible pairwise comparisons from traditional pairwise MA from the evidence base • Degree of completeness of summarizing results from NMA model All comparisons? All comparisons of trts vs control only? All comparisons of trts vs a particular trt? Balance of involving main text versus supplemental materials • Rankings and probabilities should be included if explored • Inclusion of raw data used for NMA • Inclusion of detailed study information tables A meeting program was structured to address these issues amongst a group of experts…

36. Step 3:Face-to-Face Experts’ Meeting,Cochrane Colloquium 2013 in Quebec City Front: Kristian Thorlund, Brian Hutton, Cindy Mulrow, Anna Chaimani Back: Chris Cameron, Jeroen Jansen, Georgia Salanti, Isabelle Boutron, Peter Goetzsche, David Moher

38. Our Objectives for Each Short Session • ~ 45 mins per ‘key’ topic: ~12-15 mins presentation; ~30 mins group discussion. BH to take detailed minutes for the day for future use • Address the key questions within each section-specific overview • “Big picture” goals for each session: • What belongs in the core checklist? • What belongs in the E & E? • What is our ‘best’ way to provide guidance to a wide body of researchers, reviewers, editors on how to transparently report NMA? • Attendees were asked to keep in mind: • Objective is on reporting NMAs; not conduct • NMA is a developing method; objective is not to develop an overly specific list of requirements • Developing an extension, not a full statement! Avoid redundancy

41. Without Getting Into Details of Discussions… • The meeting was successful in achieving our objectives; i.e. • Consensus on remaining items of debate • A strategy for contributors to prepare different components of the full text manuscript summarizing the NMA PRISMA checklist was established • Having editors’ perspectives (Mulrow, Moher) on how to approach task of journal endorsement also beneficial to make plans for publication pursuit in summer 2014

42. Step 4:Completing the Guidance Document

43. A skeleton of the guidance document was drafted; key checklist items which were either new items or existing items requiring elaboration were assigned to participants. Whether a new checklist item or a modification of an item from core PRISMA, authors were also asked to select examples from the literature that could be incorporated into the guidance. Educational boxes have also been included to provide sufficient background for all readers. Thirty day turnaround requested; turned into ~45 days given commitments of many busy people! However success achieved in January 2014. A completed draft was achieved in late February 2014; the draft was circulated for co-author feedback which is now being addressed. A final updated draft will be circulated amongst the authorship team and an additional small group of individuals reflecting external expertise in journal editorship and methods for systematic reviews and network meta-analysis. Objective: submission in May 2014. Submission has been invited by the Annals of Internal Medicine.

44. How will it look? What will it contain?A sneak peek...

45. Description of All Checklist Items • unchanged, modified, or new item • Statement of general instructions; example; and explanation of importance

46. Updated PRISMA checklist; explanation of modification of new items and modifications of existing items. ~30 item checklist where authors (and readers) can assess their publication for completeness to maximize transparency of reporting

47. Educational boxes on assorted topics to provide content expertise for readers; e.g. terminology in NMA…

48. Assumptions for NMA, geometry… Statistical aspects in reporting NMA

49. Considerations for reporting of network diagrams; vital component given increased complexity of the evidence base in SR/MAs dealing with treatment networks

50. Considerations for presenting summary of findings from NMA Increasingly challenging given that number of pairwise comparisons increases rapidly & traditional forest plots do not apply