Reporting systematic reviews and meta-analyses (PRISMA) and observational studies (STROBE)

1. Reporting systematic reviews and meta-analyses (PRISMA) and observational studies (STROBE) Doug Altman The EQUATOR Network Centre for Statistics in Medicine, Oxford, UK October 2012

2. Systematic review • A systematic review is a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarise the findings of similar but separate studies. • A study of studies • Objective is to summarize evidence from multiple studies using explicit methods • It may include a quantitative synthesis (meta-analysis), depending on the available data

3. Key characteristics of SR • Focused well defined research question • Clearly stated title and objectives • Comprehensive strategy for identification of all relevant studies (published & unpublished) • Explicit (and justified) predefined inclusion & exclusion criteria • Critical appraisal of studies • Clear analysis of the results of eligible studies • Quantitative (meta-analysis) • Qualitative • Structured report

4. The QUOROM Statement[Moher et al 1999] • Guidance on what information should be included in published reports of meta-analyses of randomized trials • Checklist of items which should be reported • Also recommended a flow diagram showing flow of studies through the review • to be included in the published report • Evidence-based, whenever possible • QUOROM developed in 1996 and published in 1999 Moher et al. Improving the quality of reporting of meta-analyses of randomised controlled trials: the QUOROM statement. Lancet 1999;354:1896-1900

5. Checklist of items

6. Also recommended a flow diagram showing flow of studies through the review

7. PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses • Update of QUOROM • Developed in 2005, Published in 2007 • Consists of a 27 item checklistand a flow diagram • Reporting of systematic reviews and meta-analyses that evaluate healthcare interventions • Includes long explanatory document

9. # of records identified through database searching # of additional records identified through other sources Total # of duplicates removed # of records screened # of records excluded # of articles excluded, with reasons Total # of articles assessed for eligibility Total # of studies included in qualitative synthesis of systematic review # of studies included in quantitative synthesis of systematic review identification screening eligibility included

10. Reporting vs conduct: study methods

11. PRISMA Practical

12. 2 reviews in one article • focus on progesterone

13. For each item … • Is there text relating to the item? • Does the text tell us what we need to know?

14. PRISMA: Item 6, eligibility criteria METHODS Eligibility criteria • Specify study characteristics used as criteria for eligibility, giving rationale • PICOS (participants, interventions, comparisons, outcomes and study design) • Length of follow-up • Specify report characteristics used as criteria for eligibility, giving rationale • Years considered • Languages • Publication status Can you locate any text about this issue in the report?

15. Wyatt et al • No explicit statement of study characteristics • “… clinical trials of progesterone and pro­gestogensin the management of premenstrual syndrome.” • Dates given in search criteria • “All languages were included.”

16. PRISMA – item 8, search • “Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated” Can you locate any text about this issue in the report?

17. Wyatt et alPage 2, Methods, Trials “MeSH terms used were premenstrual syndrome, progesterone, and progestogen, as well as the individual drug names, together with title and abstract searches for keywords progesterone, pro­gestogen, premenstrual syndrome, premenstrual ten­sion (PMT), late luteal phase dysphoric disorder (LLPDD), premenstrual dysphoria (PMD), and pre­menstrual dysphoric disorder (PMDD).”

18. PRISMA: Reporting search strategy • We realize that journal restrictions vary and that having the search strategy in the text of the report is not always feasible • Expensive real estate • We strongly encourage all journals, however, to find ways, such as a ‘‘Web extra,’’ appendix, or electronic link to an archive, to make search strategies accessible to readers • We also advise all authors to archive their searches so that: • others may access and review them (e.g., replicate them or understand why their review of a similar topic did not identify the same reports) • future updates of their review are facilitated

19. PRISMA: Item 9, study selection METHODS Study selection • State the process for selecting studies • Screening • Eligibility • Included in systematic review and, if applicable, included in meta-analysis Can you locate any text about this issue in the report?

20. Wyatt et al • “We searched medical databases for reports of published clinical trials of progesterone and progestogens in the management of premenstrual syndrome.” • “References cited in all trials were searched iteratively to identify missing studies. All languages were included.” • “We included trials that investigated the effect of progesterone or progestogens on premenstrual symptoms if they were randomised, placebo controlled, double blind studies that included patients with a pretreatment diagnosis of premenstrual syndrome, for which all data from the trials could be acquired.”

21. PRISMA Item 11, Data items METHODS Data items 11 List and define all variables for which data were sought (e.g., PICOS, funding sources) and any assumptions and simplifications made. (PICOS = participants, interventions, comparisons, outcomes and study design) Can you locate any text about this issue in the report?

22. Wyatt et al “We collected data on the dosage and preparation of treatment. The main outcome measure was a reduction in overall symptoms of premenstrual syndrome. Combined or overall symptoms was chosen in an attempt to overcome the clinical heterogeneity associated with the measurement and scoring of symptoms used in individual trials.”

23. PRISMA – item12, Risk of bias in individual studies • “Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level), and how this information is to be used in any data synthesis” • Can you locate any text about this issue in the report?

24. Wyatt et alPage 2, Methods, Quality assessment • “We assessed trial quality using a scale developed by Jadad et al,11 which assesses the randomisation, double blinding, reports of drop outs, and withdrawals for the trials • ... our own quality scale, which assesses the quality of the trials for study design, reproducibility, and statistical analysis. This eight point scale comprised the following: confirmation that no other medications or oral contraceptives were being taken; a power calculation to justify patient numbers or more than 65 participants in each arm (enabling detection of a small effect size of 0.3, see below); a single, clearly stated dose of drug; reproducible measurement of premenstrual symptoms; clear presentation of results; a description of the number and reason for trial withdrawals; exclusion of, or a separate analysis of, participants with a major psychiatric disorder; and whether or not the trial was supported by independent funding.”

25. Reporting risk of bias • “Authors should report how they assessed risk of bias; whether it was in a blind manner; and if assessments were completed by more than one person, and if so, whether they were completed independently. Similarly, we encourage authors to report any calibration exercises among review team members that were done. Finally, authors need to report how their assessments of risk of bias are used subsequently in the data synthesis (see Item 16).”

26. Wyatt et alPage 2, Methods, Quality assessment • “We awarded one point for each category present in the trial. • Each trial was independently scored by two investigators and the third investigator arbitrated on any disagreements. • We used predetermined criteria for the recognition of the highest quality trials. A score of 3 or more was required in the Jadad score for the trial to be designated “high quality” and included in the meta­analysis11; a score of less than 3 meant that the trial was designated “low quality.”

27. PRISMA Item 17 RESULTS Trial flow 17 Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram. Can you locate any text about this issue in the report?

28. Page 2, Results • “We identified 14 published trials that assessed the efficacy of progesterone in the management of premenstrual syndrome. We excluded four: two because of their low quality score on the Jadad scale, one because the data could not be extracted, and one because the trial failed to make a prospective diagnosis of premenstrual syndrome before randomisation. Ten trials remained ........” • No flow diagram

29. Caughey AB, Sundaram V, Kaimal AJ, Gienger A, Cheng YW, McDonald KM, Shaffer BL, Owens DK, Bravata DM. Systematic Review: Elective Induction of Labor Versus Expectant Management of Pregnancy. Annals of Internal Medicine 2009;151: 252-263

30. PRISMA Item 18 RESULTS Study characteristics 18 For each study, present characteristics for which data were extracted (e.g., study size, PICOS, follow-up period) and provide the citations. Can you locate any text about this issue in the report?

31. Wyatt et al

32. Wyatt et al Problems in Table 1 • Crossover trials – not mentioned in text • Comparator not stated • Variation in outcome measures – how combined or chose one? • Some statements disagree with Figure • No details of elements of “quality” scores • Unclear how handled trial with 3 arms

33. PRISMA – item 15, Risk of bias across studies • Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies). Can you locate any text about this issue in the report?

34. Page 2, methods, statistical analysis • “We used the method of Egger et al to detect bias (such as publication and location bias) in the included trials with a funnel plot. We assessed the asymmetry of the funnel plot quantitatively by plotting a linear regression of the standard normal deviate (standardised mean difference divided by SE) against precision (inverse of SE). A regression line that passes through the origin of the plot (within error limits) indicates symmetry and hence the absence of bias.”

35. PRISMA Item 20 RESULTS Results of individual studies 20 For all outcomes considered (benefits or harms) present, for each study: (a) simple summary data for each intervention group, (b) effect estimates and confidence intervals, ideally with a forest plot.

37. Altman & Cates complained about absence of any numerical results (BMJ rapid response) • “There are several aspects of this review that readers cannot assess without summary data from each study. • For example, we would wish • to assess the strange heterogeneity P values for Figures 1 and 2 (the quoted P values of 0.999 are implausible given the clear graphical heterogeneity); • to gain some insight into how the cross-over trials were included in the analysis (about which the authors say nothing at all) and whether the crossover and parallel group trials differed in their findings; • to seek an explanation for the apparent discrepancy for three trials (references 19, 31 and 32) between the ‘reported results’ in Table 1 and the results shown in Figures 1 and 2; • to assess the claim that random and fixed models give the same answer in the face of graphical heterogeneity;” • … and 5 more points

38. Altman & Cates complained about the absence of any numerical results • “In addition, we note that the authors make no comment about the varied nature of the outcome measures in these trials, nor do they say which outcome was used for those trials that presented more than one. It is hard to believe that all of the scales can be considered equally valid assessments of symptoms. • Also, we wonder if they can clarify the meaning in the figure legends of ‘standardised mean difference … for proportion of patients who showed improvement …’. We are puzzled by this terminology as the SMD gives no direct information about proportions of patients improving.”

39. Authors’ reply • “We have found this personal attack unpleasant and upsetting and have to question the use of unsupported attacks in the Rapid Response forum.” • “We have considerable experience in PMS (clinically as well as though our research) and believe ourselves competent to judge the clinical appropriateness of combining trial data.” • The only question they addressed was one we had not asked! • A 2nd request for the data to be provided went unanswered!

41. Observational studies • Transparent reporting is particularly important for observational studies • Vulnerable to bias and confounding • Findings are often over-interpreted • Findings often generate health scares 41

43. Scope of STROBE • Epidemiological research comprises several study designs and multiple topic areas • Initial restriction to three major areas • cohort, case-control, and cross-sectional studies • Later extensions to other study designs • STREGA for genetic association studies (published 2009) • STROBE-ME for molecular epidemiology • etc

44. TITLE and ABSTRACT  INTRODUCTION  Background/rationale  Objectives   METHODS Study design  Setting  Participants Variables  Data sources/measurement  Bias  Study size  Quantitative variables     Statistical methods RESULTS Participants Descriptive data  Outcome data Main results Other analyses  DISCUSSION Key results  Limitations  Interpretation   Generalisability   OTHER INFORMATION Funding   Final STROBE checklist 44 22 (34) items

45. Design-specific items • Participants • Statistical methods • Descriptive data • Outcome data 45

46. STROBE Statement • Guidance on how to report observational studies well • Focus on 3 main study designs: cohort, case-control, cross-sectional studies • Published in Oct 2007: short paper and E&E • Adopted by many journals Find it on: www.equator-network.org www.strobe-statement.org

47. Case-control studies • Patients with a certain outcome or disease and an appropriate group of controls without the outcome or disease are selected • (usually with careful consideration of choice of controls, possibly with matching) • Information is obtained on whether the participants have been exposed to the factor under investigation

48. STROBE exercise