Department of Epidemiology & Public Health

1. Department of Epidemiology & Public Health Criticisms and concerns regarding the STROBE statement: a review of commentaries, editorials and letters Patrizia Frei, PhD Swiss Tropical and Public Health Institute, Basel

2. Overview • Review of 34 commentaries, editorials, letters on STROBE • Focus on the criticism/concerns regarding STROBE • General • Specific STROBE Meeting

3. STROBE: judging study quality? • Concern that STROBE will be used as a tool for judging the quality of a study (Editors of Epidemiology, 2007; Keiser and Egger, 2008; Meerpohl et al., 2007) • STROBE should expressly discourage the use of its guidelines for the evaluation of studies (Editors of Epidemiology, 2007) • Strobe ventures into the matter of study conduct (Editors of Epidemiology, 2007) • It should restrict its function to the improvement of reports on epidemiological studies (Editors of Epidemiology, 2007) STROBE Meeting

4. STROBE: recommendations becoming a law? • Concern that STROBE may be considered prescriptive or dogmatic and that journals/decision-making authorities will turn checklist into requirements (Rothman and Poole, 2007; Kohlmann, 2008) • Guidelines should be formulated more as suggestions than as rigid rules (Rothman, 2007) STROBE Meeting

5. Constraint to scientific creativity • Concerns about any effort to formalize reporting in a field as heterogeneous as observational epidemiology (Editors of Epidemiology, 2007; Nijsten et al., 2008) • STROBE represents a breach in the capacity for scientific creativity; standardization of research (Potvin, 2008; Meerpohl, 2009) • STROBE may eventually make our field of research more boring (Potvin, 2008) • STROBE should give clear emphasis to the importance of good judgement over any specific criteria. We should not allow the blindly applied rule to trump the creative exception (Editors of Epidemiology, 2007) • Incorporate expiration dates into STROBE (Rothman and Poole, 2007) STROBE Meeting

6. “Polished apple” • The basic principles of epidemiology should not be introduced only at the time of reporting (MacMahon and Weiss, 2007) • Some authors might be tempted to describe what should have been done, rather than what was done (MacMahon and Weiss, 2007; Battegay, 2008; Keiser and Egger, 2008) • There is always a risk that poorly designed studies will be made more difficult to spot by superficial improvements in the way they are reported (Rothwell and Bhatia, 2007) STROBE Meeting

7. Other general points (1) • The suggestions are rather general, which can be confusing (Ebrahim and Clarke, 2007; Nijsten et al., 2008) • There are odd remnants of clinical-trial thinking in STROBE, STROBE should acknowledge the differences between RCTs and observational studies (Editors of Epidemology, 2007) • More thorough purging of inapplicable RTC concepts, and a clear statement of the crucial distinctions between RCTs and observational studies (CONSORT-STROBE) (Editors of Epidemiology, 2007) • Negative effects on the review process (MacMahon and Weiss, 2007; Bagettay, 2008) • More discussion on how observational epidemiology can be made more robust (Ebrahim and Clarke, 2007) STROBE Meeting

8. Other general points (2) • The criteria for selecting items that were ultimately included in the STROBE checklist are not clear (Nijsten et al., 2008) • Inclusion of only three study designs (Ebrahim and Clarke, 2007; Knotterus and Tugwell, 2008) • Statement about extent of any input from people with relevant clinical expertise might be an additional future STROBE recommendation (Rothwell and Bhatia, 2007) • STROBE guidelines nevertheless incorporate a reliance on statistical significance testing into the canon of reporting (by examples) (Item 12b: “…were explored by testing for biologic interaction according to Rothman”) (Rothman and Poole, 2007) STROBE Meeting

9. Specific criticism: Title and Abstract Title and Abstract Item 1a: Title • Recommendation to include study design in the title. Given the use of key words for indexing, this is not essential (Sellers, 2008) Item 1b: Abstract, E&E • Limitations are mentioned in the abstract. But abstracts are usually very short, therefore it is not practical (Sellers, 2008) STROBE Meeting

10. Specific criticism: Introduction/Methods Introduction • Not enough emphasis on presenting a clear definition of the hypothesis in the introduction, its biologic rationale, and its implication to the health of the public (Kuller and Goldstein, 2007) • Methods • It is not mentioned that a careful definition of host factors is essential, including a clear statement of results specific to race, sex and ethnicity (Kuller and Goldstein, 2007) • The definition and prespecification of outcomes is crucial, particularly in cohort studies, where composite outcomes are often used to increase statistical power (Rothwell and Bhatia, 2007) STROBE Meeting

11. Specific criticism: Methods Item 7: Variables (E&E) • STROBE recommends an inclusion of an appendix with a list of variables. Although this is helpful, it adds excess pages, with consequent effects on cost and number of articles we can publish. Maybe better idea: creation of study website (Sellers, 2008) Item 10: Study size • “Explain how the study size was arrived at” <-> „… not bother readers with post hoc justifications for study size or retrospective power calculations” (E&E); this is contradictory (Rothman and Poole, 2007) STROBE Meeting

12. Specific criticism: Methods Item 12c • Maybe the STROBE group could take an initiative promoting a more systematic approach to missing values in observational studies? (Thelle, 2009) STROBE Meeting

13. Specific criticism: Results Item 16c: Main results • “Consider translating estimates of relative risk into absolute risks for a meaningful time period” • instead of “absolute risk”, it should say “risk difference” or “absolute change in risk” (Rothman and Poole, 2007) Item 19: Limitations • STROBE does not say anything about strengths, only limitations (EpiMonitor, 2007) STROBE Meeting

14. Specific criticism: Results • Studies should be required to present the actual rates or number of events, as well as HRs and relative risk. Multiple regression analyses should provide information about the number of individuals and events related to each of the independent variables (Kuller and Goldstein, 2007) • It should be stated that the p values should be restricted only to those hypotheses that were generated prior to the data analysis (Kuller and Goldstein, 2007) • Importance of reporting data on the precision of measurement of the exposure(s) under study deserves particular emphasis (Rothwell and Bhatia, 2007) STROBE Meeting

15. Specific criticism: Discussion Item 20: Interpretation • Instead of “Cautious overall interpretation of results”: “give an interpretation of the results appropriate to the available data” (Ebrahim and Clarke, 2007) Richard Doll: “If you find something that is unexpected and is going to be of social significance you have a responsibility to be sure that you’re right before you publicize your results to the rest of the world. This does at least require repeating some of your observations” (instead of “cautious overall interpretation”) (Ebrahim and Clarke, 2007) STROBE Meeting

16. Specific criticism: Discussion Item 21: Generalisability (E&E) • “A discussion of the existing external evidence is particularly important for studies reporting small increases in risk” -> but studies with small risks are not nearly as dangerous as reports of large increases/decreases in risk! More external evidence is required in order to protect the public from false claims if high risks are presented. (Ebrahim and Clarke, 2007) STROBE Meeting

17. Specific criticism: Discussion • STROBE does not include that a health warning or recommendations should accompany observational publications (Lloyd, 2007) • The evidence to be considered in the discussion should include for and against the biologic plausibility of the findings (incubation period, temporal relationship between associations) (Bradford Hill on biologic plausibility -> not mentioned in the STROBE checklist, only in 1 paragraph in the full STROBE document) (Kuller and Goldstein, 2007) STROBE Meeting