Hot topics in peer reviewed publishing

1. PEER REVIEW:EFFECTIVENESS AND RELIABILITYProfessor Roger Jones, Editor, British Journal of General Practice

2. Hot topics in peer reviewed publishing • Open access publication • Paper versus web/online publication: continuous online publication • Peer review • Post-publication review • Alternative metrics • Dissemination and media relations • Role of social media • Institutional/research assessment requirements

4. BJGP Structure • Published monthly by the RCGP • BJGP editorial team at 30 Euston Square • Editor, Deputy Editor, Journal manager, Web Editor, senior assistant editor, assistant editor, editorial and design assistant, advertising agency • BJGP Editorial Board: 14 core members plus 5 statistical advisors and a senior ethics advisor Meets 2-3 times a year in London – and throughout the year by e mail • BJGP International Advisory Board: 40+ members from a wide spectrum of countries Meets annually, usually at WONCA Meetings/workshops. seminars at SAPC/RCGP Conferences/NAPCRG

5. BJGP Contents • Editorials (commissioned and submitted): 3 -4 each issue • Themed issues • Original papers: 2-page summaries written by authors, full versions on line: 8-10 each issue • Debate and Analysis papers (mostly submitted): 2 per issue • No more short reports or commentaries • Clinical Intelligence: 2-page guideline summaries, technology assessments, Tips for Trainees, case reports: 2 per issue • Letters to the Editor • Out of Hours (previously the Back Pages) : opinion, views, art/music/book etc reviews (mostly freely submitted), creative writing and regular columnists • 2014 readership survey under way

6. BJGP Statistics • >750 articles submitted/year • >30% rejected prior to review • >1000 reviewers involved in the peer review process • Acceptance rate around 15% of peer-reviewed papers • ‘Accept subject to revision’ commonest decision for published papers • Role of Editorial Board important in reviews, additional opinions, mediation • Time to first decision (median) 11 days • Final decision after revision about 1 month

7. REVIEWING CRITERIA • Decisions are made by reviewers and editors • Research question, background, methodology, analysis, interpretation, discussion, implications • Presentation: writing, organisation, use of tables/figures/quotes, references • Novelty, timeliness and timing/themes • Clinical message: relevance, importance and applicability to BJGP readership • Research implications • Likely citeability • Vigilant for duplication/salami publication, conflicts of interest

8. REVIEWERS • Selection from database of reviewers (which includes time taken and review quality) • Reviewer selection involves interesting balances eg between enthusiasm and objectivity, conflicts of interest/ competition • Two week turnaround • Generally <4 reviews/year • Internal review: quality grading system • Feedback: indirect via letter to authors direct access to scores of review quality and narrative feedback from editor on request • Provision of certificates for appraisal portfolio etc

9. REVIEWING • Open peer review • Confidential comments to Editor • Comments to Authors • Recommendation on publication: Yes, definitley Yes, possibly Probably not Definitely not • Even papers with two yes definitely scores may be rejected • Accompanying editorial? • Major revision – peer review of extensively revised version

10. BJGP review scoring criteria A An excellent and timely review, providing a set of comments that are comprehensive, insightful, and clear, and are informed by a close familiarity with the topic and/or the methodology of the study. There is a clear recommendation on acceptance for publication, consistent with these comments, which are structured, immediately comprehensible to the authors, and which can act as a constructive guide to redrafting and resubmission. There are useful comments to the editor about matters such as the novelty, importance, and likely interest to readers of the BJGP. These top-class reviews often suggest additional literature and references for consideration by the authors. B A very good review, with useful and timely guidance for the editor, clear comments to the authors, and sufficient detail for resubmission and redrafting, although perhaps with less subject or methodological expertise, less incisiveness, and perhaps also missing some key details. Like Grade A reviews, these reviews are likely to run to at least 40 or 50 lines of comment, providing sufficient material to not only help authors improve their manuscript but also to reflect on their methods, findings, and interpretation.

11. BJGP review scoring criteria (cont) C An adequate review that is still useful, but may not provide a comprehensive opinion or absolutely clear advice to the editor. This may be problematic when a more detailed review has come to a different conclusion about quality or a different recommendation on acceptance, so that a further review may be needed to supplement the shortcomings in the Grade C report. D An evaluation that is too brief and superficial to be useful. It not only fails to identify significant shortcomings in the study, but also is too thin to be used as a basis for rejection. A very short review of this kind recommending acceptance can be equally unhelpful, particularly when it has to be weighed against a more guarded opinion in a more detailed report. E A review that is short, dismissive, or mildly offensive, with evidence of bias or personal animosity, with no attempt to provide objective or constructive comments and with very weak academic/intellectual content.

12. How concordant/consistent are peer review judgements? • Search for “peer review concordance” in PubMed: • Over 200 studies into concordance of reviewers’ assessments for annual medical society meetings…….. • BUT only 1 study into the concordance of journal article submissions

13. Brain. 2000 Sep;123 ( Pt 9):1964-9. • Reproducibility of peer review in clinical neuroscience. Is agreement between reviewers any greater than would be expected by chance alone? • Rothwell PM1, Martyn CN. Department of Clinical Neurology, Radcliffe Infirmary, Oxford, UK. peter.rothwell@clneuro.ox.ac.uk • We aimed to determine the reproducibility of assessments made by independent reviewers of papers submitted for publication to clinical neuroscience journals . We studied two journals in which manuscripts were routinely assessed by two reviewers. • Agreement between the reviewers as to whether manuscripts should be accepted, revised or rejected was not significantly greater than that expected by chance [kappa = 0.08, 95% confidence interval (CI) -0.04 to -0.20] for 179 consecutive papers submitted to Journal A, and was poor (kappa = 0.28, 0.12 to 0. 40) for 116 papers submitted to Journal B. • However, editors were very much more likely to publish papers when both reviewers recommended acceptance than when they disagreed or recommended rejection (Journal A, odds ratio = 73, 95% CI = 27 to 200; Journal B, 51, 17 to 155). There was little or no agreement between the reviewers as to the priority (low, medium, or high) for publication (Journal A, kappa = -0.12, 95% CI -0.30 to -0.11; Journal B, kappa = 0.27, 0.01 to 0.53). • Thus, although recommendations made by reviewers have considerable influence on the fate of papers submitted to journals , agreement between reviewers in clinical neuroscience was little greater than would be expected by chance alone.

14. Methods • 400 submitted articles for each of the years 2012, 2013 and 2014 • Inclusion criteria: • Types: Quantitative (non RCT), qualitative, RCT, Systematic Review, Quant – OA, Qual - OA • At least 2 reviewers • Decision was sent to author • Data points: • Article type • Reviewer scores • Calculated score difference • Decision to publish • Total 373 articles

15. Results from 373 articles

16. Accepted articles : 131

17. Rejected articles : 242

18. Accepted article breakdown

19. Rejected article breakdown

20. Peer review - more questions • Where are the sources of bias? Which ways do they operate? • “Objective” versus “subjective” peer review?problematic because of constraints on editorial resources, print journal pages etc • Contribution of reviewers to revised and accepted papers, and to the literature • Open versus closed review – pros and cons still unclear • Reviewer selection, training and the effect of feedback • Place of open data (and open peer review ie publication of reviews) • Role of post-publication review • Impacts/distortions of papers by short and very short summaries (which may not be peer reviewed), press releases and third party reports