Open peer review at BMJ: what we know and what we don’t (yet)

1. Theodora Bloom, Executive Editor, The BMJ Open peer review at BMJ: what we know and what we don’t (yet) Transparency, Recognition and Innovation in Peer Review. HHMI, February 2017

2. Outline What I’ll talk about today • In praise of “evidence-based publishing” • Which flavours of open peer review we’ve used (for many years) • Experiments that show the effects of variants of open peer review  • Ongoing research example: conflicts of interest • Ironing out the wrinkles with open identities DOI: I am employed by BMJ, and most of the research I’ll discuss is by current or former colleagues (but not all of it)

3. The BMJ - a long history of campaigning

4. Open identities Open reports Open final-version commenting (with open participation) Which versions of ‘open peer review’?

6. Studying peer review is hard work

7. Completed research study Peer review doesn’t find most errors, and training doesn’t help much • 607 peer reviewers randomized to receive face-to-face training, or a self-taught package, or a control • Each reviewer sent the same 3 papers, each with 9 major and 5 minor methodological errors inserted. • At baseline reviewers found an average of 2.58 of the nine major errors…The mean number of errors reported was similar for papers 2 and 3. • Training had little effect. Any effect was short-lived

8. Completed research study How long should reviewers spend on a review? • Survey of 420 BMJ papers with 690 reviews • Review quality increases with time spent on review, up to 3 hours but not beyond

9. Completed research study What makes a good reviewer? • aged under 40 • known to the editors (experienced at the journal) • methodological training (statistics & epidemiology)

10. Other completed research studies • Do tables and figures change much after peer review? [Not much] • Are author-suggested reviewers different from editor-suggested ones? [Yes, more likely to recommend acceptance]

12. Independent researchers Meta-analysis of work on peer review and openness 22 reports of randomized controlled trials - only 7 since 2004 • training (n = 5): did not improve review report quality • addition of a statistical reviewer (n = 2): improved the final manuscript • use of a checklist (n = 2): did not improve the manuscript • open peer review ([open identities]; n = 7): improved quality of the review report; did not affect the time reviewers spent on review; decreased the rate of rejection • blinded peer review ([peer reviewers blinded to authors]; n = 6): did not affect the quality of review or rejection rate

13. Fabulous summaries of research in this area: • Weighing Up Anonymity and Openness in Publication Peer Review - Hilda Bastian, May 2015 http://blogs.plos.org/absolutely-maybe/2015/05/13/weighing-up-anonymity-and-openness-in-publication-peer-review/ • The Fractured Logic of Blinded Peer Review in Journals - Hilda Bastian, October 2017 http://blogs.plos.org/absolutely-maybe/2017/10/31/the-fractured-logic-of-blinded-peer-review-in-journals/

14. Ongoing research study Conflicts of interest - declaration helps, right? Physicians’ confidence in education articles is not influenced by COI statements.

15. Still learning how best to do handle openness • Developing new expectations of behaviour • Reminding authors and reviewers that open identities doesn’t equate to ‘open everything’ • Issues differ for rejected articles http://blogs.bmj.com/bmj/2016/05/16/the-bmj-research-editors-why-the-bmj-rejected-a-weekend-effect-paper/

16. Thanks for listening Web: www.bmj.com Email: tbloom@bmj.com Twitter: @TheoBloom