Reviewing Peer Review

1. Reviewing Peer Review Vivian Siegel, PhD Director, Center for Science Communication Departments of Medicine (Genetic Medicine) and Cell and Developmental Biology

2. Outline of Talk • Me • Why peer review for research manuscripts? • Does peer review work? • Something’s going wrong and we’re all to blame • A little data • Advice to authors, reviewers, and editors • “Ethics of peer review” powerpoint created with the support of the office of research integrity • Your suggestions

3. Who am I to give this talk? • Grad student UCSF 1981-1987 • Postdoc Janni Nüsslein 1988-1989 • Postdoc YN Jan 1989-1994 • Senior Editor, Cell 1994-1998 • Deputy Editor, Cell 1998-1999 • Editor, Cell 1999-2003 • Executive Director, PLoS 2003-2005 • Sabbatical, UCSF 2005-2006 • Vanderbilt 2006- researcher editor researcher

4. At Vanderbilt: • Director, Center for Science Communication • Editor-in-Chief, Disease Models and Mechanisms (the use of animal models to advance human health) • Executive Editor, Journal of the American Society of Nephrology • Monitoring Editor, CBE-Life Science Education

5. Why do you publish? • To claim priority for a discovery • To share what you have learned and how you have learned it, so that others can build on that knowledge

6. How do you publish? • Through peer-reviewed journals of the highest possible “stature”

7. Why do journals use peer review? • To determine whether the conclusions drawn by the research are justified and “new” • To get some gauge as to the significance or potential significance of the work • Ideally, this is then turned into a decision that is constructive, transparent, timely, and fair

8. Do you know any papers that weren’t peer reviewed?

9. Does peer review work? • Often, reviews fail to address the technical quality of the work • Often, reviews are dismissive • Often, reviewers seem to ask for additional work just because they can or they think they should • Often, editors insist on additional experiments that won’t change the paper • If a paper is rejected, the whole process has to be started over again, because (in general) the reviews can’t be used at another journal

10. It is ironic that, in an era known for the great speed and availability of information -- where we could choose to blog our results rather than submit them to journals -- publishing papers seems slower and more painful than ever before.-Siegel, “The Promise of Peer Review”, DMM, 2008

11. Is it really slower?some data…. • 26 papers in JCB • June 1985 • June 1997 • June 2008 • Time from Submission to Acceptance • Average and Median

13. Overall P=0.0015 (Kruskal-Wallis) Dunn’s Multiple Comparison Test June 1985 vs. June 1997: P>0.05 June 1985 vs. June 2008: P<0.05 June 1997 vs. June 2008: P< 0.01

14. It has become painful to publish “The competition among scientists is so outrageous these days, that asking for more experiments is mostly used as a tool to slow down a competitor, either to gain time to publish competing results or in general, to not letthe competitor get too far ahead.” -Günter Blobel

15. It has become painful to publish “Sadly, career advancement can depend more on where you publish than what you publish. Consequently, authors are so keen to publish in these select journals that they are willing to carry out extra, time-consuming experiments suggested by referees, even when the results could strengthen the conclusions only marginally.” -Martin Raff et al., 2008, Science

16. It has become painful to publish

17. Why are we always asked for more? • Authors send papers before they are “ready” to be published – that is, without the benefit of critical, honest, and constructive feedback, which would ideally be obtained prior to submission. • Reviewers think it’s their job to take the paper to the “next level,” no matter how good it is when submitted. • Editors are shirking their own jobs as editors. Editors should define the standards of the journal and decide, based on those standards and the comments of the reviewers, what they should publish. Instead, editors often overextend the job description of the reviewer, asking them not only to comment on the strengths, weaknesses, and potential impact of a paper, but also to serve as pseudo-editors, turning their impressions into a recommendation for publication in a particular journal.

18. Time for change • What can we do as authors, reviewers, and editors to improve the process? • What might be changed in the “infrastructure” of publishing to make the process more constructive and valuable?

19. Authors send papers before they are “ready” to be published. • Advice: seek the honest and critical advice both of people who have a deep understanding of the work, who can function as pre-reviewers, and of those who are interested but not involved, who can function as pre-editors

21. Authors • Take every opportunity (cover letter, introduction, etc.) to express clearly why the question being answered by the paper is worth answering • Provide enough experimental detail for reviewers to judge the technical quality of the work • Be honest • Be strategic? • Insist on a constructive process - vote with your feet if you have to!

22. Reviewers think it’s their job to take the paper to the “next level.” • Are we modeling the reviews that we, as researchers, have received? • Are we applying “journal club” thinking to reviewing? • Do we feel that the more critical we are as reviewers, the more likely we are to be asked to join an editorial board? • Or are we leveraging the power of reviewing to slow down our competition, as suggested by Blobel?

23. Reviewing a paper is not about demonstrating your own scientific mettle • if editors didn’t think you knew the field and the approaches within the paper well enough to help them understand its strengths and weaknesses, they wouldn’t have sent you the paper in the first place. • Advice: break the habit of automatically asking for more

24. Ethics of Peer Review: A Guide for Manuscript Reviewers The next 18 slides are adapted from: Sara Rockwell, Ph.D. Departments of Therapeutic Radiology and Pharmacology, and Office of Scientific Affairs, Yale University School of Medicine A course developed with support from the HHS Office of Research Integrity ori.dhhs.gov/education/products/rcr_peer_review.shtml

25. What do the editors look for in reviewers? • Expertise in one or more areas of paper • Objectivity • No conflicts of interest • Good judgment • Ability to think clearly and logically • Ability to write a good critique • Accurate • Readable • Helpful to editors and authors • Reliability in returning reviews • Ability to do the review in the allotted time frame

26. From an editor’s point of view the ideal reviewer • Is a researcher who is working in the same discipline as the subject of the paper yet is not in direct competition with the authors • Will understand the hypotheses underlying the work • Will be familiar with the model systems and methods used in the project • Will be able to judge the quality of the data and analyses and assess the validity of the conclusions • Will be able to assess the significance of the work

27. By agreeing to review a paper, the reviewer contracts to become a consultant to the journal and to adhere to the journal’s policies and guidelines for the review of manuscripts • The reviewer agrees to provide a review that meets the needs and standards of the journal within in the time specified. • The reviewer also incurs responsibility for setting the standards of the field of study. • The reviewer must be able to judge fairly and objectively the quality and significance of the work under review. • The reviewer is obligated to support and encourage publication of work of high quality while appropriately challenging flawed work. • Before agreeing to review a paper, the reviewer should consider her/his ability to meet these standards.

28. The reviewer also agrees to confidentiality • The reviewer may not use the information in the paper in their own research or cite it prior to publication. • This may raise serious ethical issues if the work provides insights or data that could benefit a reviewer’s own thinking and studies.

29. The reviewer is the agent of the journal, not the friend - or enemy - of the author • A seriously flawed paper must be challenged • Work of high quality must be supported

30. Reviewers must be wary of unconscious biases • Positive results are viewed as more exciting than negative results and are therefore more likely to be published • Bias toward a benefit from a experimental drug in a clinical trial • Bias toward finding a toxic effect associated with an environmental pollutant • Papers that challenge existing dogma or that present surprising findings may be dismissed too readily during the review process • Bias against surprising new ideas • Bias against very novel techniques

31. The journal needs scientific expertise, not editorial assistance • Journals rely on reviewers to evaluate the quality, importance, and novelty of the science presented in the manuscript. • Editors frequently receive reviews that focus completely on minor editorial problems (typographical errors, misspellings) and do not comment on the science in the paper. • Such reviews have limited value as they do not advise the editor on the importance and validity of the science and do not help the editor to make an informed decision concerning publication.

32. Appropriate editorial comments • sentences or paragraphs where the wording is sufficiently erroneous or ambiguous that the science is unclear • scientific misstatements • errors in referencing • A note that a manuscript requires major editorial assistance or a warning that a manuscript is so carelessly prepared that the science cannot be rigorously reviewed is always in order. • Reviewers should not waste inordinate amounts of time correcting minor problems with spelling, grammar, or punctuation.

33. Focus on the science • The review should focus on the science: the appropriateness of the techniques, the strengths and weaknesses of the experimental design, the quality of the data and analyses, and the appropriateness and impact of the conclusions drawn by the authors. • The comments made in the review should present clearly the reviewer’s analysis of the quality, novelty, and importance of the science and the effectiveness and appropriateness of its presentation in the manuscript.

34. The reviewer should consider and comment on • The importance and novelty of the work • The appropriateness of the materials, methods and experimental model systems • The rigor of the experimental design (including the inclusion of appropriate controls) • The quality of the data • The appropriateness of the statistical analyses • The rigor of the interpretation of the data • The validity of the conclusions drawn in the paper • The value of the discussion of the data

35. The reviewer may also comment on • The length of the paper • The writing quality • The clarity, accuracy, and completeness of the figures and tables • The accuracy and adequacy of the introduction which frames the area of the research, of the discussions of prior and related work, and of the citations to the literature

36. Reviewers should also raise ethical concerns, if they have them, including • Concerns about the ethics of studies using animals • Concerns about the ethics of studies using human subjects • Undisclosed conflicts of interest on the part of the authors • Failure to acknowledge or consider related literature or data that conflict with the authors’ findings or viewpoint

37. Concern about duplicative publication or plagiarism • The reviewer may recognize much or all of the paper, because some or all of the paper has been published previously by the same authors. • The reviewer may find text or ideas which have been copied without permission or appropriate attribution from the works of others.

38. Concern about the integrity of the data, analyses, and conclusions • The reviewer may feel that the data cannot possibly be correct as presented and may suspect that some data have been fabricated or falsified. • The reviewer may feel that the experiments are sound, but that data have been selected for presentation, manipulated or analyzed inappropriately, so that the conclusions drawn from them are deliberately misleading.

39. What should the reviewer do in such cases? • The reviewer should carefully review the facts underlying his/her concerns. • In the case of suspected duplicative publication or plagiarism, the reviewer should obtain and carefully examine copies of the original documents to confirm his/her initial impression. • The reviewer should contact the editor in confidence to discuss the problem and should provide the editor with copies of the relevant documentation.

40. Writing the review • Reviews must be clear, concise, and accurate. • Although their primary purpose is to advise the editor, comments to the author frequently are of value in guiding revision of the paper for the same or a different journal and in suggesting ways to improve the project by the inclusion of additional data or experiments. • Comments to the author may be very brief, especially in the case of an excellent, well prepared paper. • They may be extensive if the reviewer feels the paper has valuable elements but requires extensive revisions to present the findings effectively.

41. The reviewer should remember that the review will be sent to the authors and that it should be written in a constructive and collegial tone. • The content should be constructive and informative. • Comments and recommendations should be clear and should be supported with citations to specific areas in the text of the paper. • When the reviewer’s criticisms rely on or are supported by data in the literature, the reviewer should provide citations to the relevant papers. • A good review should help the authors to think more clearly about their work and its design, execution, presentation, and significance.

42. Rude reviews • Some reviewers submit critiques that are so rude, snide, sarcastic, argumentative, or even obscene that they must be censored before being sent to the authors. • Some are not transmitted, depriving the author of any beneficial insights the reviewer might have had. • Rudeness, personal criticism and locker room humor are never appropriate. • Even the most serious scientific criticisms can be worded and presented in such a way as to be constructive and collegial. • Reviewers should write critiques using a style and tone that they would want to see in the reviews that they or their trainees receive. • Reviewers should remember that they are setting the standards of behavior and collegiality for their field, as well as the standards of science.

43. Summary of reviewer advice • Be a reviewer, not an editor • Judge what you can judge - refrain from offering an “expert” opinion on what you don’t know • Provide constructive feedback - the type of review you would like to see. Forget about the last paper you had rejected - this isn’t about you. • Did the experiments answer the question? If not, what would they need to do to answer the question? This is not about what they could do but what they need to do. • Is the question worth answering? Why or why not? If not, what question would be?

44. Editors are shirking their own jobs as editors • A reviewer, someone who has a deep understanding of the field and of the techniques used in the paper, can provide guidance on the strengths and weaknesses of the work and on its potential impact, both to the field and even possibly beyond it. • Turning these assessments into a recommendation for whether a particular journal should publish the paper requires both an understanding of and a commitment to that journal’s standards for publication. It also requires a sense of other decisions that are being made by the journal at the same time, and the ability to make an unbiased recommendation.

45. Advice to editors: • relieve reviewers of the responsibility to make recommendations for or against publication • maintain a separate, much smaller pool of editors who can be dedicated to the journal and to its standards, and who can discuss the decision with full knowledge of other papers being considered by the journal. • take responsibility for your decisions and do not hide behind the recommendations of anonymous reviewers.

46. Editors • You set the standards for the journal, not the reviewers • Provide clear and constructive decisions guiding the author to the most important elements of the reviews. Don’t just look for “ammunition to kill a paper”. • Recognize your duty to publish - don’t hold up a paper for an experiment that won’t change your underlying view. • Listen to appeals.

47. Ideas for Change? • Double-blind peer review • Open peer review • Journal-blind peer review • Others?

48. Together, we can build a constructive and honest community of scholars that fulfills the promise of peer review by making it integral to the conversation of research, rather than simply a judge over it. Thank you!

49. ori.dhhs.gov/education/products/rcr_peer_review.shtml db.biologists.com/blog dmm.biologists.org