Peer Reviewing Manuscripts: Methods and Techniques for Advancing Science and Your Own Career

1. Peer Reviewing Manuscripts: Methods and Techniques for Advancing Science and Your Own Career Beth Bailey, PhD Department of Family Medicine East Tennessee State University

2. CME Objectives • Participants will learn how to critically appraise all aspects of a scientific journal article • Participants will learn how to write a peer review of a scientific journal article • Participants will learn how to seek out opportunities to serve as peer reviewers • Participants will learn how peer reviewing skills and service can enhance their own manuscript writing and advance their careers

3. Bonus Content • Overview of using the “Track Changes” feature of Word • important features and how to use them • customization for your use and multiple users • how to turn it off and get rid of the extra text and comments!

4. What is Peer Review? • Process of subjecting an author’s scholarly work to the scrutiny of others who are experts in the field (AKA refereeing) • Used by publishers to select and screen submitted manuscripts • Aim is to get authors to meet the standards of their discipline and science in general; it is essential to increasing the quality of communication in the health sciences • Publications that do not involve peer review are often regarded with suspicion and judged to be of lesser quality

5. What is Peer Review? • Rationale – it is rare for an author or group of authors to spot every mistake in a complicated piece of work • Allowing others to have input increases the probability that weaknesses will be identified, and with advice and encouragement, fixed • Helps insure that information contained in scientific journals will be useful and reliable • Thus, peer review aids in the decision of whether a paper should be published, and helps to improve the paper before publication

6. How Does Peer Review Work? • An author submits a paper for publication consideration to a peer-reviewed journal • An editor does a preliminary review of the manuscript (topic suitability, general format, etc) • The editor seeks out experts in the topic area to serve as peer reviewers • Reviewers are contacted and given a set amount of time to complete their review • Reviews are submitted to the editor who makes a decision about publishing the manuscript • If revisions are requested, the revised manuscript goes back to the reviewers for a second review

7. Why Become a Peer Reviewer? • Serving as a peer reviewer can be time consuming – a single review and report could take anywhere from an hour to an entire day • Peer reviewers work for free – you must take valuable time away from other duties • By serving as a peer reviewer you are working to advance a particular field – most journals cannot publish without quality peer reviewers • By serving as a peer reviewer you have input into the direction of a particular field

8. Why Become a Peer Reviewer? • Serving as a peer reviewer builds your professional network and increases your visibility in a field; editors are often asked for recommendations • Serving as a peer reviewer can help with promotion and tenure; duties can count as research or service; shows that you are recognized as a expert in a particular field • By serving as a peer reviewer you are likely to increase your own manuscript writing skills; learn what editors and reviewers are looking for; increase chances of getting your own manuscripts accepted • Potential CME credit

9. Peer Review and Anonymity • Most journals employ a double-blind review process • Some journals use a single-blind process; possibility of bias exists (personal relationship or prestige); four separate RCTs have demonstrated little difference in quality of comments or publication recommendations when the author’s identity is known to the reviewer • Anonymity of the reviewer is important to insure honest critique; however in highly specialized fields anonymity may be hard to maintain • Anonymity of the author sometimes hard to maintain in a double-blind process; in RCTs reviewers correctly identified author and/or institution 25-50% of the time

10. Peer Review and Confidentiality • In addition to involving anonymity, the peer review process should also be treated as confidential • Manuscripts given to reviewers are privileged communications and are the private property of the author • Work and ideas should not be discussed with others until after publication (simple anonymous questions vs revelation of content) • Anyone who comes in contact with a paper under review should be aware of the confidentiality issue

11. Peer Review and Bias • Reviewers should not be selected from authors’ close colleagues, relatives, or friends • Reviewers should inform the editor of any potential conflict of interest • Authors can name those who should be disqualified from reviewing a paper • Exceptionally critical or exceptionally positive reviews may be disregarded because of suspected bias • Many have spoken out about potential bias in the peer review process (ex: breach in anonymity, bias against new findings and techniques, bias toward significant findings)

12. Who Makes a Good Peer Reviewer? • A reviewer should have expertise in all or some aspects of the paper to be reviewed (i.e. main topic, methodology, etc); must understand key hypotheses • In addition to expertise, editors look for: objectivity, good judgment, ability to think clearly and logically, ability to write a good critique, reliability • Reviewers should work in the same discipline as the subject of the paper yet not be in direct competition with the authors (i.e. does paper contain results that overlap with studies reviewer is currently performing) • Studies of peer review have defied the conventional wisdom that senior, accomplished scholars provide the best reviews

13. How to Get Selected as a Peer Reviewer • Always register whenever you submit a manuscript • Let it be known you are interested in peer reviewing • Directly email editors of journals in your area of expertise • Whenever possible, accept any offer to serve as a reviewer, especially initially; make sure your review is thorough and on-time • Register with journals that allow non-authors to submit profiles

14. Journals that Welcome Beginners (see handouts) • Open registration • British Medical Journal • Annals of Family Medicine/Annals of Internal Medicine/Annals of Surgery • Journal of the American Board of Family Medicine • Academic Psychiatry • All APA journals • Vetted registration • Canadian Journal of Nursing Research • American Journal of Health Behavior

15. How to Perform a Peer Review

16. After the Review • Once you complete your review – submission to the journal (many different online systems) • The editor compiles all of the reviews, makes a decision, and sends a letter to the author (typically available to reviewers as well) • Letter contains decision as well as all of the reviewer comments; comments specifically for the editor or specific reviewer recommendations for publication are typically not included • 4 potential decisions: • Outright rejection • Rejection with invitation (explicit or implicit) to make major changes and resubmit • Acceptance conditional on responding to reviewer comments • Unconditional acceptance

17. But Wait … There May Be More… • Unless the paper is outright rejected (rate depends on journal) or outright accepted (rare for any journal), your job as a reviewer is likely not over • If the author chooses to revise and resubmit the paper, the journal will send it back to the same reviewers for re-review • The job of the reviewer at this point: • Determine whether all of your concerns have been dealt with • Reanalyze the revised manuscript for any new issues • Write a new review addressing the latest version. In some cases, a third and even fourth re-review may be necessary

18. Final Housekeeping Duties • Keep the paper and your review(s) until the final dispensation • Keep track of all manuscripts you review (journal, topic, date) • Get copies of reviewer acknowledgement pages from the journals • Be sure to include your work as a reviewer in your FAP/FAR/FAE, your CV, productivity/background summaries, and Promotion and Tenure packets (counts as research or service; demonstrates national recognition of expertise in your field) • Submit your work to journals you • have reviewed for

19. Criticisms of Peer Review • Many have spoken out about the bias of the peer review process • Currently the best system we have • See handouts for additional discussion of these issues

20. The Nuts and Bolts of Manuscript Peer Review

21. Deciding Whether to Accept • Receive offer to peer review – what’s the first step? • Carefully review the information you have been given about the paper (abstract or full paper), and decide whether you have the expertise to review the paper • Next, make sure there are no apparent conflicts of interest • Consider whether you have the time to do an adequate review in the time frame required • Let the journal know your decision as soon as possible (within 48 hours typically); if you decline, many journals welcome suggestions for alternate reviewers

22. Beginning the Review • You have decided to peer review a paper – what’s the next step? • Set aside a block of time to do the review (helps limit the amount of time you spend) • You are on your own – most journals offer no training and little guidance on how to do a peer review; do check the journal website for whatever help is there, though • Have confidence in yourself – beginning reviewers, and sometimes even experienced ones don’t feel as if they are qualified to critique others’ work

23. Review Procedure • Typically a review will consist of 2 parts – your written comments and a journal form that you complete • Start by doing your homework – what does the journal expect; locate any needed background information • Read through the paper, jotting down your thoughts – both positive and negative • Using what you jotted down as a basis, write out your critique of the paper • Complete any required evaluation forms and submit all components of your review

24. What to Look For General Issues • Is the topic appropriate for the journal (mission statement)? • Is the research important? • Does it address an important question • Will consumers of the research recognize its importance as something they need to know • Are the findings generalizable • Is the research original? • Do the study and findings add to the existing field of knowledge • Are there any ethical concerns? • In the study • Fabrication/falsification/plagiarism

25. What to Look For Specifics • TITLE • Does the title accurately reflect the purpose, design, results, and conclusions of the study? • Is it clear what question is being answered? • Is causality improperly attributed?

26. What to Look For Specifics • ABSTRACT • Is this a succinct, clear, and comprehensive summary of the main text of the paper? • Does the abstract include the essential information from each section of the paper? • Is the content (data, conclusions, etc) consistent with that presented in the main text and with other abstract content? • Are data or other key information presented here but not in the main text?

27. What to Look For Specifics • INTRODUCTION - BACKGROUND • Does the introduction succinctly state what is known and unknown about the topic? • Are any important findings from previous studies omitted or misrepresented? Are all statements properly referenced? • Is the significance of the topic established? • If previous very similar studies have been conducted: • Are their strengths and limitations described such that the need for further study is established? • Is it clear how the experimental approach to be used in the present study is likely to yield more definitive or unique insight than these previous studies?

28. What to Look For Specifics INTRODUCTION – HYPOTHESIS(ES) Is the objective of the study clearly described? Is the specific experimental question to be addressed stated? Is the experimental question formally stated in testable terms? Is the experimental question based on previous experimental observations or theory?

29. What to Look For Specifics METHODS - PARTICIPANTS Are the participants adequately described? Is the participant population appropriate for the question presented? Is the number of participants sufficient to provide the statistical power to show effects if they are really present? Will the participant population allow for extensive or limited generalizability? Is there adequate information about participation rates?

30. What to Look For Specifics METHODS – PROCEDURE AND MEASURES Is it clear from the description whether the study is experimental or correlational? Are ethical issues such as informed consent and IRB approval described? Are there any ethical concerns? Is the methodology described in enough detail to allow others to repeat the study?

31. What to Look For Specifics METHODS – PROCEDURE AND MEASURES (contd) Are the measurement techniques used sufficiently reliable, precise, and valid? Is the rationale for making each measurement either obvious or explained? Does the design allow the hypothesis to be tested in a rigorous scientific manner? Is there a better approach that could have been used? Does the design involve control for potentially confounding factors (i.e. does the approach isolate the mechanism or factors of interest)?

32. What to Look For Specifics METHODS – DATA ANALYSIS Are the details as to how variables were calculated explained so they can be reproduced? Were calculations appropriate? Is there any information about treatment of outliers and rejected participants? Are the statistical techniques used appropriate? Are any critical assumptions of the statistical techniques violated? Are the alpha-levels used to determine statistical significance appropriate and clearly stated? Is it clear how the data will be interpreted to either support or refute the hypothesis?

33. What to Look For Specifics RESULTS Are the data reported in a clear, concise, and well-organized manner? Are data presented on any measurement that was not described in the Methods? Are the data on all measurements described in the Methods presented? Have the data been presented in the appropriate units? Have the 3 tools used to present data (text, tables, figures) been used effectively? Are all of the tables and figures needed? Are more needed?

34. What to Look For Specifics RESULTS Are the tables and figures properly worded and able to stand alone? Are any data presented in the text duplicated in the tables or figures? Is reference made to all tables and figures? Do the data seem reasonable? Is data presented to address all questions of interest?

35. What to Look For Specifics DISCUSSION Are the major new findings of the study clearly described and properly emphasized? Are the key conclusions adequately supported by the experimental data? Is there any other way to interpret and/or explain the data other than that suggested by the authors? Is the significance of the present results described? Is it clear how the findings extend previous knowledge in a meaningful way?

36. What to Look For Specifics DISCUSSION Are important experimental observations from previous reports described in the context of the present results? Do the authors support their statements with appropriate references? Do the authors discuss their data in a manner that provides insight beyond that presented in previous sections? Are the unique aspects and other experimental strengths of the study properly highlighted?

37. What to Look For Specifics DISCUSSION Are the important limitations of the study described so the reader will be able to interpret the findings appropriately? Do the authors make suggestions as to how the results of their study need to be extended in the future to learn more about the issue in question?

38. What to Look For Specifics TABLES, FIGURES, REFERENCES Do all the numbers in tables and graphs add up? Are table and graph presentations consistent with those in the text? Are tables and graphs necessary and non-redundant? Can the graphs and tables stand alone? Do the references conform to the journal format? Are all necessary references included, and are all references necessary?

39. What to Look For Additional Issues Is the manuscript concise (are there unnecessary sections that should be eliminated)? Was the paper well written, properly organized and easy to follow? Was the information presented in an open-minded and objective manner? Is there a significant conflict of financial or scientific interest? Is the manuscript properly formatted for journal specifications? Additional questions may be appropriate depending on the type of study described (see handouts)

40. Preparing Your Report Very few journals provide a structured format for your written review There are many ways to organize a review; typically editors like to see a division of major (impacts primary conclusions of study) versus minorcomments, and a division of general versus specific comments Most common mistake of new reviewers – reviewing versus editing Comments should include both strengths as well as areas of concern Any criticism should be constructive and specific about problems and how they could/should be addressed

41. Preparing Your Report Typical reviews may be as short as half a page or as long as three pages (typically positive reviews are much shorter then reviews of weak papers) Keep in mind that this review will be shared, as is, with the author. Unless specifically directed otherwise by the editor, recommendations about whether or not the manuscript should be accepted should not be included here Begin your review by briefly summarizing the study – lets editor and author know you understood the paper Begin with a couple of paragraphs detailing the major/general issues – start with a positive; indicate right up front if there are any fatal flaws

42. Preparing Your Report Next detail the more specific or minor points – make it very clear what you are referring to (page numbers, quote text), what you feel the issue is, and how it might be corrected; numbering these points and ordering them by section is helpful Don’t be afraid to exclude from the review parts of the study that are out of your area of expertise; be up front about this Conclude by summarizing the key issues and whether major or minor changes are needed and how necessary they are

43. Comments for the Editor In addition to the written report that is shared with the author, many journals also ask reviewers to complete a form where various elements are rated and an overall recommendation is given Also, many journals allow you to send additional confidential comments to the editor

44. Peer Reviewing Manuscripts: Methods and Techniques for Advancing Science and Your Own Career Beth Bailey, PhD Department of Family Medicine East Tennessee State University