Download
1 / 46
470 likes | 712 Views
So… You Want to Review Scholarly Papers?. Daniel Amyot. EBC 8101, January 31, 2019. Faculté de génie | Faculty of Engineering. uOttawa.ca. genie.uOttawa.ca | engineering.uOttawa.ca. Overview. What and Why Ethics Guidelines Tools. Publication with a reputable publisher assumes:
E N D
So… You Want to Review Scholarly Papers? Daniel Amyot EBC 8101, January 31, 2019 Faculté de génie | Faculty of Engineering uOttawa.ca genie.uOttawa.ca | engineering.uOttawa.ca
Overview What and Why Ethics Guidelines Tools
Publication with a reputable publisher assumes: Peer review Editorial processes adhere to industry agreed ethical standards Among leaders within the field Role of the Publisher[Wiley, 2014] Provisions for: Copy editing Typesetting Author tools Provision of electronic editorial offices Funding of Receiving editors Provides a searchable platform Index servicing Article linking Promotion/marketing Ensures a version of record is available in perpetuity Digitization of legacy material. Maintaining the completeness of the academic record Event sponsorship Grants and awards Author/referee workshops Development of new services/technologies to assist researchers
What is a Peer Review? Scholarly peer review (also known as refereeing) is the process of subjecting an author's scholarly work, research, or ideas to the scrutiny of others who are experts in the same field, before a paper describing this work is published in a journal or as a book. The peer review helps the publisher (that is, the editor-in-chief or the editorial board) decide whether the work should be accepted, considered acceptable with revisions, or rejected. Peer review requires a community of experts in a given (and often narrowly defined) field, who are qualified and able to perform reasonably impartial review. [https://en.wikipedia.org/wiki/Scholarly_peer_review]
What a Peer Review is Not A peer review checks the likelihood of reproducibility, but it does not recreate the experiments to verify reproducibility A peer review suggests at improvements, but it does not fix the details of an experiment A peer review does not accept/reject a paper; this is only a recommendation to the journal editors or conference program chairs A peer review is not a means to block or slow down the competition A peer review is not an opportunity to evacuate frustrations, animosity, or anger!
Why Review Papers? http://www.phdcomics.com/comics.php?f=1760
Why Review Papers? • Because my supervisor wants me to… • So the system works! Each time you submit a paper, ~3 people must review it. Science needs you! • It is your responsibility to do your share of the load • To gain knowledge about a domain • To gain early access to emerging results • To contribute to knowledge through vetting • To learn how to write good papers… and bad ones • To develop critical thinking • To build one's reputation in a community • To eventually organize conferences, or edit journals or books • To get promoted • To make money? Keep dreaming… Volunteer work!
But What do I Know… • Editorial board member of • Software and Systems Modeling (SoSyM) • Empirical Software Engineering (EMSE) • International Journal of Healthcare Information Systems and Informatics (IJHISI). • Chaired 11 conferences/workshops • Program committee member in 112 conferences and workshops • Reviewed 107 journal papers and book chapters, as well as 474 conference papers, since 2002 (average 6+28/year). • Evaluated 48 PhD theses, 83 Master’s theses, and 36 PhD proposals… in 19 universities from 9 countries on 5 continents • … and still learning!
Should I Accept to Review a Paper? http://www.phdcomics.com/comics.php?f=920
Accept if All Criteria are Satisfied… • You have the required expertise… and interest • You have the time and can meet the deadline • Implies that you understand the review process, and the amount of work required • There is no conflict of interest with the authors • The venue is not about predatory publishing • You know how to write a review! • Hopefully, this course will help! Also: • If you are invited as a sub-reviewer, make sure the journal/conference allows for sub-reviewers to be involved… • If you decline, try to offer alternative candidates
Wiley’s Best Practice Guidelines on Publishing Ethics Adopt journal policy and practice that supports ethical best practice Support efficient, effective, ethical peer review Be mindful of breaches of publication ethics Disclose conflict of interest Accurately list those who contributed to the work and how Comply with discipline guidelines for reporting standards Ensure that ethical and responsible research is published Take action and alert journals to suspected malpractice Correct errors where found Protect intellectual property
Author’s Ethics and Responsibilities • To gather and interpret data in an honest way • To give due recognition to published work relating to their manuscript • To give due acknowledgement to all contributors • Notify the publisher of any errors • To avoid undue fragmentation of work into multiple manuscripts (salami publishing) • To ensure that a manuscript is submitted to only one journal at a time [Wiley, 2014]
Editor’s Ethics and Responsibilities Ensure efficient, fair, and timely manuscript processing Ensure confidentiality of submitted manuscripts Make the final decision on a submission Not use work reported in a submitted manuscript for their own research Ensure a fair selection of referees Act upon allegations of scientific misconduct Deal fairly with author appeals [Wiley, 2014]
Reviewer’s Ethics and Responsibilities Ensure confidentiality of manuscripts and respect privileged information Not to withhold a referee report for personal advantage Return to editor without review if there is a conflict of interest Inform editor quickly if not qualified or unable to review Judge manuscript objectively and in timely fashion Explain and support recommendations with arguments and references where appropriate Inform editor if plagiarized or falsified data is suspected [Wiley, 2014]
Ethical Misconduct • Falsifying data • Fabricating data • Plagiarism • Multiple concurrent/dual submissions • Image manipulation • Authorship misrepresentation • Duplicate publication
Tri-Council Ethical Rules [source] A conflict of interest may be deemed to exist or perceivedas such when review committee members, external reviewers or observers • are a relative or close friend, or have a personal relationship with the applicants; • are in a position to gain or lose financially/materially from the funding of the application; • have had long-standing scientific or personal differences with the applicants; • are currently affiliated with the applicants’ institutions, organizations or companies—including research hospitals and research institutes; • are closely professionally affiliated with the applicants, as a result of having in the last six years: • frequent and regular interactions with the applicants in the course of their duties at their department, institution, organization or company; • been a supervisor or a trainee of the applicants; • collaborated, published or shared funding with the applicants, or have plans to do so in the immediate future; or, • been employed by the institution, when an institution is the applicant; and/or • feel for any reason unable to provide an impartial review of the application.
Know Which Conflict Rules Apply! SIGSOFT Conflicts of Interest and Confidentiality of Submissions SIGSOFT requires members of the program committees of its events to adhere to the highest of ethical standards. Program committee chairs must ensure that these standards are not only met to the letter of the policy but also to the spirit of its intent. This means that even the appearance of a conflict of interest or breach of confidentiality in the selection process should be avoided. A program committee member (including the chair of the committee) is considered to have a conflict of interest on a submission that has an author in any of the following categories: the person themselves; a past or current student or academic advisor; a supervisor or employee in the same line of authority within the past three years; a member of the same organization (e.g., company, university, government agency, etc.) within the past three years; a co-author of a paper appearing in publication within the past three years; someone with whom there has been a financial relationship (e.g., grants, contracts, consultancies, equity investments, stock options, etc.) within the past three years; someone with whom acceptance or rejection would further the personal goals of the reviewer (e.g., a competitor); a member of the same family or anyone considered a close personal friend; or someone about whom, for whatever reason, their work cannot be evaluated objectively.
What Makes a Manuscript Good? Contains a clear, useful, and exciting scientific message. Flows in a logical manner that the reader can follow. Is formatted to best showcase the material. Is written in a style that transmits the message clearly. [Elsevier, 2010]
On Guidelines • Many guidelines and experience reports exist • What did you learn from • Hames, I. (2013) COPE Ethical Guidelines for Peer Reviewers. COPE Council. • Pain, E. (2016) How to review a paper. Science, AAAS. • Wilson, J. (2012) Peer Review: the nuts and bolts. Sense about Science. • I recently used Offutt’s guidelines (2007)
Questions Reviewers Ask [COPE, 2013] Does the paper fit the standards and scope of the journal it is being considered for? Is the research question clear? Was the approach appropriate? Is the study design, methods and analysis appropriate to the question being studied? Is the study innovative or original? Does the study challenge existing paradigms or add to existing knowledge? Does it develop novel concepts? Does it matter? Are the methods described clearly enough for other researchers to replicate? Are the methods of statistical analysis and level of significance appropriate? Could presentation of the results be improved and do they answer the question? If humans, human tissues or animals are involved, was ethics approval gained and was the study ethical? Are the conclusions appropriate?
The Referee Report [Wiley, 2014] Is the motivation clear and is it important? Is the work novel and original? Are the conclusions supported by the data? Are the results important? Are there any ethical questions? Were any flaws or mistakes found? Should anything be added or removed? Does the author demonstrate a knowledge of prior work in the field? How might the article be improved? Will the community find the article useful?
Beyond Genericity… Read the Criteria! • Journal/conference scopes, types of papers, and evaluation criteria vary substantially. • For example, RE’19 has 4 types of papers • Research, Industry, Artefacts, RE@Next! • MISE 2019 has very different types and criteria • Research, Short paper, Tool demo • Again, the Journal of Pathology Informatics has vastly different expectations • Research article, Review, Technical note, Viewpoint or Editorial, Commentary, Guideline, Book review, Letter to the editor, Symposium
Processes do Vary… Reviewer’s Angle Single-blind, double-blind, open reviews One pass, multiple passes Expertise documentation (or none) Bidding/preferences (or none) Identification of conflicts of interests (or none) Access to other reviews (or none) Discussion (or none) Rebuttal (or none) Access to reviews of other papers (or none) Review delegation (or none) Program boards in addition to PC (or none)
On Double-Blind Reviews • Here some experience from ASE, PLDI & OOPSLA 2016. To appear in CACM: https://arxiv.org/abs/1709.01609 • C. Le Goues et al., Effectiveness of Anonymization in Double-Blind Review, 2017 • Why not go with open, signed reviews, instead?
Typical Bidding/CoI Message Dear <conf> PC members, Our thanks to the PC members who have already completed bidding. This is a repeat of the first "bidding is open" message for those who have not yet completed bidding. We have received 96 abstract submissions for the <conf>. Thank you for your help in attracting those submissions! It is now time to indicate your conflicts of interests and bids for papers you'd like to and have the expertise to review. The hard deadline for entering your bids is February 16, 2017. Please provide many high bids, as we want to make sure that all papers are reviewed by 2 PC members (in addition to the RC members) who are interested in the paper. To perform the bidding please: 1) Login to easychair at <some URL> 2) Provide your bids by clicking on “Paper Bidding”. Please make sure to indicate all your conflicts can take many forms so please read the CoI guidelines at: http://www.sigsoft.org/policies/pgmcommittee.html#con_int If you have any questions about CoI please let us know. Easychair allows 4 levels of bids, yes, maybe, no, conflict: * yes. Means high priority, indicating that you are an expert on the paper's topics, and that you would be very pleased if you could review that paper. * Maybe. Means a lower priority, indicating that you would like to review that paper, in case not enough reviewers indicated a high priority for that paper. * no. Select this if you just don't want this paper to be assigned to you. Use this option only if you really will not be able to review the paper. * conflict. See the guidelines mentioned above Let us know if you have any questions
Typical Structure of a Review • Summary • To show your level of understanding of what is important • Highlight of pros and cons • 2-4 of each, bullet points • Detailed comments and questions, if any • Often numbered, to be referenceable in the rebuttal phase or in responses from authors • Smaller points and presentation issues • A structure can also be imposed by the journal, to get answers to specific questions
Check if the Work Already Exists! • Check the two papers on the next slide, published one month apart by the same publisher… • http://dx.doi.org/10.1007/s13721-016-0130-9 • http://dx.doi.org/10.1007/s13278-016-0362-9 • Can be in a different language too! • I had papers “translated” to Serbian and Chinese, and republished by others under their names! • Contact the program chair or editor if plagiarism or unethical behaviour is detected • Anything weird about Amelec Viloria?
Example of Bad Reviews on an SLR • Reviewer 1: Author(s) raises important, but not quite popular question. Aforementioned topic may be very popular in the future due to quick development of the medical simulation centers. • Too vague in what needs to be fixed and how • Reviewer 2: The contents in the paper in the following sections is completely theoretical/text-book based and should be removed - Literature Review Objective, LITERATURE REVIEW METHODOLOGY, Searching Stage, Evaluating Stage, and Synthesizing Stage • Reviewer definitely does not know what is an SLR • Reviewer 3: The paper is well written. I could not figure out which software tool you used for Discrete event simulation. • The paper is not using any tool… Has the reviewer read it? • Many more examples online!
On Journal Reviews… • If this is a review for a journal, beware of your comments! • Each of your comments will need to be addressed explicitly by the authors in their answer: • The authors will either implement it and show how this was done, or • The author will need to argue convincingly against it, and they might be shy to do so… • Do not come with new demands in other rounds!
On Rejection… https://twitter.com/AcademiaObscura/status/942365039658520576?s=03
On Rejection… • Technical/scientific issues • Motivation unclear/unimportant • Novelty/originality • Conclusions do not support the data • Results less important • Results uninteresting • Ethical questions • Unclear presentation
But Always Aim to Accept Papers Too easy to criticize papers… More pleasant for editors and chairs to have to choose between acceptable papers than to find the publishable ones among unrecommended papers! Many studies show that people tend to reject papers, often in a random pattern Do not compromise on quality, but think positive
CyberChairPRO: Peer Pressure! The tool shows all reviewers who has completed their reviews and who has not, in real-time. A good incentive to provide them in time!
Demo and Online Videos • CyberChairPRO (for conferences) • http://www.borbala.com/ • EasyChair (for conferences) • https://www.youtube.com/watch?v=dnx6AGMZ9k4 • https://www.youtube.com/watch?v=StvI2i48mYM • Manuscript Central (for journals) • https://www.youtube.com/watch?v=BW-FYUEARi0
Daniel’s Golden Rule • Review papers in the same way you would like your papers to be reviewed! • Read the paper! • Recognize the good things and the weaknesses • Provide serious and constructive comments, with competence and diplomacy • A paper does not need to be perfect for it to be acceptable, as long as its benefits outweigh its imperfections • If you ask for more X and if there is a space constraint, then suggest where there could be less Y.
Last Piece of Advice Start reviewing papers sooner than later during your studies If you are considering an academic career, get involved in a research community through a conference or workshop Contribute, and try to join a conference or workshop Program Committee (or Organization Committee) before you graduate. Your supervisor(s) might be able to open doors!
References Trevorrow, P. (2014) Author Workshop Publishing and Evaluating Research. Wiley. Walsh, T. (2001) How to write a review. University of York, UK Hames, I. (2013) COPE Ethical Guidelines for Peer Reviewers. COPE Council. Pain, E. (2016) How to review a paper. Science, AAAS. Wilson, J. (2012) Peer Review: the nuts and bolts. Sense about Science. Conflict of Interest and Confidentiality Agreement for Review Committee Members, External Reviewers, and Observers. Government of Canada. Alpert, J. (2012) 10 tips for reviewing scientific manuscripts – and 5 red flags. Elsevier Connect. Lucey, B. (2014) 10 tips from an editor on undertaking academic peer review for journals. Elsevier Connect.
