How to review a paper

1. How to review a paper by Fabio Crestani

2. Disclaimer • There is no fixed mechanism for refereeing • There are simple rules that help transforming a review in a constructive document • In time you will develop your own style of refereeing • This talk mostly reflects my style

3. Purpose • A review serves several roles • The precise combination varies with the type of review • Technical/experimental approach and analysis • Computation • Ignorance of related research • Presentation style • Patents or legal issues • In our case: quality control!

4. Method • Who is interested in quality? • Program chair • Journal/book editor • You! • How do you assure quality? • Papers are reviewed by expert in the field • Peer review • You!

5. The process Journal Conference Submitted to Editor in chief Program chair Area “generalist” (choose appropriate reviewers) Editors / Editorial Board members Program committee (meta reviewers) Area specialist (peer) Referees Referees Recommendations

6. The role of the referee • Advisory • Authors bears responsibility for correctness of result presentation • Editor bears responsibility for acceptance/rejection

7. Why doing it? • Several reasons • Enhance reputation (with editor/prog. committee) • Expedites processing of your own papers • Get on editorial board or program committee • Good practice • Increase your own critical appraisal ability • Your papers become better • Sometimes it gets preferential treatment for your papers • … but refereeing means more work!

8. Consideration • Most reviews have strict deadlines • By agreeing to review you take the responsibility of doing a thorough job • If you cannot commit to this, notify the editor asap • Editors understand you may not have the time, but are unforgiving if you commit and do a poor job • Good editors keep a list …

9. Types of reviews • Anonymous • Standard for journals and conference • Can be “blind” or “double blind” • Friendly • When you ask a colleague to give comments on a draft • Has limitations … • Internal • Used mostly in commercial labs for patents and legal issues

10. How to do a review • Plan to read the paper 3 times • To get a feel for it • Read the paper in depth • Mark in up • Fill out the review right after the 3rd reading, while things are still fresh in memory

11. Review structure • The actual refereeing form • General comments on the paper • Specific comments on the paper • Confidential note to editor • General idea: be professional and non-hostile: write the review in a style that you would like to receive for your paper

12. The refereeing form • Forms might look quite different but basically ask the same things (see examples) • Poorly designed ones just have yes/no answers, good ones prompt the referee to elaborate • Make sure you read and understand it well

13. General comments • Usually starts with 1-3 sentences summarising the paper to show that you understood it • Discuss author’s assumptions, motivations, technical approach, analysis, results, conclusions, references. • Be constructive, suggest improvements

14. Specific comments • Comments on style, figure, grammar, spelling mistakes, etc. • You can mark up directly on the paper or type in list (or bullet points) form, with reference to the page, section, etc. • It is up to you to decide the level of detail of your specific comments • You are not asked to rewrite the paper!

15. Confidential note • Comments to the editor that you do not want the author to see • Not necessary and do not feel obliged • Remember the review should mostly help the author, so do not “hide” comments

16. Outcome • Usually: • Accept the paper as it is • Paper requires minor changes • Paper requires major changes (with or without a new refereeing process) • Reject publication of the paper • You can only suggest, the choice is not yours • Decision is based on at least 3 reviews

17. What to consider (1) • Correctness • Of argument/method/algorithm/proof • Significance • Rule out the obvious/trivial solutions • Valid problem • Significance to area/journal • Innovation • Original, novel • Not trivial extension or combination of old work

18. What to consider (2) • Interesting • Well motivated • Relevant (when and where) • Timeliness • Of current interest to community • Take into account: publication delay, pre-exposure (WWW) • Succint • Message should be: clear, compelling, to the point

19. What to consider (3) • Accessible • Is it appropriate to the audience • Readable, good grammar, good structure • People do not have the time to read badly written papers

20. Ethics of refereeing (1) • Objectivity • Judge paper on its own merits • Remove prejudice • If you are not able to review it, return it • Fairness • Author may have different point of view / methodology / arguments • Judge from their school of thought not yours • Speed • Be fast, but do not rush. Author deserves a fair hearing

21. Ethics of refereeing (2) • Professional treatment • Act in the best interest of the author and conference/journal • Specific rather than vague criticism • Confidentiality • Cannot circulate paper • Cannot use without permission • Conflict of interest • Discuss with editor

22. Ethics of refereeing (3) • Honesty • About your expertise and confidence in appraisal • Courtesy • Constructive criticism • Non-inflammatory language • Suggest improvements

23. Need practice? • Euan Minto Prize • Make sure you read the notes that accompany the review form • Give your draft paper to a colleague for comments … … and be ready to provide some on his/her paper!