1 / 9

Thoughts on how to improve reviews

Allow PC members to ask authors simple

Antony
Download Presentation

Thoughts on how to improve reviews

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

    Slide 1:Thoughts on how to improve reviews

    Paul Francis (Presented by Robbert Van Renesse) Cornell

    Slide 2:Main problem with current system

    Reviews are often of poor quality I suspect main reason is that PC members are overworked Both ideas in this presentation are focused on this one problem alone

    Slide 3:Idea One:

    Allow PC members to ask authors simple “where in the paper can I find this?” questions Authors can answer ONLY with page, column, and line numbers Why? Often when I reject a paper, it is based on one or a few specific flaws. It would be good to verify that I’m not overlooking something.

    Slide 4:A reviewer comment

    “But you can do this today...just ask the PC chairs to forward an email” My rebuttal: Bothering the PC chairs is a significant deterrent Should really limit author reply to text already in paper...otherwise author is effectively submitting more than what is in the paper

    Slide 5:Today

    Slide 6:Proposal One: Pass reviews on

    Slide 7:Observations

    Reviewer gets the benefit of previous reviewers Various ways this can be exploited: To improve review: Reviewer first does complete unbiased review, then uses previous reviews as sanity check To reduce work: Reviewer scans paper, previous reviews, and rebuttal, and then does only enough work to decide if author has overcome previous criticisms This can be done in good faith

    Slide 8:Observations

    Good paper should be able to overcome earlier criticisms Without having to torque paper to satisfy future reviewers.... Instead author can use rebuttal Truly bad paper should not be able to overcome earlier criticisms Prevents bad paper from retrying until it gets lucky Might discourage “hail-mary” submissions

    Slide 9:A reviewer comment

    “This will promote reviewer laziness” My rebuttal: A good-faith reviewer will benefit from earlier reviews A lazy reviewer already has many ways of being lazy (form instant opinion, do minimal work to support opinion...)

More Related