Thoughts on how to improve reviews

1. Thoughts on how to improve reviews Paul Francis (Presented by Robbert Van Renesse) Cornell

2. Main problem with current system • Reviews are often of poor quality • I suspect main reason is that PC members are overworked • Both ideas in this presentation are focused on this one problem alone

3. Idea One: • Allow PC members to ask authors simple “where in the paper can I find this?” questions • Authors can answer ONLY with page, column, and line numbers • Why? • Often when I reject a paper, it is based on one or a few specific flaws. It would be good to verify that I’m not overlooking something.

4. A reviewer comment • “But you can do this today...just ask the PC chairs to forward an email” • My rebuttal: • Bothering the PC chairs is a significant deterrent • Should really limit author reply to text already in paper...otherwise author is effectively submitting more than what is in the paper

5. Today

6. Proposal One: Pass reviews on

7. Observations • Reviewer gets the benefit of previous reviewers • Various ways this can be exploited: • To improve review: Reviewer first does complete unbiased review, then uses previous reviews as sanity check • To reduce work: Reviewer scans paper, previous reviews, and rebuttal, and then does only enough work to decide if author has overcome previous criticisms • This can be done in good faith

8. Observations • Good paper should be able to overcome earlier criticisms • Without having to torque paper to satisfy future reviewers.... • Instead author can use rebuttal • Truly bad paper should not be able to overcome earlier criticisms • Prevents bad paper from retrying until it gets lucky • Might discourage “hail-mary” submissions

9. A reviewer comment • “This will promote reviewer laziness” • My rebuttal: • A good-faith reviewer will benefit from earlier reviews • A lazy reviewer already has many ways of being lazy (form instant opinion, do minimal work to support opinion...)