1 / 21

The Blind Leading the Blind

The Blind Leading the Blind. Prof. Short* Dr. Tall* * Names obfuscated and affiliations omitted to preserve anonymity. Outline. The five best features of double-blind reviewing Beyond double-blind. Feature #1.

lindaroberts
Download Presentation

The Blind Leading the Blind

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Blind Leading the Blind Prof. Short* Dr. Tall* * Names obfuscated and affiliations omitted to preserve anonymity

  2. Outline • The five best features of double-blind reviewing • Beyond double-blind

  3. Feature #1 • Enables useful feedback on half-baked papers, without fear of embarrassment • (keeping those under-worked PC members busy)

  4. Feature #2 • Slows the advancement of science to a manageable pace, by eliminating rapid dissemination of results

  5. Feature #3 • Allows job-seeking PhD students to allude to spectacular new results, which unfortunately they can’t talk about • (regardless of whether the results actually exist)

  6. Feature #4 • Discourages those annoying high-impact projects with recognizable names and many-author papers that build on one another

  7. Feature #5 • Facilitates “flow” of ideas from authors to reviewers • (without the irritating requirement of attribution)

  8. But… • Double-blind doesn’t go nearly far enough

  9. Problem #1 • Senior reviewers can intimidate the junior reviewers of a paper during discussions

  10. Problem #1 • Senior reviewers can intimidate the junior reviewers of a paper during discussions • SOLUTION:Triple-Blind • Reviewers don’t know who the other reviewers are

  11. Problem #1 • Senior reviewers can intimidate the junior reviewers of a paper during discussions • SOLUTION:Triple-Blind • Reviewers don’t know who the other reviewers are • This one is real!

  12. Problem #2 • Authors of high-impact papers become more famous than authors of insignificant papers

  13. Problem #2 • Authors of high-impact papers become more famous than authors of insignificant papers • SOLUTION:Quadruple-Blind • Authors of published papers are anonymous

  14. Problem #2 • Authors of high-impact papers become more famous than authors of insignificant papers • SOLUTION:Quadruple-Blind • Authors of published papers are anonymous • Someone (perhaps Jim Gray...) was 20 years ahead of his or her time with the 1985 “Anon et al.” benchmarking paper

  15. Problem #3 • System is biased in favor of authors who give great talks about their results

  16. Problem #3 • System is biased in favor of authors who give great talks about their results • SOLUTION:Quintuple-Blind • PC chair gives all the talks

  17. Problem #4 • Famous researchers decline to serve on PCs for second-tier conferences

  18. Problem #4 • Famous researchers decline to serve on PCs for second-tier conferences • SOLUTION:Sextuple-Blind • Conferences are anonymous — PC members don’t know what conference they’re agreeing to review for

  19. Problem #5 • Researchers insist on sending their best work to the best conferences, which is unfair to second-rate venues

  20. Problem #5 • Researchers insist on sending their best work to the best conferences, which is unfair to second-rate venues • SOLUTION:Septuple-Blind • Conference submissions are picked randomly from a global pool

  21. Acknowledgements • Thanks to … • An anonymous west-coast professor with a photography habit • An anonymous Midwest professor with the same first name as his (or her!) advisor • from whom we “borrowed” some of these • ideas (when they weren’t looking)

More Related