1 / 13

Editorial Board Review A Few Good Reviewers Don Schneider, Ph.D.

Editorial Board Review A Few Good Reviewers Don Schneider, Ph.D. CSR Advisory Council Meeting May 19, 2014. Toward Review by the Best. Reviewer pools Past experience Cost considerations. Reviewer Pools. Reviewer Status Numbers HHMI 300 NIH R37 700 NAS 2,000 R01 26,000

finn-gardner
Download Presentation

Editorial Board Review A Few Good Reviewers Don Schneider, Ph.D.

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Editorial Board ReviewA Few Good ReviewersDon Schneider, Ph.D. CSR Advisory Council Meeting May 19, 2014

  2. Toward Review by the Best • Reviewer pools • Past experience • Cost considerations

  3. Reviewer Pools Reviewer Status Numbers HHMI 300 NIH R37 700 NAS 2,000 R01 26,000 CSR needs 16,000

  4. Past Experience • Format modeled on journal manuscript review • First piloted in 2008 with 6 SBIR panels, just in time for TR01s, Challenge grants, DP1 etc. • Two stages • First Stage – Mail reviewers • Second Stage – Editors

  5. First Stage/Mail Reviewers • Subject matter experts • Provide depth in review • Focus on scientific and technical merit • 2-3 first stage mail reviewers per application • Submit full critiques • Give overall impact and criterion scores • Overall impact scores not factored into final priority score

  6. Second Stage/Editors • Hold face-to-face meeting • Recruit broad experts • Provide perspective in review (assign about 15 applications each) • Focus on impact and significance • Assign 3 second stage reviewers per application • Consider first stage critiques in review • Write overall impact paragraph • Give overall impact score • Final priority score based on second stage only

  7. Rationale • Provides both depth and breadth in review • Optimizes use of the best reviewers • Scales well for large numbers of applications (second stage discusses a fraction of the applications)

  8. Perceived Advantages • Involves no travel/teleconference for first stage reviewers • Allows small, interactive face-to-face meetings • Promotes better scoring and assessment of impact • Lessens travel and lodging expenses and inconveniences

  9. Cost Considerations(Alicia Caffi) Review # of Applications Cost/application Regular R01 F2F $518 DP1/Pioneer EB+I 244 $280 DP2/New Innov EB 593 $124 DP5/Early Indep EB+I 84 $875

  10. Challenges • Recruitment of large numbers of reviewers • Timeline • Tight, two sequential reviews • (in the 17 week cycle) • More staff time required (SROs) • Some sense of isolation by first stage reviewers

  11. Review Outcomes • Each application examined by at least 5 reviewers • Interactive, thoughtful discussions • Overall scoring by second stage members • Reviewers and staff like final review products

  12. Survey • Survey conducted by A Kopstein of reviewers participating in SBIR pilots 2008 • Outcomes were generally positive • Majority willing to participate in either review stage in future • Editorial Board Review: • Increases expert review • 3/4ths of respondents • Preferred for their own applications • 2/3rds of respondents

  13. Discussion Hopes for a few good reviewers?

More Related