1 / 14

Standing Panels for Grant Application Peer Review: A Pilot Project

Standing Panels for Grant Application Peer Review: A Pilot Project. Presentation by Phyllis Newton Office Director, Office of Research and Evaluations National Institute of Justice. OBJECTIVES. Stronger Science Greater Transparency Greater Consistency Increased Diversity

amandag
Download Presentation

Standing Panels for Grant Application Peer Review: A Pilot Project

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Standing Panels for Grant Application Peer Review:A Pilot Project Presentation by Phyllis Newton Office Director, Office of Research and Evaluations National Institute of Justice

  2. OBJECTIVES • Stronger Science • Greater Transparency • Greater Consistency • Increased Diversity • High Quality Feedback • Cost-Effective and Cost-Efficient • Safeguards against Bias and COIs • Effective, Timely, Concise, Consistent Input

  3. Current peer review • Annual Reviews • Changes across review cycles • Identity of reviewers not disclosed • Mixed panels of researchers and practitioners • Substantial labor on: • Selecting complete panels • Contacting, confirming, scheduling

  4. NIJ’s Pilot Peer Review Process • Pilot project: 5 scientific review panels in 2012 • Testing a single model for review panels • Drawing on review processes at NIH, NSF • Final design currently being finalized • Process open to input

  5. Journey for Applications SOLICITATION APPLICANT NIJ DIRECTOR FUNDING DECISION NIJ RECEIPT SCIENTIFIC REVIEW PANEL PROGRAMMATIC REVIEW

  6. Key Features of SRPs • “Standing Panel” model • Three-year “rolling” membership • 15 researchers and 3 practitioners • Possibility for additional “ad hoc” reviewers

  7. Scientific Review Panel Process Initial Merit Review Panel Review Documentation Recommendations to NIJ

  8. Initial Merit Review • Each application reviewed twice • Reviewed by two scientific members of panel • Reviewers provide written comments for each proposal • Score individual elements defined in solicitation • Provide overall score • Scores above median subject to secondary review • Exceptions • Abstract and written comments to all members of scientific review panel

  9. Panel Review • In-person meeting for all members • Entire proposals provided upon request • Readers from merit review present overview • Discussion • Members provide written overall impact score • Outlier scores discussed • Funding recommendations determined

  10. Documentation • Reviewer A provides comments and scores • Reviewer B provides comments and scores • Reviewer A provides summary report • Additions/subtractions from leader review • Minority report • Submit scores and reports to NIJ program manager

  11. SCORING • 1.0 TO 1.5 • 1.6 TO 2.0 • 2.1 TO 2.5 • 2.6 TO 3.0 • 3.1 TO 4.0 • 4.1 TO 5.0 • Outstanding • Excellent • Good • Very good • Fair • Poor SCORE RANGE ADJECTIVAL EQUIVALENT

  12. Programmatic Review • NIJ substantive experts review scientific review panel recommendations • Provide written determination if substantive experts disagree with panel recommendation • Provide panel and substantive expert recommendations to NIJ office director • Provide office-level recommendations to NIJ director • Funding decision

  13. Timeline for Pilot Implementation JUNE NIJ Conference Announcement SEPTEMBER Stand-up 5 Review Panels MAY NIJ Review MAY NIJ Director Review/Funding Decision MARCH Proposals Submitted JUNE – AUGUST Review and Comment Period MARCH Proposals to Scientific Panel JUNE Awards Processed SEPTEMBER Incorporate Changes APRIL Scientific Review Panel Meets SEPTEMBER Awards Presented

  14. Evaluate Scientific Review Panel • Qualitative Process Review for Pilot • Review NIJ Receipt • Focus Group for Scientific Merit Review • Review Scientific Review Panel • Review Panel Recommendation • Focus Group for Scientific Review Panel • Review NIJ Expert Review • Review Office Director Briefing • Review NIJ Director Briefing

More Related