The New NIH Review System: Reviewer’s perspective

1. The New NIH Review System: Reviewer’s perspective Liz Madigan, FPB School of Nursing

2. NIH information/guidance • Advantages/disadvantages as a reviewer • Recommendations for further improvement

3. Relationship of Old vs. New Scores

4. NIH Guidance • The NIH grant application scoring system uses a 9-point rating for the impact/priority score •  Assigned reviewers also provide ratings for each review criterion (Significance, Investigator(s), Innovation, Approach, Environment) using the same 9-point scale. • These criterion ratings are provided in the summary statement of all applications, both discussed and undiscussed. • Criterion ratings should be considered in determining the overall impact/priority score, but because the relative importance of each criterion to the overall impact/priority score differs for each application, reviewers should not apply a formula of unweighted or weighted criterion scores across applications.

5. NIH Guidance • Reviewers are strongly encouraged to utilize the full range of the rating scale in determining ratings. Optimally, scores will be normally distributed with very few 1’s and 9’s and a majority of scores in the middle of the range (4-6). • Discussed applications will receive impact/priority scores from all eligible (not in conflict) reviewers, and these scores will be averaged and multiplied by 10 to determine the final impact priority score (range of 10 to 90). • Because the relative importance of each individual criterion to the overall score differs for each application, reviewers should not use a formula of weighted or unweighted averages across applications to determine the overall impact/priority score.

6. Minor: An easily addressable weakness that does not substantially lessen the impact the project Moderate: A weakness that lessens the impact of the project Major: A weakness that is severely limits the impact of the project

9. NIH-provided Word Template • Specified template provided that reviewers were to use • “Limit text to ¼ page” for each of the criterion areas • Significance • Investigators • Innovation • Approach • Environment

10. Reviewer Advantages • Broader range of scores and more descriptions made it somewhat easier to rank applications within each criterion • Identifying strengths and weaknesses made the review more focused • Review was easier to write in some ways—bullet points of strengths and weaknesses

11. Reviewer Disadvantages • Impact/priority score was still difficult for some applications (e.g. very experienced team, well funded in the past, application extended the work somewhat but not in a very exciting way OR application from relatively new investigator that may move field forward but approach was not precise or defined) • Worries about the scope and extent of comments being given with the suggested limitations in the review length

12. Reviewer Recommendations • Need to indicate for the applicants which weaknesses are minor, moderate and major so the applicants can revise accordingly—we were not forced to do this in the review so I worry that the applicants are getting unprioritized comments • Recommendations for ¼ page of text is difficult and may not result in higher quality reviews, applications and revisions