Reviewer Orientation: Enhanced Peer Review Guidelines

1. REVIEWER ORIENTATION TO ENHANCED PEER REVIEWApril 2009 1

2. Changes to ReviewBeginning with May/June 2009 Meetings • Enhanced Review Criteria for certain mechanisms • Templates for Structured Critiques • Scoring of Individual Review Criteria • All applications will receive criterion scores from assigned reviewers • New 1 to 9 Scoring Scale 2

3. Goals of the Changes • Clearer understanding of the basis of application ratings • More emphasis on impact and less emphasis on technical details • Succinct, well-focused critiques that evaluate, rather than describe, applications • Routine use of the entire rating scale 3

4. Before the Review Meeting When reading applications the assigned reviewers should: • Address all applicable criteria and other review considerations • Identify major strengths and weaknesses • Assign scores to each of the 5 “core” criteria • Assign an overall impact/priority score 4

5. Preparation of Critiques When writing your critiques: • Use bulleted points to make succinct, focused comments • Short narratives may occasionally be appropriate, but should be rare • Focus on major strengths and weaknesses (ones that impacted your overall rating of the application) 5

6. Features of Critique Templates • Boxes for evaluating: • Each core review criterion • Other applicable review criteria and considerations • Overall impact of the application • A box for “advice to applicants” • Hyperlinks to web pages providing descriptions of review criteria and additional review considerations 6

7. Excerpt from a Critique Template: Criterion • List major strengths and weaknesses that influenced the overall impact/priority score • Limit text to ¼ page per criterion, although more text may occasionally be needed • Do not enter scores on critiques

8. Excerpt from a Critique Template: Protected Form Fields and Drop-downs • Protected elements (Drop-down boxes and form fields) are shaded gray • Part of each template is a PROTECTED form • Reviewers should NOT unprotect the forms!

9. Scoring Individual Review Criteria • There are 5 “core” criteria for most types of grant applications • For example, the core criteria for Ks are: • Candidate • Career Development Plan/Career Goals & Objectives/Plan to Provide Mentoring • Research Plan • Mentor(s), Consultants(s), Collaborator(s). • Environment and Institutional Commitment to the Candidate. • Use the 9-point scale (1 = exceptional, 9 = poor)for the five “core” review criteria. • Do not enter scores in the critique 9

10. Overall Impact/Priority Scores • Consider criterion strengths and weaknesses of each application in determining an overall impact/priority score • Recognize this is a NEW scoring system and focus on the guidelines for its use • This new scoring system is intended to reflect the “real-world” range of the quality of applications typically seen in actual study sections • It is ESSENTIAL that reviewers take advantage of this unique opportunity to use the entire 1 to 9 range 10

11. Scoring Descriptions 11

12. Scoring Descriptions

13. Before Attending the Review Meeting • Post critiques to the Internet Assisted Review (IAR) Web module • Enter criterion scores and overall/priority score in IAR • Do not enter scores as part of the critique! • Ensures better data integrity • Allows scores to be placed where needed • i.e. Summary Statements, Commons Status • Makes scores available for future analysis 13

14. IAR: New Drop Down for Five Core Criteria • Reviewers will see new drop-down menus in IAR for entering scores for each criterion New drop-down Candidate Career Development Research Plan Mentor(s)

15. IAR: Assigned reviewers must submit a critique to upload scores • Reviewers must close the critique file before submitting

16. IAR: Entering Scores and Critiques • Assigned reviewers may not submit Criterion or Preliminary Scores without a critique • If a reviewer tries to save the criterion and/or preliminary score without uploading the critique, an error message will occur • The maximum file size for a critique is 1 MB

17. IAR: New Header Information in Critique • Preliminary IAR Critique now includes criterion scores

18. IAR: Updating Criterion Scores • Criterion scores can be updated in IAR during the submit phase, edit phase and the final scoring phase • If criterion scores are edited, the PDF of the critique file is regenerated each time because the critique has header information with the criterion scores • If the criterion scores change, the PDF critique changes

19. IAR: New Popup Listing Criterion Scores • New link on List of Applications screen will display criterion scores for each application View All Scores

20. At the Review Meeting:Procedure for Discussed Applications • Assigned reviewers will discuss strengths and weaknesses of each application • Recommend overall impact/priority score • Criterion scores will not be discussed by the committee • All eligible members will record an overall impact/priority score (as is presently true) 20

21. IAR: Edit Criterion Scores on Voter Sheet • Criterion scores can easily be edited by using the voter sheet

22. After the Review Meeting: Updating Scores or Critiques • Assigned reviewers whose opinions changed as a result of discussion at the meeting should use IAR: • To modify their criterion scores • To post revised critiques • If criterion scores are edited, the PDF of the critique file is regenerated 22

23. Summary Statements • Overall impact/priority scores of discussed applications will be the average of scores voted by all eligible reviewers, multiplied by 10 • Final scores will range from 10-90, in whole numbers • Summary statements for ALL applications will include the criterion scores and critiques posted by assigned reviewers 23

24. For additional information: Enhancing Peer Review at NIH Web Site http://enhancing-peer-review.nih.gov Thank you for your review service 24