The NIH Peer Review Process

1. The NIH Peer Review Process Sally A. Amero, Ph.D. Alan L. Willard, Ph.D. NIH Review Policy Officer Deputy Director Office of Extramural Research NINDS Extramural Program 2011 NIH Regional Seminars

2. The NIH Peer Review Process NIH Peer Review Cornerstone of the NIH extramural mission Standard of excellence worldwide Partnership between NIH and the scientific community Per year: ~ 80,000 applications ~ 18,000 reviewers

3. The NIH Peer Review Process NIH Peer Review: Our topics today Core values Initial peer review process Advisory Council process

4. The NIH Peer Review Process Core Values: Freedom from bias Confidentiality Balanced representation Expertise Informed recommendations Transparency

5. The NIH Peer Review Process Core Value: Freedom from bias • Conflict of Interest (COI) • Financial - Professional • Employment - SRG membership • Personal - Other interests • Depending on nature of COI, individual with a COI • must be excluded from serving on the SRG, or • must be recused from discussion and scoring of application. • Each SRG member must sign two COI certifications.

6. The NIH Peer Review Process Core Value: Confidentiality • All confidential materials, discussions, documents • are deleted, retrieved, or destroyed. • Reviewers are sent guidance with applications. • Application information is provided on • secure websites or protected portable devices. • All questions must be referred to the SRO. • Do not contact reviewers directly!

7. The NIH Peer Review Process Core Value: Balanced Representation Recruitment of SRG members must include representation of diverse individual backgrounds. Both genders Variety of racial/ethic groups Variety of geographic areas Seniority Managed by the Scientific Review Officer

8. The NIH Peer Review Process • Scientific Review Officer • Ensures compliance with applicable • laws, regulations, and policies • Identifies and recruits reviewers • Assigns reviewers to individual applications • Manages conflicts of interest • Arranges and presides at review meetings • Prepares summary statements – official written • outcome of initial peer review

9. The NIH Peer Review Process Core Value: Expertise Impact and significance are identified and given appropriate consideration. The approach and feasibility of each research project and qualifications of the investigators are judged competently.

10. The NIH Peer Review Process • Recruiting Peer Reviewers • Expertise • Stature in field • Mature judgment • Impartiality • Ability to work well in a group • Managed conflicts of interest • Balanced representation • Availability

11. The NIH Peer Review Process Core Value: Informed Recommendations The final evaluation and scoring is performed by SRG members participating in the discussion of the application. Consensus of all SRG members is required for an application to be designated “Not Discussed”.

12. The NIH Peer Review Process Types of Reviewers • Regular reviewers • Participate in the discussion and committee deliberations • Contribute preliminary impact scores, criterion scores, • written critiques, final impact scores • “Mail” reviewers • Contribute preliminary impact scores, criterion scores, • written critiques • Cannot submit final impact scores

13. The NIH Peer Review Process Core Value: Transparency/Review Criteria Must follow established criteria to ensure that the process is transparent, impartial, credible, and fair. Review criteria must be published in the Funding Opportunity Announcement (FOA).

14. The NIH Peer Review Process Review Criteria: Overall Impact • Overall consideration for all applications • Defined differently for different types of applications • Research grant applications: Likelihood for the project to • exert a sustained, powerful influence on the research field(s) • involved • See “Review Criteria at a Glance” • (http://grants.nih.gov/grants/peer/reviewer_guidelines.htm)

15. The NIH Peer Review Process Review Criteria: Overall Impact • Impact is assessed in consideration of: • Scored review criteria • Additional review criteria • Reviewers also are asked to comment on • additional review considerations.

16. The NIH Peer Review Process Scored Review Criteria • Receive individual, numerical scores from the • assigned reviewers. • Criterion scores are reported on the summary • statement. • For research grant applications: • Significance - Approach • Investigator(s) - Environment • Innovation

17. The NIH Peer Review Process Additional Review Criteria • Are considered in determining the impact score, • as applicable for the project proposed • Are not given individual scores. • For research grant applications: • Protections for Human Subjects • Inclusion of Women, Minorities, and Children • Vertebrate Animals • Resubmission, Renewal, and Revision Applications • Biohazards

18. The NIH Peer Review Process Additional Review Considerations • Are not considered in determining impact score. • Are not given individual scores. • For research grant applications: • Applications from Foreign Organizations • Select Agent Research • Resource Sharing Plans • Budget and Period of Support

19. The NIH Peer Review Process Initial Peer Review Process Overview of grants process Post-submission materials Requesting a particular SRG Scientific Review Groups NIH scoring system Meeting procedures Summary statements After the review

20. The NIH Peer Review Process Application received  NIH Center for Scientific Review (CSR) Assignments made   Initial peer review Funding considerations SRG; study section Institutes or Centers (ICs) IC or CSR Duals possible Scientific Review Officer Program Officer   Second level of review  Funding decisions Council or Board (IC) IC Director  Award!

21. CSR Review Most R01’s, F’s and SBIR’s Some Program Announcements Some Requests for Applications (RFAs) Institute/Center Review IC-specific features P’s, T’s, K’s Most RFAs The NIH Peer Review Process Assignments for Initial Peer Review • The locus of review (CSR/IC) is usually stated in the FOA. • SRG assignment is available in the PD/PI’s Commons account.

22. The NIH Peer Review Process Post-submission Materials • Only accepting administrative materials resulting from • unanticipated events, such as • Revised budget page(s) (e.g., due to new funding) • Biographical sketches (e.g., due to loss of investigator) • Letters of support or collaboration (e.g., due to loss of • investigator) • News of an article accepted for publication • Special provisions for training grants and certain FOAs • See NOT-OD-10-115: http://grants.nih.gov/grants/guide/notice-files/NOT-OD-10-115.html

23. The NIH Peer Review Process Post-submission Materials • Must be received by 30 days before the SRG meeting • Must show concurrence from the Authorized • Organization Representative • Cannot fix an incomplete application • See FAQs: http://grants.nih.gov/grants/funding/post_submission_faq.htm

24. The NIH Peer Review Process Scientific Review Groups (SRGs) Make recommendations on: Scientific and technical merit/impact Impact scores Criterion scores Written critiques Other review considerations

25. The NIH Peer Review Process Requesting a Particular SRG • Cover letter of application • Application title • FOA # and title • Request: • Particular SRG or study section • Particular IC for funding consideration • Disciplines involved, if multidisciplinary • Not all requests can be honored

26. The NIH Peer Review Process Types of Scientific Review Groups (SRGs) • “Chartered” SRGs • Multiyear terms • Formal appointment process • May include temporary members for special expertise • Special Emphasis Panels (SEP) • Ad hoc membership • Often meet only once

27. The NIH Peer Review Process Requesting a Particular SRG • SRG rosters are available on NIH websites • http://era.nih.gov/roster/index.cfm • http://www.csr.nih.gov/committees/rosterindex.asp • Permanent membership is available anytime • Membership for a given meeting is posted 30 days before • the SRG meeting • Subject to change • Some CSR rosters are posted in aggregate

28. The NIH Peer Review Process Reviewer Assignments • For each application: • ≥Three qualified reviewers are assigned • Assignments are made by the SRO • Expertise of the reviewer • Suggestions from the PI on expertise – not names! • Suggestions from Program staff and SRG members • Managing conflicts of interest • Balancing workload • Assignments are confidential

29. The NIH Peer Review Process Pre-Meeting SRG Procedures • Reviewers • Examine assignments • Submit Conflict of Interest and Confidentiality certification • Read applications, prepare written critiques in templates • Enter preliminary scores into secure website • Read and consider other critiques and preliminary scores

30. The NIH Peer Review Process Templates for Reviewers Links to definitions of review criteria

31. The NIH Peer Review Process SRG Meeting Agenda • Discuss applications one at a time • Cluster where feasible: • New Investigator (NI) applications • Clinical applications • Chairperson summarizes main points • Score each application after its discussion • Discuss other considerations • Budget • Resource Sharing Plans

32. The NIH Peer Review Process SRG Meeting Procedures • Discussion format • Members with conflicts excused • Initial levels of enthusiasm stated (assigned reviewers) • Primary reviewer - explains project, strengths, weaknesses • Other assigned reviewers and discussants follow • Open discussion (full panel) • Levels of enthusiasm (assigned reviewers) re-stated • All SRG members vote • Other review considerations discussed (budget)

33. The NIH Peer Review Process NIH Scoring System • Numerical scores • 1 (exceptional) to 9 (poor) • Integers • Final impact scores • Voted by all eligible (w/o COI) SRG members • Voted by private ballot at the meeting • Individual criterion scores • Given by assigned reviewers as part of their critiques • Generally not discussed at the meeting

34. The NIH Peer Review Process Phases of Process Score Descriptors

35. The NIH Peer Review Process Streamlining • Allows discussion of more meritorious applications • Less meritorious applications are tabled at the SRG meeting. • Requires full concurrence of the entire SRG • Summary statement: • Designated Not Discussed (ND) • Reviewer critiques and criterion scores • No numerical, overall impact score • Initial streamlining candidates are based on preliminary • impact scores.

36. The NIH Peer Review Process After the Review • eRA Commons (http://era.nih.gov/commons/index.cfm) • Final Impact Score is available in 3 days. • Summary statement is available in 4 – 8 weeks. • Available to: • PD/PIs • NIH officials • Advisory Council members • NIH Program Officer = Point of Contact

37. The NIH Peer Review Process Summary Statement • First page • NIH Program Officer (upper left corner) • Final Impact Score or other designation • Percentile (if applicable) • Codes • Human subjects • Vertebrate animals • Inclusion plans • Budget request • A favorable score does not guarantee funding!

38. The NIH Peer Review Process Summary Statement - continued • Subsequent Pages • Description (provided by applicant) • Resumé and Summary of Discussion (if discussed) • Reviewer critiques – essentially unedited • Administrative Notes • Meeting roster

39. The NIH Peer Review Process After the Review • If the outcome is favorable, congratulations! • If the outcome is unfavorable, consider your options: • Revise and resubmit application • Appeal the review outcome • Procedural deficiencies • Factual errors • Differences of scientific opinion cannot be appealed

40. The NIH Peer Review Process National Advisory Councils • Broad and Diverse membership • Scientists • Clinicians • “Public” members • Nominated by Institutes; Approved by HHS • Awards cannot be made without Council approval • Council procedures vary across IC’s

41. The NIH Peer Review Process National Advisory Councils • Advise IC Director about • Research Priority Areas • Diverse Policy Issues • Concept Clearance for future initiatives • Funding Priorities • Approve applications for funding • Expedited awards • En bloc concurrence

42. The NIH Peer Review Process Advisory Councils and Appeals • Unresolved appeals are presented to Council • Council options: • Support the SRG review • Support the appeal, recommend a re-review • Application could be deferred for next round • Application cannot be modified or updated • Results of a re-review cannot be appealed further • Council cannot overturn the SRG review or • impact score

43. The NIH Peer Review Process Additional Information • Enhancing Peer Review Initiative • http://enhancing-peer-review.nih.gov/ • Office of Extramural Research Peer Review Process • http://grants.nih.gov/grants/peer_review_process.htm • Peer Review Policies & Practices • http://grants.nih.gov/grants/peer/peer.htm • Center for Scientific Review • http://cms.csr.nih.gov/AboutCSR/Welcome+to+CSR/

44. The NIH Peer Review Process Contact Information Sally Amero, Ph.D. NIH Review Policy Officer Extramural Research Integrity Liaison Officer Office of Extramural Programs Office of Extramural Research National Institutes of Health ameros@od.nih.gov

45. The NIH Peer Review Process Contact Information Alan L. Willard, Ph.D. Deputy Director Division of Extramural Research NINDS National Institutes of Health alanw@ninds.nih.gov