The Ins and outs of nih peer review

1. The Ins and outs of nih peer review Rebecca Wagenaar-Miller, PhD Scientific Review Officer National Institute of Dental and Craniofacial Research NIH

2. Context-My Background • PhD Vanderbilt University Department of Cancer Biology (2002) • Matrix Metalloproteinase-7 in Mouse Models of Intestinal Tumorigenesis • Vanderbilt University Postdoctoral Fellow (2002-2003) • MMP-7 in pre-clinical model of colorectal cancer • NIH, NIDCR Postdoctoral Fellow (2003-2007) • Type II transmembrane serine proteases • MMP-mediated and uPARAP-mediated collagen degradation in development • Scientific Review Officer (SRO), NIDCR, NIH (2007-present)

3. Goals • Give a brief look into the NIH peer review process • Update you on changes in NIH peer review • Discuss strategies for a successful application • Provide a few resources

4. Is this your view of the NIH?

5. How about this?

6. Or this?

7. Research grant Review process National Institutes of Health Research Grant Application School or Other Research Center Center for Scientific Review Assigns to IRG/Study Section & IC Study Section Initiates Research Idea Submits Application Evaluates for Scientific Merit Institute Evaluates for Program Relevance Advisory Councils and Boards Allocates Funds Conducts Research Recommends Action Institute Director Takes final action for NIH Director

8. The peer review process

9. THE MISSION OF PEER REVIEW To ensure that grant applications and contract proposals are evaluated by qualified scientific peers in a fair, objective, and timely manner.

10. The NIH peer review process • Conducted according to the Federal Advisory Committee Act • Meetings are closed to the public • All materials and discussions – strictly confidential • Over 80,000 applications reviewed per year • Almost 18,000 reviewers

11. NIH Application review • 80% of applications are reviewed by CSR in Scientific Review Groups (SRGs) that are part of Integrated Review Groups (IRGs) • The rest are reviewed by individual ICs • Requests for Applications (RFAs) • Program Announcements with IC Review (PARs) • Training Applications (F, K’s) • ‘Complex’ Applications (P, U’s, Clinical Trials) • Contract Proposals

12. Summary statements • The legal summary of the review meeting • Includes all three reviewer critiques, largely unedited • Includes a resume, which is a summary of the discussion of the meeting, focusing on the major strengths and weaknesses that resulted in the overall impact score • Used by council, applicants, program staff

13. grants and peer review life cycle Award and Post Award Pre-application Peer Review Council Program Scientific Review Officer Program and GM Peer Review Meeting Summary Statement Applications Arrive Council 2-3 months 2-6 weeks One month

14. The NIH Peer Review Process To Request a Scientific ReviewGroup • Cover letter of application • Application title • FOA# and title • Request: • Assignment to particular SRG or study section • Assignment to particular IC for funding consideration • Disciplines involved, if multidisciplinary • Explanation for late application Not all requests can be honored. SRG rosters are posted 30 days before the SRG meeting: http://era.nih.gov/roster/index.cfm http://www.csr.nih.gov/committees/rosterindex.asp

15. The NIH Peer Review Process Reviewer Assignments • For each application: • ≥ Three qualified reviewers are assigned (“2 + 1”) • Assignments are made by the SRO • Based on the scientific content of application • Expertise of the reviewer • Suggestions from the PI on types of expertise – • not names! • Suggestions from Program staff Assignments are confidential!

16. 9 Point rating scale Non-numeric score options:NR = Not Recommended for Further Consideration, DF = Deferred, AB = Abstention, CF = Conflict, NP = Not Present, ND=Not Discussed • Minor Weakness: An easily addressable weakness that does not substantially lessen impact • Moderate Weakness: A weakness that lessens impact • Major Weakness: A weakness that severely limits impact 1 3 2

17. The NIH Peer Review Process NIH Scoring System • Preliminary scores (before the meeting) • Entered by assigned reviewers and discussants in • secure website • Made available to other SRG members • Final overall impact/priority scores (at the SRG meeting) • Voted by private ballot • All eligible members vote Reviewers are instructed to revise their criterion scores after the meeting.

18. The NIH Peer Review Process Post-submission Materials • Applications submitted for Sept. 25th, 2010 and later: • Will only accept administrative materials resulting from unforeseen administrative issues, such as • Revised budget page(s) (e.g., due to new funding) • Biographical sketches (e.g., due to the loss of an investigator) • Letters of support or collaboration (e.g., due to the loss of an investigator) • Adjustment resulting from natural disaster (e.g., loss of animal colony) • Adjustments resulting from change of institution (e.g., PI moves to another university) • News of an article accepted for publication (do not sent article) • Special provisions for training grants and certain FOAs Notice OD-10-115

19. The NIH Peer Review Process SRG Meeting Procedures • Discussion format • Members with conflicts excused • Initial levels of enthusiasm stated(assigned reviewers and discussants) • Primary reviewer - explains project, strengths, weaknesses • Other assigned reviewers and discussants follow • Open discussion (full panel) • Levels of enthusiasm (assigned reviewers) restated • Individual SRG members vote • Other review considerations discussed (budget)

20. Peer review changes

21. New NIH Application Formats

22. Scoring and review criteria Impact Score Do not Impact Score

23. Overall impact/priority score • Research Project Grant (RPG): likelihood for the project to exert a sustained, powerful influence on the research field(s) involved, in consideration of the five core review criteria, and additional review criteria • Fellowship (F): likelihood that the fellowship will enhance the candidate’s potential for, and commitment to, a productive independent scientific research career in a health-related field, in consideration of the scored and additional review criteria • Career Development (K): likelihood for the candidate to maintain a strong research program, taking into consideration the criteria in determining the overall impact/priority score

24. New Investigators and EARLY STAGE INVESTIGATORS

25. New investigator • Background: • Concern about the number of New Investigators (NI) and the increasing age at first award • Goals: • Enrich the applicant pool with enough Early Stage Investigator (ESIs) to reduce the average age at first award • Accelerate the period of training leading to independence

26. Early Stage Investigators • NIH created a new ‘Early Stage Investigator’ (ESI) category designed to accelerate the early transition of new scientists to research independence by receiving their first R01 earlier. http://grants.nih.gov/grants/guide/notice-files/NOT-OD-08-121.html • A PD/PI who qualifies as a New Investigator is considered an Early Stage Investigator (ESI) if he/she is within 10 years of completing his/her terminal research degree or is within 10 years of completing medical residency (or the equivalent). 

27. New investigator/early stage investigator • New Investigator Policies are limited to applications for traditional research project grant (R01) support. • All New Investigators are encouraged to apply for NIH R01 awards. • During Peer Review: • NI/ESI applications will be clustered during review • Consider the career stage of the PD/PI when evaluating elements of the applications, such as the availability of preliminary data • NOT-OD-08-121 September 26, 2008

28. Implementation of ESI Definition • The NIH modified the collection of information on degree dates and medical residency within the personal profile of the eRA Commons. • PD/PIs must update their personal profile in the eRA Commons in order to be considered for the ESI classification.  Investigators who enter degree and residency completion dates will be notified of their ESI status by email. • A procedure and guidelines for requesting an extension of the period of ESI eligibility is in place to accommodate individuals with various medical concerns, disability, pressing family care responsibilities, or active duty military service (instructions in Commons).

29. NI AND ESI POLICY FY 2010 • Over the past three years, about 25 percent of all competing R01 awards have gone to New Investigators • Continue to equilibrate success rates for established and New Investigators submitting new R01 applications • Should permit the NIH to support 1,650 or more New Investigators • A majority of the New Investigators will be Early Stage Investigators http://grants.nih.gov/grants/guide/notice-files/NOT-OD-10-039.html NOT-OD-10-039,January 8, 2010

30. http://grants.nih.gov/grants/oer.htm

31. Tips for a successful grant application

32. Grant Writing for Success • Start early • Start with a good idea • Talk to your NIH Program Official(s) • Remember review criteria • Show you draft application to colleagues • who do not already know what you intend to do • who are not your best friend • who has/had the same type of grant • Follow instructions carefully

33. Foa SOURCE OF REVIEW GUIDELINES

34. Make YOUR strengths the focus of your application • Propose experiments that take advantage of • YOUR training • YOUR expertise • YOUR environment • Don't say that you're going to learn a sophisticated new technique and then apply it in later years • If you need a collaborator(s), get a strong one who publishes in peer-reviewed journals and who has a grant

35. Focus! • A major problem with applications from first time investigators is being overly ambitious. • Focus • Focus • Focus • Did I remember to say “Focus”? • Be certain every aim and experiment is clearly related to the overall goal of the application.

36. grants1.nih.gov/grants/guide/listserv.htm

37. Insider tips and tricks • Late submission policy for NIH study section service for any PI/PD of grant application (NOT-OD-08-027) • Standard receipt dates (1 or 2 weeks late) • “Late applications have been accepted for reasons such as…temporary or adhoc service on a NIH extramural peer review group” • “No NIH staff member whether in the Center for Scientific Review or any of the other Institutes/Centers has the authority to give permission in advance for a late application.” • Cover letter important • Identify study section • Identify areas of expertise needed • Can list conflicts that should not review application (but be prepared to justify) • Submit to R01 to take advantage of NI/ESI mandate • Submit to RFAs if possible… free review of application

38. Research Portfolio Online Reporting Tool (RePORT)http://report.nih.gov/index.aspx A Searchable database of federally supported biomedical research Access reports, data, analyses, expenditures, results of NIH supported research activities Identify, Analyze IC(s) research portfolios, funding patterns, funded investigators: Identify areas with many or few funded projects Identify NIH-funded investigators and their research Identify potential mentors/collaborators

39. Websites • NIH Guide: http://grants.nih.gov/grants/guide/ • NIH Office of Extramural Research http://grants.nih.gov/grants/oer.htm • NIAID “how to” website for developing a grant application: http://www.niaid.nih.gov/ncn/grants/default.htm • Writing a Grant: A Technical Checklist from NINDS: http://www.ninds.nih.gov/funding/grantsmanship_checklist.htm •  NIH RePorter: database of all funded grants including abstracts, funding amounts, study section, PI, and much more info: http://projectreporter.nih.gov/reporter.cfm • NIH Peer Review Policies and Practices http://grants.nih.gov/grants/peer/peer.htm • Changes to Peer Review (and applications) http://enhancing-peer-review.nih.gov/docs/application_changes.pdf

40. Thanks