AAI Boston May 5, 2012

1. NIH Peer Review:Where are we and where are we going? Richard Nakamura, PhDActing Director, Center for Scientific Review, NIH AAI Boston May 5, 2012 National Institutes of HealthU.S. Department of Health and Human Services

2. National Institutes of Health • Much of the biomedical research in the United States is supported by the Federal Government, primarily the National Institutes of Health (NIH)

3. National Institutes of Health Office of the Director National Institute on Aging National Institute on Alcohol Abuse and Alcoholism National Institute of Allergy and Infectious Diseases National Institute of Arthritis and Musculoskeletal and Skin Diseases National Cancer Institute Eunice Kennedy Shriver National Institute of Child Health and Human Development National Institute on Deafness and Other Communication Disorders National Institute of Dental and Craniofacial Research National Institute of Diabetes and Digestive and Kidney Diseases National Institute on Drug Abuse National Institute of Environmental Health Sciences National Eye Institute National Institute of General Medical Sciences National Heart, Lung, and Blood Institute National Human Genome Research Institute National Institute of Mental Health National Institute of Neurological Disorders and Stroke National Institute of Nursing Research National Institute of Biomedical Imaging and Bioengineering National Center for Complementary and Alternative Medicine National Library of Medicine National Institute on Minority Health and Health Disparities John E. Fogarty International Center National Center for Advancing Translational Research Clinical Center Center for Information Technology Center for Scientific Review

4. NIH Extramural & Intramural FundingFY 2012 Enacted: $30.9 Billion • $3.4 B Intramural Research • $1.5 B Research Management & Support • $0.3 B Buildings and Facilities, Other Spending at NIH $5.2 B 17% • Supports over 300,000 Scientists & Research Personnel • Supports over 2,500 Institutions Spending Outside NIH$25.7 B 83%

5. FY 2011 Percent Distribution of Basic and Clinical Research

6. Grant Success RatesFY 1978-2013 ?

7. PI Applicants: PI Initiative/RFAs The NIH Peer Review Process Peer Review Applications  Study Sections  Ranking  Percentiling IC Strategic Goals/Awards/ Funding • Research • Outcome Progress (Publications/Citations) • Public Health

8. Your Application Goes to the NIH Center for Scientific Review (CSR) • Receives all NIH applications • Refers them to NIH Institutes/Centers and to scientific review groups • Reviews majority of grant applications for scientific merit Focal Point for Initial Review at NIH

9. CSR Peer Review – Fiscal Year 2011 • 85,000 applications received • 58,000 applications reviewed at CSR • 16,000 reviewers • 230 Scientific Review Officers • 1,465 review meetings

10. Enhancing Peer Review

11. 1. Improving Study Section Alignment Input from the community Internal IRG reviews Open Houses Advisory Council

12. General Map of Health Research Source: Compiled by Science-Metrix from Medline, NIH data, and Scopus

13. The Network of CSR Study Sections

14. 25 30 35 40 45 50 55 60 65 70 75 80 85 90 Age 2. Addressing Review and Funding for New Investigators Projection of Age Distribution of NIH RPG Investigators: 2020 7% PIs in 1980 6% 5% 4% Percent of PIs 3% 2% 1% 0%

15. 2. Addressing Review and Funding for New Investigators NIH decided to control the process at the level of funding rather than during review. The only difference in review is that all the applications from New Investigators are now clustered for discussion Different paylines for New Investigators and Early Stage Investigators.

16. 2. Funding New Investigators

17. 3. Advancing Additional Review Platforms Additional Review Platforms Help Recruit Reviewers Electronic Review Modes Reduce Travel Electronic Review Telephone Assisted Meetings Video Assisted Discussions Internet Assisted Meetings Telepresence Meetings

18. 3. Telepresence Study Sections

19. 4. Funding the Most Promising Research Earlier Months: Submission to Posting Critiques

20. Funding the Most Promising Research Earlier

22. 4. Funding the Most Promising Research Earlier

23. 5. Recruiting the Best ReviewersSome Successful Strategies • Move a meeting a year to the West Coast • Additional review platforms • Develop a national registry of volunteer reviewers • Searchable database with 5,000 reviewers • Provide tangible rewards for reviewers • No submission deadlines for chartered members of study sections (effective February 2008) • Provide flexible time for reviewers • Choice of 3 times/year for 4 years • or 2 times/year for 6 years

24. 6. Focusing More on Impact and Significance and Less on Approach • Shorten Applications • Scoring Significance • Discussed applications receive additional overall impact score • Training of Reviewers and Chairs

25. 7. Saving Reviewers Time • Shorter Applications • Bullet Critiques • Additional Review Platforms

26. 8. Scoring

27. 8. ScoringPriority Scores of R01 and R21 Reviewed by CSR June 2008 June 2009

28. 9. Continuously Reviewing the Changes • 12/09 Applicant and Reviewers Survey (64% response) • 1/10 Advisory Council Survey (291 responses) • 5/11 Planned Survey on Shorter Applications

29. New Investigators

30. Your Career Stage Is Considered • If you submit an R01 grant application • If you are a New Investigator or Early Stage Investigator • If NIH has correct info on your career stage

31. New and Early Stage Investigators • New Investigator (NI) • PD/PI who has not yet competed successfully for a substantial NIH research grant • Early Stage Investigator (ESI) • PD/PI who qualifies as a New Investigator AND is within 10 years of completing the terminal research degree or is within 10 years of completing medical residency (or equivalent) http://grants.nih.gov/grants/new_investigators/

32. Early Career Reviewer Program

33. Purpose of the Early Career Reviewer (ECR) Program • To train qualified scientists without significant prior review experience so that they become effective reviewers • To help emerging researchers advance their careers by exposing them to review experience • To enrich the existing pool of NIH reviewers by including scientists from less research-intensive institutions as well as those from research-intensive institutions.

34. Requirements for Being an ECR • Has not reviewed for NIH beyond one mail review • Demonstrates scientific qualifications, such as: • A faculty appointment or equivalent • An active independent research program and recent publications in good research journals • Does not have R01 NIH or equivalent funding

35. Apply to the ECR Program • Send a current CV and a list of terms that describe your scientific expertise to: CSREarlyCareerReviewer@mail.nih.gov • If eligible, your name will be placed into our ECR database • You will be invited to serve as an ECR when your expertise is needed for particular applications

36. The Future • Better distribution of applications across study sections • Predictive evolution of study sections • Better applicant tools for requesting study sections • Better SRO tools for finding reviewers • Increased diversity and reduced award disparities • Develop a science of peer review

37. This is CSR September 2009

38. Questions? Comments?CSRDirector@csr.nih.gov

39. What is the Difference Between Significance and Impact? • Significance addresses: • Does the project address an important problem or a critical barrier to progress in the field? • If the aims are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? • Impactaddresses: • Probability of whether the research will exert a sustained, powerful influence on the research field.

40. Determination of New Application Status Following Unsuccessful A1 A new application is expected to be substantially different in content and scope • substantial changes in all sections of the Research Plan, particularly in the Specific Aims and the Research Strategy sections • fundamental changes in the questions being asked and/or the outcomes examined • Changes to the Research Plan should produce a significant change in direction and approach for the research project

41. Determination of New Application Status: No No's…. • Rewording of the Title and Specific Aims does not constitute substantial changes in scope, direction or content. • Requests for review by a different review committee or funding consideration by a different NIH Institute are not sufficient reasons to consider an application as new. • Submission to a different FOAis also not sufficient to make an application new.

42. Statistics for 144 Study Sections

43. If group bibliometric differences reflect quality differences, can we determine the sources of the variance? What is the relative contribution of the quality of the reviewers and the quality of the applications?

44. NIH Guide For Grants and Contracts U.S. Department of Health and Human Services • Announces NIH Scientific Initiatives • Provides NIH Policy and Administrative Information • Supplies links to application forms • Available on the NIH Web Site: http://www.nih.gov

45. A Window to Your Application: eRA Commons eRA Commons is an online interface where a grant applicant can: • Check submitted grant application for errors and warnings and view final image • Track review assignment, view review outcomes (score, summary statements), find contact info • Update Personal Profile to ensure Early Stage Investigator eligibility is in place • Submit pre-award information (just in time) • View Notice of Award and other key documents And much more! https://commons.era.nih.gov/commons/

46. View the Videos • NIH Peer Review Revealed • NIH Tips for Applicants • What Happens to Your NIH Grant Application http://www.csr.nih.gov/video/video.asp

47. Recruiting the Best ReviewersAcademic Rank of ALL CSR Reviewers

48. Recruiting the Best Reviewers Academic Rank of CSR Standing Reviewers

49. CSR Mission Statement To see that NIH grant applications receive fair, independent, expert, and timely reviews – free from inappropriate influences – so NIH can fund the most promising research.

50. Longer Term Optimization:Structure of CSR