NIH Peer Review

1. NIH Peer Review Pedro Delgado, MD

2. Funded NIH Grant Awards Searchable Database: RePORT http://projectreporter.nih.gov/reporter.cfm

3. Funding Opportunity Announcements (FOA) Available Grant Programs http://grants.nih.gov/grants/guide/

4. NIH eRA Commons Electronic Connection – “For the purposes of exchanging research grants administration information”…this is how the NIH communicates with applicants and grantees [Obtain password help from the Office of Sponsored Programs (OSP), UAMS]

5. “R01’s and K08’s and T32’s, Oh My!”So What Do All Those Letters Mean? • NIH uses a variety of funding mechanisms, including: grants, cooperative agreements & contracts. • Grants & Cooperative Agreements are classified by “Activity Codes” that indicate the kind of activity that will be conducted under the project: • The R-series indicates Research Project Grants, e.g., R01, R03, R21. • The P-series indicates Program Project Grants, e.g., P01, P30, P50. • The T-series indicates institutional NRSA training grants. • The K-series indicates career development awards. • The F-series indicates individual NRSA fellowships. The IMPAC Manual contains complete explanations of all activity codes and other arcane NIH classification codes. It may be found at: http://grants1.nih.gov/grants/funding/funding.htm

6. Your Grant ? NIH or Other Funding Agency “Pink Sheet” months later.. Critique and Funding Decision

7. President Cabinet Departments HHS Office of the Secretary FDA CDC NIH OD NHGRI NIDCR NIGMS NCCIH NIAID NIAMS NINR NINDS NHLBI NCI NIA NIDDK NIBIB NCMHHD NCATS CSR Annual $$$ from Congress $

8. National Institutes of Health Office of the Director National Institute of Allergy and Infectious Diseases National Institute on Aging National Institute on Alcohol Abuse and Alcoholism National Institute of Arthritis and Musculoskeletal and Skin Diseases National Cancer Institute National Institute of Child Health and Human Development National Institute on Deafness and Other Communication Disorders National Institute of Dental and Craniofacial Research National Institute of Diabetes and Digestive and Kidney Diseases National Institute on Drug Abuse National Institute of Environmental Health Sciences National Eye Institute National Institute of General Medical Sciences National Heart, Lung, and Blood Institute National Human Genome Research Institute National Institute of Mental Health National Institute of Neurological Disorders and Stroke National Institute of Nursing Research National Institute of Biomedical Imaging and Bioengineering National Center for Complementary and Integrative Health National Library of Medicine National Center on Minority Health and Health Disparities National Center for Research Resources Fogarty International Center Clinical Center Center for Information Technology Center for Scientific Review Study Sections

9. Understanding the NIH Grants Process • NIH receives over 28,000 competing grant applications per year. • The Center for Scientific Review (CSR) receives almost 10,000 applications per grant cycle. • NIH funded about 17,000 applications in FY01. • The success rate for competing R01 applications hovers around 30%. CSR’s mailroom on deadline day.

10. Institute1 $ Institute2 $ Council Council Your Grant & ~30,000 others NIH CSR Institute Assignment Reviewer 1 Provide Recommendation (numerical) Study Section Assignment Reviewer 2 Reviewer 3

11. Relationship of Study Sections (Scientific Review) to Institutes ($$) ~200 Study Sections -- CSR ~ 20 Institutes NCI NEI NIDDK NIGMS NHBL Etc. Biochemistry Clinical Oncology Genetics Epidemiology of Cancer Lung Injury, Repair Microbial Physiology Oncological Sciences Etc. • Scientific Review is Done by Study Sections • Funding Decisions made by Institutes

12. The GUTS of your APPLICATION!

13. What will reviewers look for in your grant? Core Review Criteria (in instructions to reviewers on NIH website) Significance Investigators Innovation Approach Environment

14. Innovation • Innovation. Does the application challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions? Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field of research or novel in a broad sense? Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions proposed?

15. SCIENTIFIC PREMISE

16. SCIENTIFIC PREMISE, CONT’D

17. UPDATED APPLICATION INSTRUCTIONS • SIGNIFICANCE (SCIENTIFIC PREMISE): • Describe the scientific premise for the proposed project, including consideration of the strengths and weaknesses of published research or preliminary data crucial to the support of your application. • APPROACH (RIGOR & TRANSPARENCY): • Describe the experimental design and methods proposed and how they will achieve robust and unbiased results. • Explain how relevant biological variables, such as sex, are factored into research designs and analyses for studies in vertebrate animals and humans. For example, strong justification from the scientific literature, preliminary data, or other relevant considerations, must be provided for applications proposing to study only one sex.

18. Key Resource Authentication! • What is considered a key biological or chemical resource? • Generally, these are resources that: • May differ from laboratory to laboratory, over time • Have qualities or qualifications that can influence results • Are integral to the proposed research • Examples: cell lines, specialty chemicals, antibodies, other biologics

19. Reviewers Written Review = Critique • major strengths/weaknesses • issues • concerns • primary basis for the recommended Impact Scoreand Summary Statement • usually must be specifically addressed if application revised/resubmitted

20. Peer Review Is Critical!! Feasibility is Critical!!

21. Scored Components K Grants R Grants • Overall Impact • Candidate • Career Development Plan • Research Plan • Mentor (s) • Environment & Commitment to Candidate • Overall Impact • Significance • Investigator(s) • Innovation • Approach • Environment

22. Scores also Provided for Each Component • 1 exceptional • 2 outstanding • 3 excellent • 4 very good • 5 good • 6 satisfactory • 7 fair • 8 marginal • 9 poor

25. Impact/Priority Scores A numerical rating that reflects the scientific merit of the proposed research relative to the "state of the science”.

26. NIH Study Section Scores • 1 exceptional – exceptionallystrong with no weaknesses • 2 outstanding – extremelystrong with negligible weaknesses • 3 excellent – very strong with only some minor weaknesses • 4 very good – strongbut with numerous minor weaknesses • 5 good – strongbut with at least one moderate weakness • 6 satisfactory – some strengths but also some moderate weaknesses • 7 fair – some strengths but with at least one major weakness • 8 marginal – a few strengths and a few major weaknesses • 9 poor – very few strengths and numerous major weaknesses

28. How does the Study Section Operate? • SS has 15-25 members (regular & ad hoc) selected for their expertise in the area of the applications the study section reviews and is overseen by a Scientific Review Officer • Scientific Review Officers (SRO) oversee individual SS’s; receive applications from CSR main office and assign each application to primary & secondary reviewers before the review meeting • Primary & Secondary Reviewers prepare written reviews and post in advance on NIH website • All SS members meet to review all grants • Chair runs meeting (not NIH employee) • SRO serves as a ‘resource’ about policy & protocol • Study Sections are far more diverse than one might anticipate

29. The primary and secondary reviewers will: • read your proposal thoroughly (you hope!) and understand it (you really hope!!) • evaluate it (priority score), • explain it AND justify their evaluation to the study section, • prepare a written summary and evaluation. They present to the full study section (~15-20 people) most of whom will not have read the complete grant. After discussion, every member gets an equal vote. 50-100 grants in 2 days – You must be clear and to the point!!

31. Reviewers will see new drop-down menus in IAR for entering scores for individual criteria New drop-down

32. Edit Criterion Scores on Voter Sheet

33. View All Scores Link to Pop-up to View All Scores

34. For Standing Study Sections, Priority Scores Are OftenPercentile Ranked

35. Percentiles Represents the relative position or rank of each priority score (along a 100.0 percentile band) among the scores assigned by a particular study section.

36. Priority Scores and Percentile Rank • Priority Scores assigned by Study Sections based on average of all reviewers scores • Percentile Rank assigned afterwards to normalize scoring across study sections and over time • Funding determined by percentile rank

37. Streamlined Review • Prior to the meeting, reviewers categorize applications as either in the upper half or lower half in quality • Applications in the upper half = fully discussed and scored at the study section meeting • All other applications are not discussed nor scored

38. 1) Study Section Review 2) Institute Funding Decision Approved Applications Receive Percentile Scores 1st John Doe (Biochem. SS) 2nd Your Grant (Oncology SS) 13th Stancel (Genetics SS) 27th Jane Doe (Micro. SS) Etc. Etc. “Streamlined” Not Recommended for Further Consideration National Cancer Institute (Budget from Congress) 1) Your Grant 2) John Doe 3) Stancel 4) Jane Doe Etc. Etc. Etc. $ $ $ Payline - - -

39. The Institute’s Council may make some changes in the rankings based upon their particular goals and needs. It’s Great Science, BUT will it really help us find a cure for the plague?

40. Grant Contacts, Interactions, and Information in Various Phases • Pre Submission: Institute Program Officials • Assess Institute’s Interest in Your Proposal • Provide Advice & Potential SS Assignment • Submission – Review : CSR’s Scientific Review Officers (SRO’s) who oversee study sections • Questions about application • Submitting additional information • Post Award : Assigned Program Officer at Funding Institute • NEVER CONTACT A STUDY SECTION MEMBER ABOUT YOUR GRANT!!!

41. Help your proposal be assigned appropriately – You can Recommend Assignments -Call Institute staff (be respectful) – they want to help -Study section rosters are public information (‘regular’ members) Remember – The agency does not care about supporting you as an individual. They only want to fund you if the research you propose helps them achieve one of their objectives.

42. (1) Get peer review from critical colleagues: EARLY and OFTEN!! (2) Write different sections of the grant for ‘different’ reviewers (3) Grant must be crystal clear and explicit! Page limits make this critically important. (How long should one plan to write?)

43. Important Sources of Information • NIH Website – • general information about types of Grants • ‘standard’ information, e.g., general instructions, receipt dates, etc. • NIH Guide – weeklyinformation about program changes, Request for Applications (RFA’s) • Institute Home Pages • Information about areas of interest • Names and Contact information for Program Officers • Center for Scientific Review Website • Detailed information about preparing applications • Names and Contact information for Scientific Review Officers • Study Section Descriptions and Rosters • Instructions for Reviewers!!!

44. USE THE FOLLOWING TOOLS – IN ORDER – TO PREPARE YOUR GRANT • The Internet to identify • Study section rosters • Institute contact personnel • The Institute’s goals and interests • The telephone to • Assess interest in your project • Consider study section/institute assignments • The word processor to -Actually write the grant!

45. HELP THE REVIEWERS – When you Write the Grant! • Read and follow your instructions and know the reviewers’ • Give the proposal a focus, and have colleagues help you evaluate it (write a novel, not a collection of short stories) 3) Build the application around your focus (the hypothesis or scientific question being addressed) • Help the reviewer prepare his/her critique and presentation to the study section (know what he/she is looking for and write it for them - CLEARLY!) • Get rigorous, critical review (before you send the grant!) • from “experts” • from well rounded “generalists”

46. And Don’t Ever Forget….. Talk to People at the Funding Agency Get peer review from critical colleagues: Do It EARLY and OFTEN!! Never Overestimate the Reviewers