J. Scott Osborne, III, Ph.D., M.P.H. Scientific Review Administrator

1. The Review of Epidemiologic Research Applications at the National Institutes of Health: Process, Products, and Best Possible ScienceInvestigator Viewpoint, from Idea to Submission, andStudy Section Viewpoint, from Submission to AssessmentPresented at the 44th Annual Conference on Cardiovascular Disease Epidemiology and Prevention of the American Heart Association, March 3, 2004, San Francisco, CA J. Scott Osborne, III, Ph.D., M.P.H. Scientific Review Administrator Epidemiology of Chronic Diseases Study Section Center for Scientific Review, National Institutes of Health Department of Health and Human Services

2. PROCESS OF REVIEW • From Idea to Application, Submission, and Assessment • PRODUCTS OF REVIEW • Assessment of Scientific Merit and Summary Statement • BEST POSSIBLE SCIENCE

3. PROCESS OF REVIEW • IDEA proposed in Application (PHS 398) • SUBMISSION of APPLICATION to NIH Division of Receipt and Referral, Center for Scientific Review • ASSIGNMENT to Institute(s) and Study Section (suggestions welcome) • REVIEW by NIH STUDY SECTION • ASSESSMENT OF SCIENTIFIC MERIT (score and percentile rank) for Institute Advisory Councils and Center Advisory Boards

4. PRODUCTS OF REVIEW • PRE-REVIW: APPLICATION – development of an idea into a meaningful project • REVIEW: RECOMMENDATION (score; percentile rank) regarding ASSESSMENT of Scientific and Technical Merit of Application • POST-REVIEW: SUMMARY STATEMENT

5. BEST POSSIBLE SCIENCE:Contributions of the Process of Review • PRODUCTS OF REVIEW LEAD TO BEST POSSIBLE SCIENCE. • ASSESSMENTS regarding Scientific and Technical Merit guide decision making. • SUMMARY STATEMENTS provide written feedback, allowing improvement of projects and meaningful revisions of applications for resubmissions. • NIH STAFF provide guidance.

6. BEST POSSIBLE SCIENCE:QUALITIES • Moves field forward. • Has clinical and pubic health significance. • May provide clues in areas outside specific framework of study.

7. BEST POSSIBLE SCIENCE:“The Impossible Dream” • Perfectly Justified Aims • Perfect Clarity of Exposures, Confounders, Effect Modifiers, and Outcomes • Perfect Design for a Perfect Sample • Perfect Methods for Ascertainment and Measurement, Perfect Validity and Reliability of Data • Perfect Analysis • Perfect Interpretation and Response

8. BEST POSSIBLE SCIENCE • CONVENIENCE vs. QUALITY – Select the best (e.g., valid, generalizable, etc.) possible sample and approach to collecting data; do not opt for convenience. • The “Ideal” or “Gold Standard” – Fully explore strengths and limitations of an approach, and consider a selected approach in comparison to alternatives. • “Best Approximation” – Propose a study well justified in the context of the current state of the field.

9. 3 “TIERS” OFEPIDEMIOLOGIC RESEARCH • DESCRIPTION • ETIOLOGY • INTERVENTION (including clinical trial epidemiology as well as primary, secondary, and tertiary prevention)

10. INVESTIGATORVIEWPOINT From IDEA to SUBMISSION

11. From Idea to Application: IDEA • IDEA: Significance -- “So what”? • IDEA: Appropriately focused or too broad? • IDEA: Innovative, standard, or already in progress or done? (Ask colleagues, and do a thorough search of the literature. Check CRISP at the NIH Website for funded NIH research.) • IDEA: Evidence to support in literature (animal and human studies)? Need pilot data to support? • IDEA: Plausibility; Mechanism

12. INNOVATION • New Question, New Approach • New Question, Old Approach • Old Question, New Approach

13. From Idea to Application: IDEA • Is the NIH interested in this area? (Check for a Program Announcement or Request for Applications. Call NIH Program Staff at different relevant Institutes to discuss your idea and how it fits with their programs.) • SELECT APPROPRIATE MECHANISM

14. From Idea to Application: SUPPORT STANDARD NIH MECHANISMS OF GRANT SUPPORT REVIEWED IN ECD • FELLOWSHIPS • SMALL GRANTS (R03) • R21 • R01

15. From Idea to Application: INVESTIGATIVE TEAM • Investigative Team: Convenience vs. Quality. NETWORK!!! • The Principal Investigator should develop an appropriate team. This may include one or more Co-Principal Investigators as well as Co-Investigators, Collaborators, and Consultants. Persons on the investigative team are often from different institutions.

16. From Idea to Application: INVESTIGATIVE TEAM • INTELLECTUAL PROPERTY: An application is submitted by an “applicant institution” (e.g., your university) but is the “intellectual property” of the investigators. • CONSULTANT ROLES vary according to the extent of involvement, but they are generally viewed as a service and do not involve a proprietary intellectual contribution to a project. • SERVICES GENERALLY AVAILABLE (e.g., assays) are not viewed as involving a proprietary intellectual contribution to a project.

17. From Idea to Application:ENVIRONMENT • General Institutional Support • Specific Resources Available for Project • Evidence of Departmental Level Support • Potential for Success in Overall Environment

18. From Idea to Application:METHODS • DESIGN • SAMPLE • REPRESENTATIVENESS AND GENERALIZABILITY (External Validity) • POWER (and justification for Effect Size) • DATA COLLECTION • ANALYSIS OF DATA (linked to Specific Aims) • POTENTIAL BIASES AND LIMITATIONS

19. From Idea to Application:Critical Thinking • Be your own worst critic. (About Everything!) • “So what”? (Significance) • Would you believe findings from this study? (Methods)

20. From Idea to Application:PHS 398 APPLICATION FORM RESEARCH PLAN • SPECIFIC AIMS • BACKGROUND AND SIGNIFICANCE • PRELIMINARY STUDIES or PROGRESS REPORT • RESEARCH DESIGN AND METHODS • HUMAN SUBJECTS

21. From Idea to Application:SUBMISSION • RECEIPT DATES • REQUEST STUDY SECTION ASSIGNMENT

22. STUDY SECTION VIEWPOINT From SUBMISSION to ASSESSMENT OF SCIENTIFIC MERIT

23. NIH SUPPORT • CONTRACTS (Solicited with Request for Proposals -- RFP) • COOPERATIVE AGREEMENTS • SOLICITED SET-ASIDES: RFA (Request for Applications; 1 submission, 1 review) • Program Announcements (PA) • UNSOLICITED APPLICATIONS

24. Epidemiology of Chronic Diseases (ECD) Study Section • Unsolicited applications • 3 cycles (“rounds”) per year • Submission/Receipt (FEB2004/June2004/Oct2004) • Review (JUNE 2004/Oct2004/Feb2005) • Council (OCT 2004/Jan2005/May 2005) • Award by Institute to “Applicant Institution” vs. “Principal Investigator” • “New Investigator” Status – A “new investigator” has not previously been a principal investigator for an R01.

25. APPLICATION ASSIGNMENT NUMBERS • ORIGINAL SUBMISSION • 1 R01 HL 012345-01 • REVISED SUBMISSIONS • 1 R01 HL 012345-01A1 • 1 R01 HL 012345-01A2 • CONTINIUATIONS • 2 R01 HL 012345-06 • 2 R01 HL 012345-06A1 • 2 R01 HL 012345-06A2 • SUPPLEMENTS (within project period) • 3 R01 HL 012345-01, -01A1, -01A2

26. Function of ECD Study Section • ECD assesses NIH applications for scientific and technical merit; this is the first level of peer review. • NIH Institute Advisory Councils assess applications for relevance to NIH goals and public health needs; this is the second level of peer review. • The Institute Directors use these assessments to make funding decisions.

27. Criteria to Evaluate Scientific and Technical Merit • Significance: Is it? Why? Advance science? • Approach: Design/Population and Sample/Data Collection/Analysis • Innovation: Aims or approach? • Investigator(s) • Environment

28. Study Section Perspective:Pre-Meeting • Submission/Receipt of Applications • Self-Referral: Request Study Section in cover letter • SRA decides if appropriate for Study Section after initial referral • Institutes determine their interest and roles as primary, secondary, etc. • Additional Information (contact SRA) • Human Subjects: IRB approval not needed for review

29. Study Section Perspective:Pre-Meeting • Administrative Issues • Assignments • Chartered members (20 for 4 year terms each; about 5 members nominated by SRA each year; Chair nominated by SRA, usually for 2 years) • Special reviewers (as needed) • Outside opinions/Mail Reviewers

30. Confidentiality andConflict of Interest • Confidentiality: in perpetuity • Conflict of Interest • Assessed by SRA • Assessed by reviewers • Member Conflicts: If a member is on your application, it will be reviewed by a special emphasis panel or another study section.

31. Study Section Perspective:Meeting Procedure of Review • Initial levels of enthusiasm from assigned reviewers • Description and critique from first reviewer (including assessment of potential risks to human subjects and the inclusion of women, minorities, and children) • Critiques from other reviewers • Discussion by entire Study Section • Final levels of enthusiasm • Vote conscience

32. Study Section Perspective:Meeting Procedure of Review • After scoring • Budget recommendations from reviewers, discussed by entire study section • Discussion of Administrative issues (e.g., overlap) • Deferral and Site Visits • An application may be deferred if it is viewed as highly significant and if a small amount of additional information (obtained by the SRA) would address concerns. • Site visits are conducted ONLY if there is no other way to acquire such needed information.

33. Scoring • Judgment by reviewers based on assessment of scientific and technical merit • Criteria not equally weighted • Budget not a factor • Other administrative issues not factors • HS/Gender/Minority/Children inclusion may be factor (as related to aims)

34. Scoring • Streamlining (the “UN”score): about 50% of applications reviewed in a “normal” round • Range: 1.0 - 5.0 (ideal median = 3.0) • Percentile Rank calculated using the current and previous two rounds as a base • Exception: F-series: 1.0-5.0, NR

35. Study Section Perspective:Post-Meeting • Scores/codes/budgets verified by SRA and entered in NIH computer system • Summary Statements prepared by SRA • SRA advises investigators & discusses review • SRA represents the assessments of scientific and technical merit by Study Section at the National Advisory Councils of Institutes • Note: We are receiving the next round’s applications while preparing summaries for the current round.

36. Reorganization of Epidemiology Study Sections in October 2003 • 1965-2003: Epidemiology and Disease Control-1 (EDC-1) Study Section and EDC-2 • 2001: Added EDC-3 • October 2003 reorganization: • EDC-1  Epidemiology of Chronic Diseases (ECD) Study Section • EDC-2  Epidemiology of Cancer (EPIC) Study Section • EDC-3  Epidemiology of Clinical Disorders and Aging (ECDA) Study Section

37. Inside the NIH Grant Review Process Video • CSR has developed a video of a mock study section meeting to show how NIH grant applications are reviewed. http://www.csr.nih.gov/video/video.asp

38. CSR Web Site: http://www.csr.nih.gov • News and Events • Resources for Applicants • Study Section Information • Employment Opportunities • Contact Information

39. Information on the World Wide WebSelected Sites of Interest • National Institutes of Health (http://www.nih.gov) • Office of Extramural Research (http://www.nih.gov/grants/oer.htm) • Grants Policy (http://www.nih.gov/grants/policy/policy.htm) • Center for Scientific Review (http://www.csr.nih.gov) • Referral and Review (http://www.csr.nih.gov/refrev.htm) • CSR Study Section Rosters (http://www.csr.nih.gov/committees/rosterindex.asp) • Review Group Meeting Dates (http://www. csr.nih.gov/committees/meetings/ssmeet1.asp) • CSR Reorganization News (http://www.csr.nih.gov/review/reorgact.asp)