NIH Peer Review of Small Business Applications

1. NIH Peer Review of Small Business Applications Dana Jeffrey Plude, Ph.D. Acting Director Division of AIDS, Behavioral and Population Sciences 11th NIH SBIR/STTR Conference July 2009 National Institutes of HealthU.S. Department of Health and Human Services

2. NIH DUAL PEER REVIEW • Scientific Review Group • Evaluate scientific and technical merit • Recommend level of support, duration • Does not make funding decisions • Advisory Council • Reviews the review • Funding recommendations to Institute/Center • Evaluate priorities/relevance • Advise on Policy • Funding decision made by Institute/Center

3. AIDS, Behavioral and Population Sciences Basic and Integrative Biological Sciences Physiological and Pathological Sciences Translational and Clinical Sciences Neuroscience, Development and Aging Endocrinology, Metabolism, Nutrition & Reproductive Sciences Biobehavioral & Behavioral Processes Biological Chemistry & Macromolecular Biophysics Cardiovascular and Respiratory Sciences Brain Disorders & Clinical Neuroscience Risk, Prevention& Health Behaviors Surgical Sciences, Biomedical Imaging and Bioengineering Molecular, Cellular & Developmental Neuroscience Bioengineering Sciences & Technologies Immunology Population Sciences & Epidemiology Integrative, Functional & Cognitive Neuroscience Cell Biology Infectious Diseases & Microbiology Musculoskeletal, Oral And Skin Sciences Healthcare Delivery & Methodologies Genes, Genomes & Genetics Emerging Technologies & Training in Neuroscience Oncology 2 – Translational Clinical Digestive, Kidney & Urological Systems AIDS & Related Research Oncology 1 – Basic Translational Vascular and Hematology Biology of Development & Aging Interdisciplinary Molecular & Training CSR Review Divisions

4. PEER REVIEW IN CSR • Scientific Review Groups/Study Sections/Review Committees/Special Emphasis Panels - Small Business applications are reviewed in Special Emphasis Panels. • Scientific Review Officer is responsible for the management of the meeting. • Chair and 10 - 24 members who are from academia and small businesses • 40 - 100 applications are reviewed at each study section meeting (face to face meeting). • CSR is experimenting with review platforms – video conferences, asynchronous extended discussion, and others.

5. What It Looks Like: Video Enhanced Discussions

6. What It Looks Like: Asynchronous Electronic Discussions

7. SCIENTIFIC REVIEW OFFICER • Performs administrative and technical review of applications to ensure completeness • Selects reviewers based on broad input • Assigns reviewers (generally at least two reviewers and one reader per application) • Manages review meeting/Designated Federal Official • Prepares Summary Statement • Provides requested information about review recommendations to Institutes/Centers and advisory councils

8. CRITERIA FOR SELECTION OF PEER REVIEWERS • Demonstrated scientific expertise • Doctoral degree or equivalent • Mature judgment • Work effectively in a groups • Breadth of perspective • Impartiality • Interest in serving • Adequate representation of women and minority Scientists • Geographic distribution • Small Business representation

9. REVIEW MEETINGS • Closed to the public • Introductions • SRO instructions: Confidentiality, conflict of interest, new NIH policies, procedures • Approximately 60% of the applications are Discussed, the ‘less competitive’ 40% of the applications are Not Discussed • Applications are discussed in ‘score order’ based on Preliminary “Overall Impact Score” given by the three assigned reviewers • Note - any member can nominate an application for discussion

10. REVIEW MEETINGS (cont). Individual review of top 60% applications • Conflicts excused • Assigned reviewers/readers: preliminary scores, evaluations • General discussion, including human subjects, vertebrate animals, biohazard issues • Private assignment of priority score • Discussion of budget, data sharing, resource sharing issues “Mock” Study Section video http://cms.csr.nih.gov/ResourcesforApplicants/InsidetheNIHGrantReviewProcessVideo.htm

11. REVIEW LOGISTICS • Reviewers receive applications and assignments one to two months in advance of meeting. • Reviewers post preliminary scores and critiques on a secure web site in advance. • Reviewers may see other critiques only after their own are posted. • Are not aiming for consensus but outliers must explain their views • Critiques are modified in light of discussion. • SRO prepares Resume and Summary of Discussion.

12. 2008: The Year of Peer Review Enhancing Peer Review “Fund the best science, by the best scientists, with the least administrative burden…” Elias Zerhouni, MD, Former Director, NIH

13. Recommendations

14. Amended Applications: To speed the funding of meritorious science and minimize reviewer burden: • As of January 25, 2009, all original new applications (i.e., never submitted) and competing renewal applications will be permitted only a single amendment (A1).

15. Balanced and Fair Reviews Across Career Stages and Scientific Fields • New Investigator: Not previously competed successfully as PD/PI for a significant NIH independent research award.  • Early Stage Investigator: Within 10 years of completing terminal research degree or is within 10 years of completing medical residency (or the equivalent).  • These Designations are IRRELEVANT for Small Business applications – only R01 research grant applications are coded for New PI/ESI

16. Scoring To improve the transparency of the scoring process: • Score applications on five review criteria using a scale of 1-9. • Preliminary overall impact score using 1-9 scale. • Discussed applications will receive an overall score from each eligible (i.e., without conflicts of interest) panel member and these scores will be averaged to one decimal place, and multiplied by 10. The 81 possible priority scores will thus range from 10-90. • Percentiles will be reported in whole numbers. • All applications will receive scores: • Not Discussedapplications will receive initial criterion scores from the three assigned reviewers

17. Scoring on a 9-Point Scale • Goal: Improve the transparency of the scoring process: Score applications on five review criteria using a scale of 1-9. • Preliminary overall impact score using 1-9 scale. • Should not be the average of the criterion scores. • Not Discussed applications will receive initial criterion scores from the three assigned reviewers.

18. ScoringDescriptions Strengths

19. Scoring Expectations: • Reviewers participate in entire meeting . • To rank the applications in order of review based on preliminary score, all applications must be scored; applications with initial scores of 7-9 are those unlikely to be discussed. • Once at the meeting, the full range of scores is available to assign an overall impact score for all discussed applications.

20. Scoring (cont.) • Criterion scores are considered part of the critique and will not be discussed at the review meeting. • They may be changed in the EDIT phase in IAR.

21. Clustering • Phase I, Phase II and Fast Track Applications will be clustered together (provided sufficient numbers). • Phase I: FEASIBILITY • Phase II: EFFICACY • Fast Track: Both Components

22. ClusteringofClinicalApplications • The goal of clustering clinical applications in review is to increase fairness in the process of reviewing clinical applications. • A clinical application will be defined as human subjects research minus Exemption 4.

23. Order of Review • Goal: To discuss applications in order of average preliminary score. • Why? • Concern: variation of scores during different times of the meeting. • One recommendation was to recalibrate scores at the end of the meeting. • The Solution • Recalibrate “dynamically” throughout meeting.

24. Order of Review • For calibration purposes… • Begin meeting by discussing the best scored application (any activity code). • All Activity codes are clustered if f feasible (if at least 10 discussed, may include R41/43, R42/44 and Fast Track as groups that can be clustered).

25. OrderofReview Summary • Discussion order is based on the average of the impact scores from assigned reviewers. • Final scores of discussed applications may differ from preliminary scores as re-calibration happens dynamically.

26. Not Discussed • Discuss ~ 40% of small business applications. • SRO will then ask if there are any other applications that panel wishes to discuss. • The remaining applications will not be discussed (applications receive criterion scores only).

27. Final Scores • Discussed applications will receive an overall score from each eligible (i.e., without conflicts of interest) panel member and these scores will be averaged to one decimal place, and multiplied by 10. The 81 possible priority scores will thus range from 10-90. • Percentiles will be reported in whole numbers.

28. Summary Statements • Summary statement will beshorterand more focused. • Discussed applications will also have a summary of the panel’s discussion at the meeting. • ALL applications will be scored. • Not discussed applications will receive criterion scores only.

29. Critiques To improve the quality of the critiques and to focus reviewer attention on the review criteria: • Electronic template for critiques that will prompt for strengths and weaknesses for each criterion. • Summary statement that will be shorter and more focused. • Discussed applications only also will have a summary of the panel’s discussion at the meeting. • In 2009, ALL applications will be scored. • Not discussed applications will receive criterion scores only in addition to the reviewers’ critiques

30. Template-Based Critiques • Critiquetemplate contains a total of 18 boxes. • Reviewers should provide text for only those criteria that are applicable.

31. Template-BasedCritiques • Goal: Write evaluative statements and to discourage summarizing the application. • Comments should be in the form of bullet points or, if necessary, short narratives. • Do not record scores on the critique template. • The entire template is uploaded to IAR to become part of the summary statement.

32. REVIEW CRITERIA Specifically tailored for small business applications: • Significance • Investigator(s) • Innovation • Approach • Environment • Overall Impact – weigh as appropriate, consider impact on scientific field

33. SIGNIFICANCE • Does the project have commercial potential to lead to a marketable product, process or service? • If the aims of the project are achieved, how will scientific knowledge or clinical practice be advanced? • Does this study address an important problem and what commercial and societal benefits might be derived from the proposed research? • What will be the effect of these studies on the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field? • Does the project lead to enabling technologies (e.g., instrumentation, software) for further discoveries? • Will the technology have a competitive advantage over existing/alternate technologies that can meet market needs?

34. INVESTIGATORS • Is the PD/PI appropriately trained and capable of coordinating and managing the proposed project? • Are the investigators well suited to carry out this work and does the investigative team bring complementary and integrated expertise to the project (if applicable)? • Is the work proposed appropriate to the experience level of the PD/PI and other researchers, including consultants and subcontractors (if any)? • Are the relationships of the key personnel to the small business and to other institutions appropriate for the work proposed?

35. INNOVATION • Are the aims original and innovative? • Does the project challenge existing paradigms or clinical practice; address an innovative hypothesis or critical barrier to progress in the field? • Does the project develop or employ novel concepts, approaches, methodologies, tools, or technologies for this area?

36. APPROACH • Are the conceptual or clinical framework, design, methods, and analyses adequately developed, well-integrated, and appropriate to the aims of the project? • Is the proposed plan a sound approach for establishing technical and commercial feasibility? • Does the applicant acknowledge potential problem areas and consider alternative strategies? • Are the milestones and evaluation procedures clearly described and appropriate? • For applications designating multiple PD/PIs, is the leadership approach, including the designated roles and responsibilities, governance and organizational structure consistent with and justifies by the aims of the project and the expertise of each of the PD/PIs?

37. ENVIRONMENT • Is there sufficient access to resources (e.g., equipment, facilities)? • Does the scientific and technological environment in which the work will be done contribute to the probability of success? • Do the proposed studies benefit from unique features of the scientific environment, or subject populations, or employ useful collaborative arrangements? • Is there evidence of institutional support?

38. SPECIAL REVIEW CRITERIA • Resubmissions • Responses to comments from the previous review • Improvements in application • Phase II Applications • Progress in Phase I – objectives met, feasibility demonstrated • Commercialization Plan • Commercial potential • Fast Track Applications • Measurable goals/milestones • Commercialization Plan • Commercial potential • Funding commitments/resources

39. ADDITIONAL REVIEW CRITERIA • Protection of human subjects from research risk • Inclusion of women, minorities and children • Care and use of vertebrate animals • Biohazards Problems in these areas will have a deleterious impact on the review outcome/score

40. OTHER REVIEW CONSIDERATIONS • Budget – amount, duration • Plans for Data Sharing if required • Plans for Model Organism sharing • Genome Wide Association Studies – plans for sharing data generated These do not contribute to the score

41. REVIEW OUTCOME • Not recommended for further consideration – rare, usually due to serious ethical or safety concerns • Deferral – also rare, study section does not have sufficient information to make a determination • Not Discussed 40% (approximately) • Scored 60% (approximately)

42. SUMMARY STATEMENT • Not Discussed/Priority Score • Overall Resume and Summary of Discussion if Discussed • Essentially unedited critiques • Budget recommendations • Administrative Notes • Available in Commons only to Principal Investigator

43. Example 1 – Phase II project PROTECTION OF HUMAN SUBJECTS UNACCEPTABLE RESUME AND SUMMARY OF DISCUSSION: The proposed project is intended to provide web-based education to and collaboration between parents and teachers of children with ADHD. The proposed program addresses a timely and important problem and if effective, this product would be a welcomed tool. The investigative team is exceptionally well qualified, Phase I successfully demonstrated the program’s feasibility and Phase II is well poised to refine the prototype. During discussion, however, the Committee noted several issues that limit the potential of the Phase II project. For example, the proposed assessment of the program’s effectiveness does not rely on parents and teachers of students with ADHD, and it does not assess the actual collaborative experiences of parent-teacher dyads that use it. Another concern centers on the plan to provide parents with clinical tools without proper guidance from mental health professionals. …There is also some question about the feasibility of creating a product that is relevant and useful for the full range of experience with ADHD that characterizes the target audience.

44. Example – Fast Track project RESUME AND SUMMARY OF DISCUSSION: The proposed project develops a brief screen for depression in antenatal and postpartum women. Although it is unclear how feasible it is to obtain a reliable screening tool that is shorter than the 10-item Edinburgh Postpartum Depression Scale, if the project is successful the product would provide an important clinical tool. The project is especially significant because there is not an adequate scale to assess depressive symptoms among pregnant women who have physical symptoms. Using item response theory to develop the screener is innovative and will potentially result in a highly marketable product. The application outlines a strong commercialization plan. There are, however, some aspects of the proposed methodology that are questionable and there is concern that all of the members of the expert panel are directly involved in the project. Also Phase II lacks a gold standard assessment of depression for comparison with the final product. … These issues notwithstanding, the proposed product is innovative and, if successful, would be a significant contribution to the field.

45. ADVISORY COUNCIL/BOARD CONSIDERATION • Second required step in dual peer review process • Most applications considered in enbloc action; a few are specifically discussed (budget change, deferral)

46. QUESTIONS