Challenges and Opportunities in Peer Review: Ensuring Strategic National Value

1. Challenges and Opportunities in Peer Review A Vision for Ensuring Its Strategic National Value toni scarpa scarpat@csr.nih.gov 301-435-1109 FASEB Board Bethesda, MD May 31, 2009 National Institutes of HealthU.S. Department of Health and HumanServices

2. NIH and Peer Review The Drivers for Change Enhancing Peer review: CSR Enhancing Peer review: Corporate NIH ARRA, The Stimulus NIH Peer Review

3. NIH Peer Review

4. 30 Years of Medical Innovation MRI and CT Imaging ACE inhibitors Angioplasty Statins Mammography Coronary Interventions H inhibitors and H2 Blockers Antidepressant Cataract and Lens Replacement Knee and Hip Replacement Ultrasound Imaging Asthma Treatment • Cardiac Enzymes • Fluoroquinolones • Hypoglycemic Agents • HIV Testing and Intervention • Tamoxifen • PSA • H. Pylori Test and Treatment • Bone Densitometry • Cephalosporins • Calcium Blockers • Conscious Sedation Fuchs and Sox, Health Affairs, 20, 30-42

5. Why Has The U.S. Biomedical-Behavioral Research Been So Successful? Evolution of unique dynamic partnerships -- through NIH -- between Government and academic/medical schools 100% of NIH funds to universities and medical centers awarded through peer review (Only 5-20% in Europe) Peer Review

6. The Overarching Influence of NIH Peer Review in the USA For U.S. Universities and Academic Medical Centers The promotion committee of medical schools The prestige and finances of universities and medical schools • For People and for Public Health • Which research is done • Which cures people get • 7000 Diseases Affect Humankind • 6000 are Orphan Diseases

7. CSR Peer Review: 2008 77,000 applications received 16,000 reviewers 1,600 review meetings 240 Scientific Review Officers • 2,500

8. CSR Peer Review: 2009 77,000 applications received 16,000 reviewers 1,600 review meetings 240 Scientific Review Officers 115,000 38,000 2,500

9. The Drivers for Change

10. 1st Driver: The NIH Budget $30 $ In Billions $29.5 $29.5 $29.1 $28.6 $28.6 $28.0 $25 $27.1 $23.3 $20 $20.5 $17.8 $15 $15.6 $13.7 $10 $5 $0 2003 2005 2006 2007 2002 2004 2008 1998 1999 2000 2001 2009 Doubling

11. 2nd Driver: Number of Applications Historical Growth

12. 3rd Driver: Reviewer’s Load Applications Per Reviewer

13. 4th Driver: CSR Budget $ Millions

14. Annual Savings in Reviewers’ Expense Budget • Non-refundable tickets with one possible change • $15 million • 3,000 fewer reviewers • $3 million • 15% reviews using electronic platforms • $5 million • One meeting a year on the West Coast • $1.8 million

15. CSR’s Efforts to Enhance Peer Review

16. Major Complaints About NIH Peer Review • The process is too slow • There are not enough senior/experienced reviewers • The process favors predictable research instead of significant, innovative, or transformative research • The time and effort required to write and review are a heavy burden on applicants and reviewers

17. Reorganizing CSR and Recruiting Staff Improving Study Section Alignment Shortening the Review Cycle Advancing Additional Review Platforms Recruiting the Best Reviewers AATS Peer Review Outcome CSR’s Efforts to Enhance Peer Review

18. AIDS, Behavioral, Population Basic- Integrative Biological Sci Physiological Pathological Sci Translational and Clinical Sci Neuroscience, Development and Aging Biobehavioral & Behavioral Processes Biological Chemistry & Macromolecular Biophysics Endocrinology, Metabolism, Nutrition & Reproductive Sciences Cardiovascular and Respiratory Sciences Brain Disorders & Clinical Neuroscience Risk, Prevention& Health Behavior Surgical Sciences, Biomedical Imaging and Bioengineering Molecular, Cellular & Developmental Neuroscience Bioengineering Sciences & Technologies Immunology Population Sciences and Epidemiology Integrative, Functional & Cognitive Neuroscience Cell Biology Infectious Diseases & Microbiology Musculoskeletal, Oral And Skin Sciences Healthcare Delivery & Methodologies Genes, Genomes &Genetics Emerging Technologies & Training in Neuroscience Oncology 2 – Translational Clinical Digestive, Kidney & Urological Systems AIDS & Related Research Oncology 1 – Basic Translational Vascular and Hematology Biology of Development & Aging Interdisciplinary Molecular & Training 1. Reorganizing CSR

19. 2. Improving Study Section Alignment Input from the community Internal IRG reviews Open Houses PRAC

20. 3. Shortening the Review CycleWhy?First Response was 5.2 months, far too longThe GoalTo review and post score and critique application within 3 months of submission. To enable resubmission, when doable and desirable, 4 months earlier than in the past.The ResultEvery New Investigator and Most Established Investigators are eligible13% of those eligible apply

21. 4. Advancing Additional Review Platforms Additional Review Platforms Help Recruiting Reviewers Electronic Review Modes Reduce Travel Electronic Reviews Telephone Enhanced Discussions Video Enhanced Discussions Asynchronous Electronic Discussions (AED)

22. 4. Advancing Additional Review PlatformsReviewer Satisfaction with AED Review

23. 5. Recruiting the Best Reviewers

24. 5. Recruiting the Best ReviewersAcademic Rank of ALL CSR Reviewers

25. 5. Recruiting the Best ReviewersSome Successful Strategies • Move a meeting a year to the West Coast • Additional review platforms • Develop a national registry of volunteer reviewers • Searchable database with 5,000 reviewers • Provide tangible rewards for reviewers • No submission deadlines for chartered members • of study sections (effective February 2008). • Provide flexible time for reviewers • Choice of 3 times/year for 4 years or • 2 times/year for 6 years

26. Enhancing Peer Review The NIH Director’s Recommendations

27. Corporate NIH: Enhancing Peer Review • The Charge from Dr. Zerhouni: “Fund the best science, by the best scientists, with the least administrative burden…” Two advisory committees to the NIH Director http://enhancing-peer-review.nih.gov

28. The Process June 2007 – Feb. 2008 March2008 – June 2008 September 2008 2

29. 1. Early Stage Career Investigators Definition of New Investigator: • Not previously competed successfully as PD/PI for a significant NIH independent research award.  Definition of Early Stage Investigator: • Within 10 years of completing terminal research degree or within 10 years of completing medical residency (or the equivalent). • The NIH corporate policy is to fund R01s of New Investigators and ESIs at different paylines • 3 Paylines for R01s Applies only to R01 applications • Advice for ESI (and to New PI) • Apply NOW!!! • Apply for R01!!!!!

30. 2. Review Highly Transformative Research OD Transformative RO1 (T-RO1) Once a year, 5 years, $40 million total budget Deadline January 29, 2009 8-page application 740 Editorial Board Review Heavy triage based on innovation and potential science transformation by a small study section of distinguished, broad-science reviewers (the editors) Specific science reviewed by appropriate reviewers (theeditorial board) Final ranking by the editors

31. 3. Funding the Best Research Earlier More flexible deadlines Abolish A2 applications

32. 4. Enhanced Review Overall Impact: Assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) New Core Criteria Order: Significance Investigator(s) Innovation Approach Environment

33. 4. Template-Based Critiques The objective is to write evaluative statements and to avoid summarizing the application Comments should be in the form of bullet points or if necessary short narratives The entire template is uploaded to IAR to become part of the summary statement.

34. 4. Scoring

35. 4. The Essence of Scoring Before the Meeting Every assigned reviewer will post criteria scores AND overall impact scores. The impact scores will be used to determine the order of discussion. During the Meeting For the discussed applications: The overall impact score is stated by the assigned reviewers. Criteria scores are not mentioned during the discussion. After the Meeting The applicant with a discussed application will see 3 or more criteria scores AND the overall impact score (the one that will be percentiled) The applicants with a non discussed application will see ONLY the criteria scores (3 or more for each criterion)

36. 4. Enhancing Peer Review Training CSR and NIH Review Staff 6 face to face training sessions, January 2009 6 face to face training sessions, April 2009 Continuous updating Chairs For Chairs appointed in 2008, 8 sessions in January 2009 For Chairs appointed in 2009, 7 sessions in April-May 2009 Reviewers Training material (Power Point, interactive training, frequently asked questions, mock study section video, etc,in April-May 2009 Senior CSR staff at the first meeting in May-July 2009

37. American Recovery and Reconstruction Act

38. The Stimulus

39. ARRA Applications Under Review • RC1. Challenge Grants • Deadline April 27 • Received so far 20,847 • Verified and Assigned to IRGs 19,107 • Assigned to 3 Reviewers 14,313 • 2 Stages Editorial Board Review • Each application assigned to 3 reviewers (20,000), reviews due by June 5th, extended to June 12th • 30 Special Study Sections in early July • Competitive Revisions • Deadline April 23 • Received 1,985 • Large Majority Assigned to the Standard 250 Study Sections • Study Sections will review as a SEP in May-June • Stem CellsChallenges

40. This is CSR September 2008