1 / 26

Archived File

Archived File. The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files. PEER REVIEW ADVISORY COMMITTEE

diallo
Download Presentation

Archived File

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.

  2. PEER REVIEW ADVISORY COMMITTEE General Principles of Application Clustering Don Schneider, Ph.D. December 3, 2007 National Institutes of HealthU.S. Department of Health and Human Services

  3. Clustering Topics • Spread of Integrated Review Group (IRG) review assignments • Overall review outcomes in low & high clustered IRGs • New investigator review outcomes in low & high clustered IRGs • Discussion items

  4. Clustering and Peer Review • Review by people of the same rank (equals) • Broad sense – any research scientists who appreciate/understand the scientific problem • Narrow sense – scientists with direct experience in the area of proposed research • Clustering of similar applications (bunching of the same sort together) lends itself to peer review in the narrow sense

  5. Clustering Factors • Panel on Scientific Boundaries for Review (PSBR) set a very high bar 30% (Jan 2000 report) • PSBR de-clustered some communities • Some areas of science, esp. new areas, bridge existing study sections, e.g., genetics of human behavior • Some areas of science are inherently diverse, e.g., complications of diabetes • Research in some areas of science is diminishing, e.g., thyroid metabolism • Extreme clustering essentially establishes an entitlement and is counter to broad study sections

  6. Numbers of Applications • CSR reviews more than 15,000 applications a cycle • Study sections average about 80 applications a cycle • The PSBR 30% clustering would be a set of about 24 applications • “Low” clustering here is an average 5 or fewer applications per IRG per cycle • Three subsets to be examined, each about 200 applications per cycle, 1.3% of total

  7. Possible Actions To Promote Clustering • Cluster better within Integrated Review Groups (IRGs) • Cluster within fewer IRGs • Form a Special Emphasis Panel (SEP) • Constitute a Working Group to consider formation of a new chartered study section

  8. Possible Actions by Program • Designate low represented areas high program priority (“just pay them”) • Write Program Announcements (PAs)/Requests For Applications (RFAs)

  9. PRAC – Clustering Working Group • Aim – to examine clustering and to develop recommendations for PRAC & CSR Director • First meeting 11/30/2007 • Noni Byrnes, CSR • Patricia Greenwel, CSR (Data mining and graphics) • Ann Hagan, NIGMS • Leslie Leinwand, PRAC, Colorado • Don Schneider, CSR • Ross Shonat, CSR • Phil Smith, NIDDK

  10. Subset A review assignments, 2005-2007 Councils 300 250 200 # of Applications 150 100 50 DIG RES RUS CB CVS IFCN EMNR IDM BCMB BDA BDCN GGG HEME HOP IMM MOSS ONC BST MDCN (258) (185) (169) (129) (116) (104) (52) (44) (37) (36) (35) (22) (22) (18) (16) (15) (13) (9) (4)

  11. R01s in CSR IRGs 80 60 40 Standard Distribution Cumulative Percent 20 0 0 20 40 60 80 Percentile

  12. Subset A R01s in CSR IRGs* 80 60 Trend in High Representation IRGs Cumulative Percent 40 20 20 40 60 80 Percentile *2005-2007, excluding population-based IRGs

  13. Subset A R01s in CSR IRGs* 80 60 40 Trend in Low Representation IRGs Cumulative Percent 20 20 40 60 80 Percentile *2005-2007, excluding population-based IRGs

  14. Subset A R01s in CSR IRGs* 80 60 Trend in High Representation IRGs 40 Cumulative Percent Trend in Low Representation IRGs 20 Standard Distribution 0 0 20 40 60 80 Percentile *2005-2007, excluding population-based IRGs

  15. CSR “Scatter Plot” Committee • Aim – to provide QVR search guidelines, plot instructions, and statistical tool for measuring significance • Sam Edwards • Brian Hoshaw • Malgorzata Klosek • Kristin McNamara • Marc Rigas • Don Schneider • Chris Sempos

  16. Subset B review assignments, 2005-2007 Councils 1200 (1062) 1000 800 # of Applications 600 400 200 (73) (6) (40) (21) (6) (2) (9) (14) (9) (6) (25) (3) (12) (1) ONC IMM GGG BDA HOP CB BST IDM IFCN RPHB SBIB BCMB MOSS BDCN MDCN

  17. Subset B R01s in CSR IRGs* 70 60 Trend in High Representation IRGs 50 40 Cumulative Percent Trend in Low Representation IRGs 30 20 Standard Distribution 10 20 40 60 80 Percentile *2005-2007, excluding population-based IRGs

  18. Subset C review assignments, 2005-2007 Councils 400 (380) 300 # of Applications (185) 200 (122) (100) 100 (23) (17) (12) (12) (10) (11) (9) (10) (3) (3) (4) DIG RES BDA HOP ONC GGG IMM IFCN BST CVS EMNR MOSS RPHB BDCN MDCN

  19. Subset C R01s in CSR IRGs* 80 60 Trend in High Representation IRGs 40 Cumulative Percent Trend in Low Representation IRGs 20 Standard Distribution 20 40 60 80 Percentile *2005-2007, excluding population-based IRGs

  20. R01s in CSR IRGs 50 45 40 Standard Distribution (all PIs) 35 30 Cumulative Percent 25 20 Standard Distribution (only new PIs) 15 10 5 20 40 60 80 Percentile

  21. Subset A R01s in CSR IRGs* New Investigators only 50 45 40 Trend in High Representation IRGs 35 30 Trend in Low Representation IRGs Cumulative Percent 25 20 15 10 Standard Distribution 5 20 40 60 80 Percentile *2005-2007, excluding population-based IRGs

  22. 50 45 40 35 30 25 20 15 10 5 20 40 60 80 Subset B R01s in CSR IRGs* New Investigators only Trend in High Representation IRGs Trend in Low Representation IRGs Cumulative Percent Standard Distribution Percentile *2005-2007, excluding population-based IRGs

  23. 50 45 40 35 30 25 20 15 10 5 20 40 60 80 Subset C R01s in CSR IRGs* New Investigators only Trend in High Representation IRGs Trend in Low Representation IRGs Cumulative Percent Standard Distribution Percentile *2005-2007, excluding population-based IRGs

  24. Conclusions • Clustering is a complex matter, but advantageous to some • Low clustering presents challenges (many variables) and perhaps occasional advantages • Tools for assessing clustering and for determining statistical significance of clustering are needed and are under development (by CSR “Scatter Plot” Committee)

  25. Discussion items • What should our clustering goal be? • Is low clustering generally a disadvantage? • Do we need a more comprehensive study of clustering, including vulnerable applicants, with broad SRO and PO input? • Should CSR provide interim realignment solutions, such as reversible SEPs?

  26. Discussion

More Related