Archived File

1. Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.

2. CSR Review of Member Conflict Applications Working Group Members: Samuel Edwards, SRA, CMIA Study Section, CHR Anshumali Chaudhari, SRA, VCMB Study Section Valerie Durrant, SRA, HOP-B Special Emphasis Panel Marc Rigas, SRA, BDMA Study Section

3. Acknowledgements • Teresa Lindquist, Program Analyst Office of Planning, Evaluation and Analysis, CSR • QVR Development Team (NIH/CIT), particularly Heather Allison and Don Tiedemann • CSR IRG Chiefs • Michael Martin, Director, Division of Physiology and Pathology, CSR

4. Overview • What is the Law and what is Policy • What is Practice • Demographics of Member SEPs • Comparison of member applications reviewed in Member SEPs and regular study sections with all CSR-reviewed applications • Conclusions and Recommendations

5. Peer Review Regulations • Revised February 4, 2004 • 42 CFR 52h.5d: “When a peer review group meets regularly, it is assumed that a relationship among individual reviewers in the group exists and that the group as a whole may not be objective about evaluating the work of one of its members. In such a case, a member’s application or proposal shall be reviewed by another qualified review group to ensure that a competent and objective review is obtained” • Does not define what constitutes a different review group (previous version stated: no more than 50% members)

6. CSR Policy • Member applications can be reviewed in an appropriate standing study section or special emphasis panel (SEP) • SEP Composition • Limited number (<50%) may be current or recent members of the conflicted member’s study section • Chair of SEP cannot be a member of conflicted member’s study section • SEP percentiling rules (30 – 50%)

7. Distribution of review of member applications (combined 2005 council rounds) What is practice

8. The proportion of member applications reviewed in regular study sections (SRG), member SEPS, and others have remained relatively constant

9. Survey of IRG chiefs of review of member applications • Member applications are reviewed in a variety of ways within and across IRGs, mostly sister study sections and small phone SEPs. • 95% of IRGs use multiple formats to review member applications. • Appropriate expertise is the driving factor for how applications are reviewed

10. Percent of IRGs that reported using various formats for the review of member applications (Note: Chiefs could select more than one option)

11. Most IRGs review some or all member applications in groups (either sister study sections or SEPS) that do not contain any other current or recent members of the conflict study section

12. MEM SEPS are held throughout the round with the peak being 1 month after the peak for regular study section meetings

13. Most (approx. 90%) Member SEPs review 1 – 5 applications

14. Seniority is similar for Member SEPs and standing study section meetings

15. Overview • What is the Law and what is Policy • What is Practice • Demographics of Member SEPs • Comparison of member applications reviewed in Member SEPs and regular study sections with all CSR-reviewed applications • Conclusions and Recommendations

16. How do study section member applications fare? • Examined only R01 applications reviewed by CSR • CSR Member SEP based on ZRG1 prefix and M suffix in the meeting code (plus Decade coding (02 – 07) beginning 2005) • Compared scores (percentiles) for member applications reviewed in Member SEPs and standing study sections (SRGs), and to all R01 applications reviewed in CSR (CSR All). • Percentage of applications in the • Top 10% • Top 20% • Bottom 70% (including streamlined applications) • Differences in Type 1 and 2 applications? • Differences for applications using human subjects?

17. Comparison of scoring of Types 1 and 2 applications reviewed in 2005 in Member SEPs and SRG meetings and to CSR All Type 1 applications Type 2 applications Cumulative Percent Percentile Percentile

18. 0 Type 2 applications Types 1 and 2 applications reviewed in Member SEPs and Standing Study Section meetings (SRG) for Years 2000 – 2005 compared to CSR All Type 1 applications Percentage of applications reviewed 0

19. Fewer member applications reviewed in Member SEPs are streamlined compared to those reviewed in standing study section (SRG) meetings

20. 0 Types 1 and 2 applications the year prior to beginning SS service compared to those reviewed in Member SEPs or study section (SRG) meetings and to CSR All Percentage of applications reviewed

21. Comparison of applications reviewed in small SEPs (Series 90) to member SEPs and to CSR all

22. Conclusions • Member conflict applications comprise a significant segment of applications. • Lots of members + high productivity of members = many member conflict applications • Members are providing a service to NIH • Goal is a fair review for member applications given that by regulation they cannot be reviewed in their parent study section, which may be the most appropriate study section

23. Conclusions: Key findings No SYSTEMATIC bias in the review of members’ applications and, in particular, with the use of member SEPs. Composition (seniority) is similar The percentage of Type 2 applications that receive scores in the top 10 percentile are very similar when reviewed in member SEPS or in regular study sections Members receive comparable scores before and during SS service.  Member applications do better—likely because members are excellent scientists who write good applications.

24. The current system provides appropriate means for reviewing member applications • IRGs differ, study sections differ, and the review of member applications will differ accordingly. • Flexibility is important to ensure most appropriate review venue for member applications. • IRG chiefs are integral!

25. Concerns with member SEPs • Confidentiality of reviewers in small SEPs – adequate protection? • Designer reviews? • SRA workload

26. Recent changes • Encouraged to schedule meetings earlier • Encouraged to avoid 1 or 2 application meetings