1 / 20

Archived File

Archived File. The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files. Enhancing Peer Review at the National Institutes of Health

ila-bell
Download Presentation

Archived File

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.

  2. Enhancing Peer Review at the National Institutes of Health Andrea Kopstein, PhD, MPH AEA Annual Meeting Denver, CO, November, 2008 National Institutes of HealthU.S. Department of Health and HumanServices

  3. ENHANCING PEER REVIEWPRIORITY 1: ENGAGE THE BEST REVIEWERS

  4. Electronic Reviews • Video Enhanced Discussions • Asynchronous Electronic Discussions Electronic Reviews: Expanding Peer Review Platforms Study Sections Necessity ●Clinical reviewers Preference ● Physicists, computational biologists New Opportunities● Fogarty, International Reviewers Our Goal: 15% of all reviews to be electronic in 2008

  5. Expected Benefits for AED • Greater flexibility for scheduling and running peer review meetings. • Expand potential reviewer base - eliminate travel and time constraints. • Allow for more thoughtful and thorough discussions. • Potentially less confrontational environment • Improved management of conflicts. • Website is straightforward for reviewers and requires little training. • Reduced costs

  6. AED Progress …. • Improvements to Interface (Version 3.1 now being used). • Enhancements to AED: • Unread comments • Unscoring • Identification of conflicts- scoring table • Assigned reviewer display, etc. • 170 review meetings done using AED in 2008 • Wider adoption and acceptance – 5 new NIH IC users and 4 agencies educated about AED in FY2008.

  7. The AED Evaluation • Web-based survey to reviewers and SROs. • Fixed choice and open-ended questions. • Response rates: • Reviewers - 56% (786/1460). • Scientific Review Officers - 60% (55/92) • Both target populations had two follow-up reminders • Although response rates were somewhat lower than desired, the summary of qualitative comments provide some insight into the AED process and AED is an ongoing initiative

  8. SOME EVALUATION FINDINGS – SUMMER 2008

  9. Reviewer Satisfaction with AED Technology

  10. Scientific Review Officer (SRO) Satisfaction with AED Technology

  11. Reviewers: Was the duration of discussion sufficient to allow an appropriate level of consideration of applications?

  12. SRO: Compared to a face-to-face meeting, do you feel the quality of deliberation was:

  13. REVIEWERS: Some of the most frequent positive responses to open questions: • #1 response – AED works well! • #2 - AED provides a review that is more fair because of deliberation time • Many times a reviewer may misinterpret certain facts on an application. If one is not a primary reviewer it is impossible to check out these facts. With this system there is plenty of time to check things out if something does not ring true, or if there is a strong disagreement about a certain point.

  14. REVIEWERS: Frequent positive responses…… • This format relies EXCLUSIVELY on the strengths of an opinion, not on the aggressiveness of a reviewer and their ability to speak loud. • The technology works well • Not having to travel is a big benefit

  15. REVIEWERS: Some of the most frequent negative responses to open questions: • Complaints about some reviewers tending to ignore applications for which they were not assigned or going offline for extended periods of time. In a face-to-face meeting, it is possible to absorb the discussion as it occurs. Passive listening not viable in AED • More difficult to make your case for an applications than in face-to-face meetings

  16. REVIEWERS: Negative responses - continued • Sometimes questions are asked but then get no response • A major disadvantage is the an inability to talk to colleagues during the meeting • Technology was slow, navigation issues, trouble for MACs

  17. SROs: Top positive responses to open questions: • #1- Easier to recruit stellar reviewers • Easier to get clinical reviewers • Easier to get international reviewers • Easier to schedule • Good for reviewing diverse applications

  18. SROs: Top positive responses…….. • Allows multitasking while the review meeting is ongoing • The technology works well • SROs and Reviewers like not having to travel • Meeting can be run from any location where there is Internet access

  19. SROs: Some of the most frequent negative responses to open questions: • Loss of direct interaction with reviewers – personal rapport lost • Concerned that unassigned reviewers don’t review or score carefully • Spotty reviewer participation in discussion, sometimes difficult to engage • More Labor intensive than face-to-face meeting for smaller numbers of applications • Sometimes applications are nitpicked because there is so much time

More Related