1 / 14

Methods and Uses of Peer Review - Challenges and Lessons Learned From a Canadian Perspective

Methods and Uses of Peer Review - Challenges and Lessons Learned From a Canadian Perspective. Jennifer Birta, National Research Council Canada New Frontiers in (R&D) Evaluation Conference Vienna, Austria 25 April 2006. Presentation Overview. Overview of the National Research Council

amir-glenn
Download Presentation

Methods and Uses of Peer Review - Challenges and Lessons Learned From a Canadian Perspective

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Methods and Uses of Peer Review -Challenges and Lessons Learned From a Canadian Perspective Jennifer Birta, National Research Council Canada New Frontiers in (R&D) Evaluation Conference Vienna, Austria 25 April 2006

  2. Presentation Overview • Overview of the National Research Council of Canada (NRC) • Terminology • Case Study • Overview of initiative • Peer reviews conducted • Challenges and opportunities • Conclusions

  3. National Research Council Canada (NRC) • Agency of the Canadian federal government • Principal scientific research body conducting intramural research • Range of research from fundamental to applied in areas such as: • Molecular sciences • Astronomy • Biotechnology • Environmental technologies • Provision of innovation support • Industrial Research Assistance Program • Technology cluster strategy 2004-2005 # of papers published: 1,286 # of patents issued: 86 # of licenses issued: 105

  4. National Research Council Canada (NRC) • 20 institutes and 10 other research centres • 4,000+ employees • 1,300+ students, visiting/guest workers • Laboratories and facilities across the country • Total expenditures 2004-05: $712M • Income 2004-05: $103.7M

  5. Terminology PURPOSES Journal manuscript review Proposal review Evaluation review • Review of research results for competence, significance and originality prior to publication • Assessment of proposals for future funding decisions • Performance assessment of level of excellence and impact of past work METHODS Paper-based Panel Standing Committee • Provide written opinion only, no interaction • Come together to conduct a single review • Conduct many reviews over a term

  6. Case Study: Overview of NRC’s Genomics and Health Initiative • Horizontal initiative across 10 NRC Institutes • Research focus  fundamental and applied (e.g., diagnosis of disease, crop enhancement) • Annual expenditures ~ $25M (CDN) • Started in 1999, divided into phases of 3 year lengths, internal competitive process used to select programs • Responsibility of the VP Life Sciences, managed by a coordination office and a standing committee

  7. Case Study: Peer Reviews of NRC-GHI • Research program selection every 3 years • Competitive approach for funding • Scientific and market-driven assessment criteria • Two-tiered approach used • Periodic retrospective evaluation • Assessment of relevance, success, alternatives and effectiveness of design/delivery • Seven methodologies used • Peer review  achievement of objectives and progress made • Two-tiered approach used

  8. Case Study: Challenges Subjective Bias • Reliance on human judgement • Trust scientists’ desire not to diminish their reputations Confidentiality • Possibility of inappropriate use of knowledge gained during review • Precautionary measure to sign a confidentiality agreement • Best defence  not to disclose sensitive or proprietary information

  9. Case Study: Challenges Conflict of Interest • Ability of reviewers to participate without conflict with their own interests • Difficult to find qualified individuals with no prior knowledge of program under review • Sources of potential reviewers • Disclosure of conflict of interest Timing • Short timeframe between two peer reviews • Most qualified individuals had already participated • Result  low return rate

  10. Case Study: Challenges Reviewer Fatigue • More S&T organizations relying on peer review • Difficulty finding specialists, willing and able to participate Level of Commitment • Paper-based reviews lack sense of community that exists with panels/committees • Dependent on profile/reputation of initiative under review • Might be increased with provision of honoraria

  11. Case Study: Challenges Supporting Materials • Existing documentation has limited applicability • Customized materials required Costs • Travel and hospitality expenses, honoraria (if applicable) • Loss of productivity for reviewers and those being reviewed

  12. Case Study: Opportunities Standing Committees • Benefit from familiarity and sense of obligation Integration • Combine retrospective and prospective portions of reviews Honoraria • Revisiting NRC’s policy

  13. Conclusions Way forward… • Has the support of most researchers/scientists • Alternatives will do little to reduce pressures on the system • Remains the most appropriate means to assess scientific merit of past, present and future work

More Related