1 / 20

Peer review and knowledge dynamics Terttu Luukkonen Research Institute of the Finnish Economy

Peer review and knowledge dynamics Terttu Luukkonen Research Institute of the Finnish Economy. ERC Workshop Monitoring the performance and quality of peer review systems 28-29 November , 2013, Brussels. Focus. Evaluation of research proposals

colum
Download Presentation

Peer review and knowledge dynamics Terttu Luukkonen Research Institute of the Finnish Economy

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Peer review and knowledge dynamicsTerttu LuukkonenResearch Institute of the FinnishEconomy ERC WorkshopMonitoring the performance and quality of peer reviewsystems28-29 November, 2013, Brussels

  2. Focus • Evaluation of research proposals • Groundbreaking/pathbreaking/frontier/ transformative research

  3. Content • Defining groundbreakingresearch • potential differences between fields in the understanding • Phases in the emergence and growth of new research areas • Comparison of peer review with other methods in its ability to select groundbreaking research • Conclusions

  4. Categories of groundbreaking • Discovery of a novel phenomenon (serendipitous discoveries and others) • New method or technique or their combination as an enabler • Access to new data • General explanations (paradigms and other) • unsolved ‘big questions’ (Laudeland Gläser, 2012)

  5. Impacts of groundbreaking researchon • Owndisciplineorresearchfield • Severalotherresearchfields • Openingup new researchfields • Merging fields/interdisciplinary areas of research

  6. Different perspectives • Selection of proposals - Forward look • Promise of opening up new avenues of research • Enabling new research directions • New perspectives • Paradigm shifting, revolutionary • Great uncertainty • Backward • Pin down what gave rise to the observed development – often a longer process and several contributors

  7. TerttuLuukkonen: interviews with 24 ERC peer reviewers, Luukkonen, 2012

  8. Different definitions of groundbreaking and excellenceERC peer review panelists (Luukkonen, 2012) • Groundbreaking • Synonyms: pathbreaking, cutting-edge, frontier • Originality, novelty , revolutionary, paradigm shifting • Excellence • Originality, novelty, going beyond current state of the art, making a difference for the development of science, but also • Robustness of the research, methodological rigour, use of up-to-date methodology, coherent discussion of the research problem and purpose

  9. Development of ideas and scholars over time NR OF IDEAS & SCHOLARS SCHOLARS IDEAS Early adaptors Early Majorities Latecomers Late majorities TIME Brown, 2012

  10. Project selection methods and cycles of ideas NR OF IDEAS & SCHOLARS National RCs National RCs Bibl. methods National RCs ERC HHMI Early adaptors Early Majorities Latecomers Late majorities TIME

  11. ERC peer review system Luukkonen, 2012 • Evaluation criteria • Quality of the peers • Panels have to consider feasibility and risks • Capabilities of the investigator • Instruments and equipment • Contingency plans • Avoid speculation and dilettantism • Put in context, tradition • Reasonable risk

  12. Laudel and Gläser on ERC, 2012 • “ERC grants have impact on research because, at least, some of them fund scientific innovations, the exploitation of recent discoveries, or answers to ‘big questions’ across all discipline groups” • The funded research has epistemic properties not usually met by grants from national funding agencies • Contradicting the mainstream • Addressing the community’s blindspots • Linking otherwise separate communities • Strategic & technical uncertainties • Complexities in equipment, approaches • Length of time it takes to conduct the research

  13. Project selection methods and cycles of ideas NR OF IDEAS & SCHOLARS National RCs National RCs Bibl. methods National RCs ERC HHMI Early adaptors Early Majorities Latecomers Late majorities TIME

  14. Bibliometric measures suggestedHörlesberger et al., 2013 • Novelty • Timeliness: how recent are the publications listed on the reference list of application • Similarity of proposal with emerging topics • Risk • Similarity of the proposed research to the investigator’s previous research • Interdisciplinarity • There has to be some body of publications in the field for the measures to be counted

  15. Project selection methods and cycles of ideas NR OF IDEAS & SCHOLARS National RCs National RCs Bibl. methods National RCs ERC HHMI Early adaptors Early Majorities Latecomers Late majorities TIME

  16. Varieties of peer review • Robustness • Use of international vs national experts • Independence of the panels • Their evaluation instructions and criteria • Further features: • Fine-grained vs. rough marking (Langfeldt, 2001) • Remote reviews vs. or, in addition, panelists’ reviews • Degree of interdisciplinarity of panels • Panels rank or rate • Evaluation criteria: ground-breaking vs. excellence

  17. Conclusions • Peer review conservative? • Not just one peer review, but many varieties in quality, criteria, organisation • The way peer review is organised and applied makes a difference • Peer review in combination with the conditions of the scheme can make a difference for progress of science • Quality of peer review provides legitimacy to the scheme and affects quality of the applicants • A risk that a thorough monitoring of peer review shifts the system towards more conventional proposals – short term indicators

  18. Thank you for your attention! more information: terttu.luukkonen@etla.fi

  19. Literature • Braun, Dietmar. 2012. Why do scientists migrate? A diffusion model, Minerva, 50: 471-491. • Grant, Jonathan and Allen, Liz. 1999. Evaluating high risk research: an assessment of the Wellcome Trust’s Sir Henry Wellcome Commemorative Awards for Innovative Research, Research Evaluation, 8: 201-204. • Hörlesberger, Marianne, Roche, Ivana, Basagni, Dominique, Scherngell, Thomas, Francois, Claire, Cusax, Pascal, Schiebel, Edgar, Zitt, Michel, and Holste, Dirk. 2013. S concept for inferring ‘frontier research’ in grant proposals, Scientometrics, 97: 129-148. • Langfeldt, Liv. 2001. The decision-making constraints and processes of grant peer review, and their effects on the review outcomes, Social Studies of Science, 31/6: 820-841. • Laudel, Grit and Gläser, Jochen. 2012. The ERC’s impact on the grantees’ research and their careers (EURECIA Work package 4 summary report). January 2012. • Luukkonen, Terttu. 2012. Conservatism and risk-taking in peer review: Emerging ERC practices, Research Evaluation, 21 (2012), No. 1, pp. 48–60. • Nedeva, Maria. 2012. Peer review and path-breaking research: selection practices of research funding organisations. Unpublished. • Wagner, Caroline S. and Alexander, Jeffrey. 2013. Evaluating transformative research programmes: A case study of NSF Grants for Exploratory Research Programme, Research Evaluation, 22: 187-197. UNI project: Universities, funding systems, and the renewal of the industrial knowledge base – a project funded by Tekes, 2012-2014; coordinated by TerttuLuukkonen; empirical data gathering with research group leaders in universities in Finland and the UK.

More Related