1 / 25

A concept for inferring « Frontier Research » in research project proposals

Dirk Holste Edgar Schiebel Thomas Scherngell Marianne Hörlesberger. Ivana Roche Dominique Besagni Claire François Pascal Cuxac. A concept for inferring « Frontier Research » in research project proposals. ISSI 2011 – Durban, South Africa, July 4-7. Plan of the presentation.

ginata
Download Presentation

A concept for inferring « Frontier Research » in research project proposals

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Dirk Holste Edgar Schiebel Thomas Scherngell Marianne Hörlesberger Ivana Roche Dominique Besagni Claire François Pascal Cuxac A concept for inferring « Frontier Research » in research project proposals ISSI 2011 – Durban, South Africa, July 4-7

  2. Plan of the presentation Introducing the concept The ERC framework Scientometrics & Text mining Frontier research & Indicators The indicators Proposal selection function Discussion 2 A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  3. Introducing the concept Concept for inferring attributes of frontier research in peer-reviewed research project proposals under the scheme of the European Research Council (ERC) • Design, implement, test and refine a bibliometric/scientometric model that investigates whether submitted and/or selected grant applications fulfill the requirements of frontier research as this notion was enounced by the High Level Expert Group of the ERC • Build and compare outcomes of the model with the review decision in order to obtain further insight and to bring a reflection on the influence of frontier research in the peer-review process A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  4. The ERC framework (1/4) First European funding body to support investigator-driven (frontier) research through open and direct competition • Main goals: • Scientific excellence as the only scientific selection criteria • Major grants for the truly best and creative researchers and their ideas, to identify and explore new opportunities and directions in all fields of research • Scientific domain clusters (panels): • Physical and Engineering Sciences (PE)  10 panels • Life Sciences (LS)  9 panels • Social Sciences and Humanities (SH)  6 panels • Grant Application schemes: • Starting independent researcher grants (StGs) • Advanced investigator grants (AdGs) • proposals submitted by principal investigator (PI) • to every project is allocated a main panel that becomes its “home discipline” A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  5. The ERC framework (2/4) • StGs  Up-and-coming investigators: bring means and opportunities for young investigators to develop independent careers • AdGs  Established senior researchers: foster pioneering, far-reaching research endeavors, with high-risk/high-impact potential, break of established disciplinary boundaries or exploration of new productive lines of enquiry, methodology or techniques • Funded through EU’s Framework Programme FP7 • Annual calls: 1st in 2007 A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  6. The ERC framework (3/4) • ERC annual budget evolution (2007-2013): • Rate of selected proposals: • StGs (2009)  10% (244 out of 2,503 submitted proposals) • AdGs (2009)  15% (244 out of 1,584 submitted proposals) • Selection process: • 25 expert panels • members selected by the Scientific Council • two-step peer-review process A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  7. The ERC framework (4/4) The issue of the frontier research for the ERC  the ERC’s High Level Expert Group enounced key attributes of frontier research: Standing at the forefront of creating new knowledge… responsible for fundamental discoveries… achieving occasional revolutionary breakthroughs Intrinsically risky Reducing the barriers between basic and applied research Pursues questions irrespective of established disciplinary boundaries The peer-review process shall indentify such activities /researchers out of an immense number of submitted applications A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  8. Scientometrics & Text mining • Identification of 4 key attributes according to the understanding of the ERC High Level Expert Group’s notion of frontier research: • Novelty • Risk • Applicability • Interdisciplinarity • Formalisation of the key attributes in terms of indicators build by combining scientometric and text mining approaches: A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  9. Frontier research & Indicators A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  10. Timeliness indicator It is used to infer the « innovative degree » of the proposal through the bibliographic references cited by the PI in his(her) proposal • Data source:ERC data • Hypothesis: • the more recent references are, the more likely the work is at the frontier research • Calculation: • this indicator focuses on the time elapsed since the publication of the proposal cited references and Timeliness is obtained from the difference between the year of the proposal submission and the year of publication of each cited reference in the proposal A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  11. Proximity indicator • It is employed to infer the « innovative degree » of the proposal through the dynamic change of the scientific landscape corresponding to the proposal’s allocated panel • Data source: ERC data & bibliographic DB • Hypothesis: • the closer a proposal is to regions of positive dynamic change, the more innovative it is • Calculation: • raw data are obtained from bibliographic DB and ERC panel description to identify and extract discriminating terminological information • a clustering step produces a cluster map that groups similar references and represents the publication landscape corresponding to the considered panel • a diachronic analysis is used to study the evolution of the landscape between two time periods • clusters are ranked with respect to its innovativeness and the similarity of each proposal to its nearest clusters is calculated A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  12. Risk indicator • It is used to infer the « personal risk » of the PI in executing the proposed research • Data source: ERC data & bibliographic DB • Hypothesis: • references serve as one informative source for getting a « bibliometric research profile » • the lower the overlap between two reference profiles, the more risky is the proposal for the PI • Calculation: • comparison between the profile of the proposal and, on one hand, the profile of the past research of the PI and, on the other hand, the profile of the research work in the « home » panel of the proposal A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  13. Pasteuresqueness indicator • It is used to infer the applicability of expected results of the proposal by considering evidence for immediate or intended application • Data source: ERC data & bibliographic DB & patent DB • Pasteuresqueness… What does it mean?! • this neologism comes from the Pasteur’s Quadrant which gives a categorization of scientific issues that seek both fundamental understanding and social benefit • Calculation: • patents granted by PI • information related to industry involvement • self-citations published in journals with prescribed « applicability » A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  14. Interdisciplinarity indicator • It is used to infer self-consistently the presence, in the proposal, of characteristic terminologies which belong to several ERC panels different than the proposal’s « home » panel • Data source: ERC data • Hypothesis: • the higher the occurrence of keywords belonging to different ERC panels, the more interdisciplinary the proposal is considered • the higher this number of different involved ERC panels, the more interdisciplinary the proposal is considered • Calculation: • keyword labelling according to its statistical frequency of occurrence across all panels • assessment of the concentration in the proposal of keywords labelled as belonging to different ERC panels A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  15. Proposal selection function (1/2) We use a statistical discrete-choice model (DCM) to estimate the influence of indicators on the selection probability of a research project proposal to be funded Statistically, we model a dependent variable that takes on the form: i , j= 1, … , n The selection probability of a proposal to be accepted, Pr(Yi = 1), depends on the set of indicators summarized in a matrix X such that where Xi is a set of k observed factors for proposal i , β is the estimated k-by-1 parameter vector reflecting the impact of changes in Xi on the probability Pr(Yi = 1) A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  16. Proposal selection function (2/2) The logistic distribution is chosen for model estimation leading to the empirical model given by: The parameter estimation is based on Maximum-Likelihood procedures. A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  17. Discussion The developped concept… • is ambitious difficult, by employing bibliometric methods, to faithfully represent attributes of frontier research and validly quantify them to evaluate the grant decision • cannot substitute expertise intends to bring us a bibliometric model in which indicators are expected to have a positive effect on the decision probability for ERC grant applications • does not rely on performance measures combination of content analysis and scientometric methods using textual information present in the proposals • can be reused focused on the ERC grant scheme but the concept might be applicable more generally • results in a methodology allowing the grant agency to monitor, from a bibliometric perspective, the peer-review process and providing a basis for its further refinement A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  18. Acknowledgements This work was partially funded by the « Ideas » specific Programme of the EU’s 7th Framework Programme for Research and Technological Development (project reference no. 240765) Project website:http://www.ait.ac.at/dbf A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  19. [dirk.holste; edgar.schiebel; thomas.scherngell; marianne.horlesberger]@ait.ac.at [ivana.roche; dominique.besagni; claire.francois; pascal.cuxac]@inist.fr A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  20. ISSI 2011 - A concept for inferring "Frontier Research" in research project proposals - Durban, South Africa, July 4-7

  21. Timeliness indicator (2/2) A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  22. Proximity indicator (2/2) A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  23. Risk indicator (2/2) A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  24. Pasteuresqueness indicator (2/2) A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

  25. Interdisciplinarity indicator (2/2) A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

More Related