1 / 15

Bureau du F.R.S. – FNRS 23 juin 2009

The role of evaluation and ranking of Universities in the quality culture. Bureau du F.R.S. – FNRS 23 juin 2009. Possible role of a research funding agency. Prof. Véronique Halloin; General Secretary of F.R.S.-FNRS Dr. Pascal Perrin; head of research evaluation department.

mick
Download Presentation

Bureau du F.R.S. – FNRS 23 juin 2009

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The role of evaluation and ranking of Universities in the quality culture Bureau du F.R.S. – FNRS 23 juin 2009 Possible role of a researchfundingagency Prof. Véronique Halloin; General Secretary of F.R.S.-FNRS Dr. Pascal Perrin; head of research evaluation department

  2. 1. Research performance in ranking of universities Ranking: need for a multidimensionalframework (see Ph Vincke; see report of the EC Expert Group on Assessment of University-based Research) addressingusers’ needs and interests(students, scientists, governments, universities, industries) • The quality of a Universitydepends on: • Academic/Faculty performance • Research performance • Quality of education • Library • Supports (administration, IT, infrastructure, …) • Campus culture • Quality of life • …

  3. 1. Research performance in ranking of universities • The present contribution of research performances: • Shangai: 80% research • Nobel prizes: 20% (by staff) • Highlycited staff in 21 disciplines: 20% • Articles published: 20% (Nature, Science) + 20% (citation indexes) • THES: <= 40% research • Academicpeerreview : 40%/2 • Citation per Faculty: 20% • CHE: no weights; 4 fields: biology, chemistry, mathematics, & physics • Size indicator: ouput volume in citation indexes • Perception indicators: citations • Number of often-citedstaff & Nobel prize winners at the university • Europe indicator: # projects in Marie Curie SHS poorly considered

  4. 1. Research performance in ranking of universities Ellen Hazelkorn; Dean of the Graduate Research School, Dublin Institute of Technology, Ireland; International Symposium on University Rankings; University of Leiden, 6-7 February 2009

  5. The role of evaluation and ranking of Universities in the quality culture • Research performance in ranking of universities • The possible role of a researchfundingagency • Challenges of bibliometrics • Research and science mapping • Ranking of researchactors • Pending questions

  6. 2. The possible role of a researchfundingagency • A researchfundingagencyis a universitystakeholder: allocation of money &researchpolicymaking UNIVERSITIES UNIVERSITIES POLITICS, CITIZENS UNIVERSITIES • Funding evaluation is needed for allocation of money • Research evaluation might be outsourced (if bibliometrics)but used to describe the global landscape of research to be funded • Research policy is fed by funding evaluation and research evaluation CAF - 26.06.09 - V3

  7. 2. The possible role of a researchfundingagency Data bases, Researchers, Institutional repositories INPUT research evaluation indicators (economy & society) bibliometrics (research community) studies (retrospective & prospective) Methodology: bibliometrics academic research landscape at international, national & community levels FNRS FRS Aggegation to be avoided!! Multi-dimensional space crashed into linearity, information is definitely lost!! Indicator – metrics Graphs - maps OUTPUT Universities: monitoring Ranking makers: indicators; possible counterweight? Funding agency: research policy OUTCOME

  8. The role of evaluation and ranking of Universities in the quality culture • Research performance in ranking of universities • The possible role of a researchfundingagency • Challenges of bibliometrics • Research and science mapping • Ranking of researchactors • Pending questions

  9. 3. Challenges of bibliometrics • DATA: many problems linked to the collection of data • FIELDS – DISCIPLINES: various classifications, not satisfactory • INDICATORS: which ones are valid/needed to quantify research performance? • MAPPING: to assess potential rather than past performances (metrics) workshop

  10. 3. Challenges of bibliometrics Realibility of data is a precondition for all ranking exercices, the best methodology and soundestmathematicalapproachcannot correct what data collection might have distorsed… • DATA • Variousdatabases: Web of Science (ISI); Scopus; Google scholar • Overlap/coverage: strongdependencywith the scientificfield • (Exemple: for management, onlylessthan 20% of publications are published in the ISI listedjournals ; Harzing & van der Wal, Ethics in science and environmentalpolicy, 2008) • Do not include productions such as art pieces or books, and limited open access publications (Scopus, Google) or conferenceproceedings • institutionalrepositories to capture ouputs! • Choicelinked to whatmeasurementisneeded for • (Google Scholar not suited to macro-levelstudies)

  11. 3. Challenges of bibliometrics • DATA: problems of identification • Spellingvariances of ULB on WoS (1st encoder) • 2001 – 2008: # publications? • First step: 20.545 publications • ULB, univ, univlibre, free univ • IRIBHM, IBMM, ECARES, IIHE, erasme, • Brugmann, St Pierre, Bordet • Second step: - 3.850 “false” publications • Vrije or dutch or flemish or vlaams or UZ or AZ or ziekenhuis • St Luc, St Louis, catholique, ….. • Re-coding errors (last encoder) • Univ. Zimbabwe, DeptThorc.Surg., Leuven,Belgium (Ziekenhuis) • Free Univ. Brussels, Inst Math 1, D-14195 Berlin, Germany (FreieUniversität Berlin) 16.695 Belgium, Brussels Bruxelles Anderlecht, 1070… Exceptco-publications

  12. 3. Challenges • scientific fields mapped along time  evidence of change in networks • each block in a column represents a field (ordered by size) • the height of the block reflects citation flow through the field • apparition of a new scientific field, neuroscience, from the convergence of neurology, psychology, and a part of molecular & cell biology • MAPPING: a field of bibliometrics in very active development (to detect clusters, new disciplines, emergence…)

  13. on-line behaviour of scientists accessing different scientific journals (on-line access of a paper recorded by the servers of scholar web portals) • objective: to visualize the links between disciplines • circles = individual journals • colours = disciplines • lines = scientists navigation between publications • unexpected relations revealed: connection between ecology and architecture • prominent and central position for HSS (while maps based on citations favor natural sciences) Map of science 2007 - 2008 Los Alamos

  14. The role of evaluation and ranking of Universities in the quality culture • Research performance in ranking of universities • The possible role of a researchfundingagency • Challenges of bibliometrics • Pending questions

  15. 6. Pending questions • Research evaluation is usefull • But • Needs to be carefully handled • And • Raises several questions • Impact of evaluation on research activity, strategy (researchers, universities) • Impact of evaluation on research funding (disciplines?) • Risk to miss emergence of teams, themes (bibliometrics assess the past) • Choice of methodology (peer review? bottom-up?) • How to feed the research part of the (probably unavoidable) rankings with indicators developed as a support of research policy • Possible role of networks (Eurohorcs, Unica, …)

More Related