Performance Based Funding of Research in OECD Member States

1. Performance Based Funding of Research in OECD Member States • Marc Luwel • Hercules Foundation • CWTS, Leiden University • Washington DC • October 17, 2012

2. Overview • Highlights / Conclusions from the 2010 OECD Workshop ‘Performance-based Funding for Public Research in Tertiary Education Institutions’ • Flemish approach to performance based (research) funding (PB( R)F) • Background • Built-up • Impact/outcome • Personal comments

3. TERMINOLOGYDifferences between Europe & US • Fundamental / ‘Blue-sky’ research ↔ Basic research • Basic research & development ↔ Applied research

4. PART I 2010 OECD Workshop ‘Performance-based Funding for Public Research in Tertiary Education Institutions’

5. HigherEducation system transformed • Education / formation : from elitist to mass education but highest form in education chain • Knowledge production: fundamental and applied research • Outreach activities : Knowledge dissemination/ valorisation

6. University funding : diversification • Education • Higher educational landscape : strongly entrenched in national (regional) culture • Europe – Implementation of the Bologna Declaration • Tuition fee : for-free to full cost • Public funding of education : inverse relation with tuition fee • (Continental) Europe : von Humbuldt model – all universities are research universities & (fiction ?) equal quality among universities • Research at universities • Increased public funding (knowledge critical for competitiveness, societal challenges) • Revenues from contract research for/in collaboration with firms, public authorities • Outreach activities • ‘Bayh-Dole Act’ type legislation • Valorisation of IPRs : spin-offs, licencing , … • Endowments : Tax code / large cultural differences – Limited in Continental Europe

7. Government’s changing role • From actor to facilitator and regulator • Growing pressure on state treasury • Large societal challenges (aging population, health, security, resources, …) • Public infrastructure (mobility, …) • State of public finances and tax rates – factor in international competitiveness

8. Impact on HEIs • Greater autonomy • Government stops (micro)managing universities • Government lays down regulatory framework • Introduction of lump sum / block funding • Univ :(some) freedom to set up degree programs • Accountability to: • Parliament and Government • Public at large

9. New forms of management • Performance based research funding (PBRF) in HEIs • PBRF: Concept with many faces • 14+ OECD-countries use some form of PBRF • RAE/UK is oldest

10. PBRF : Rationale and assumption • Rationale (implicit / explicit) • Promote excellence / greater selectivity • Concentrate / better manage limited resources • Assumption • (Research) Performance can be made visible/ measured • Multidimensional concept

11. PBRF : Methodology • Peer view: • Generic process of self regulation of the business of science • High credibility • But limitations and biases • “It has been said that democracy is the worst form of government except all the others that have been tried.” • Indicators based on ex-post evaluation • Combination of both → Increasingly more sophisticated but no ideal methodology

12. PBRF : 3 types of indicators • First order • Input, processes, results • Second order • JIF, CWTS-crown index, (family) H-index (!) • Third order • Review panels / ratings (departments) → Indicators aggregated at university level to allocate block funding

13. PBRF : Implementation • Conception and implementation of PBRF often in close collaboration with HEIs • BUT critical comments (from academics) • Validity of the indicators • Their use in funding formula • Intended and unintended consequences → anecdotal and limited evidence-based information (mostly on RAE)

14. PBRF : Effects • More transparent university funding • Universities more accountable • University management response : towards a culture of excellence but also strategic positioning • Improvement of information management systems • A driver for the development of Science of Science and Innovation Policy

15. PART 2 Flemish approach to performance based (research) funding (PB( R)F): a case study

16. Background • Belgium : in a process of devolution of competences from federal government to the regions • 1988 : Flanders became responsible for (higher) education and nearly all competences related to Science and Innovation

17. Background -2 • Beginning of the 90’s : Public R&D expenditures well below EU average • Since 1995 (except for 2009-2010) : increase in public R&D funding • Today : around EU average with 1.85% GDP • Objective : 2020 – Public sector 2% GDP (and 1% private sector)

18. Background – 4 • 5 universities ↔ 6 mio inhabitants • 2 large universities : KU Leuven (37.000 students and 1.500 faculty) and UGent • 2 medium sized universities : VUB (10.000 students, 700 faculty) and UAntwerp • 1 smaller university (only a few degree programs): UHasselt (3.000 students and 204 faculty)

19. Built-up PBRF – starting point Step by step process with Ministers and Universities working closely together Starting point : 1990 • Very limited experience with quantitative science studies in Flanders • Universities had some degree of autonomy and lumpsum funding (block funding) consisting of 2 parts • a baseline funding (dependent on ‘historical’ data) and • an amount based in weighted number of students • (Correction for inflation)

20. Built-up PBRF – 1990-2000 (1) • Mainly an increase of the university research funding • Lump sum: teaching & research • Impression too many degree programmes /the number of students → lack of political will to increasing substantially the lump sum • Additional funding mechanism for blue sky research : separate block grant / lump sum for research (‘BOF’)to allow universities to development their own research strategy

21. Built-up PBRF - 1990-2000 (2) • BOF funding formula : the fraction allocated to each university based on a) block grant, b) number of master students and c) number of PhD’s-degrees • Increasing interest in research evaluation • Having limited in-house knowledge, the universities commissioned from CWTS a series of bibliometric studies on research in natural, life and technical sciences (first study: 1991, UGent) and also in humanities and social sciences (1999, large scale study) • The Flemish Government started using bibliometric tools in evaluation studies (first study commissioned from CWTS: 1994, ICT in Flanders)

22. Built-up PBRF – 2001 – 2005 (1) • Fl. Gov. (2001) funded an S&T Indicator Unit to carry out quantitative studies (prof. K Debackere) • 2003 revision of the regulation on BOF : New funding formula using bibliometric data • Bibliometric data commissioned from S&T Indicators Unit

23. Built-up PBRF – 2001 – 2005 (2) • A (input) : each university’s share in total number of Master degrees, total number of PhD-degrees and in the total lump sum allocated to all the Flemish universities • B (output and visibility): each university’s share • in the total number of SCI publications authored by the Flemish universities; • in the total number of citations to these publications • relative measures to partition yearly a global amount of money! • Weight of part B : starting at 10% evolved to 30%

24. Built-up PBF – 2006-2010 (1) • Fl. Gov. subsidises the Centre for R&D Monitoring (ECOOM); an interuniversity consortium (successor of the S&T Indicators Unit) (http://www.ecoom.be/) • Revision of the BOF funding formula : 2006 (minor) and 2008 (major) • Additional funding mechanism for applied/ basic research : separate lump sum / block funding (‘IOF’) to allow universities to development their own policy for applied research and innovation • Revision of the funding formula for lump sum

25. Built-up PBRF – 2006-2010 (2) • Revision of the BOF funding formula in 2006: PART A • Added as a component (human resources management) : the university’s share in newly appointed faculty from three groups: • Underrepresented gender • Obtained their PhD-degree at another university • With a substantial period working abroad

26. Built-up PBRF – 2006-2010 (3) • Major revision of the BOF funding formula in 2008 – • PART A : each university’s fraction of the total lump sum allocated to the Flemish universities REPLACED BY each university’s share in the total number of academic staff (tenured and non-tenured (incl. PhD-students) at the Flemish universities

27. Built-up PBRF – 2006-2010 (4) • PART B • Publications with weighting factors (incl. JIF) from • SCIE or SSCI with an impact factor • SCIE or SSCI without an impact factor • AHCI • Proceedings STP and SSHP • From 2011 – publications in VABB-SHW • Citations

28. Built-up PBRF – 2006-2010 (5) • VABB-SHW : the Flemish bibliometricbasic database for the Humanities and the Social Sciences • Along the lines of the Norwegian approach • Responsibility of the Flemish universities / technical assistance by ECOOM • Standing Committee selects publications (serial literature, monographs, edited books, …) not covered by WoS • Criteria : starting point is the Governmental regulation but elaborated by Standing Committee ; weighting factors, … • BUT : Quality review by independent panel; its members not working in Flanders ! (First time : 2012) ‘ Trust but verify’ • http://www.ecoom.be/index.php?id=101&L=0

29. Built-up PBRF – 2006-2010 (6) • IOF : block funding for applied research and valorisation • Funding formula based on a university’s share in: • Number of academic staff (BOF definitions) • Number of PhD-degrees • Number of publications and citations (BOF formula) • Revenues from contract research • Revenues from EU Framework Programmes • Number of patent applications (0.5) at and patents granted (1.0) by EPO and USTPO • Number of spin-offs

30. Built-up PBRF – 2006-2010 (7) • Revision of the funding formula for the university’s lump sum / block funding • Component education takes o.a. the number of Ba & Ma degrees delivered by each university • Component research based on BOF funding formula

31. Some effects on performance

32. Trends in use of English and WoS coverage.Percentages of all scholarly journal articles from Flemish (F) and Norwegian (N) universities that were published in English and covered by Web of Science 2005-2009(‘Sivertsen G., Ossenblok T. & Engels T.C.E. (to be published)’.

33. Could be done better • BOF funding formula : by far and large too complex • Allocation model for BOF and IOF and research component of university’s lump sum is based on a university’s share in a sum of money available for all universities ►not taking into account absolute but relative variations ! • Benchmarking is between Flemish universities, should be with top universities world wide, but like with like • Some strategic behaviour ? But the effects of the funding model still badly documented except for the macro trends

34. Questions? Thank YOU !