Forskningsrådsevalueringen - metoder, utfordringer, resultat

1. Forskningsrådsevalueringen- metoder, utfordringer, resultat Erik Arnold Technopolis EVA September 5 2002

2. Panel members … Pieter Drenth, Amsterdam/ALLEA Anders Flodström, KTH Jecqueline Godet, CNRS Philippe Laredo, CSI Ecole des Mines Ben Martin, SPRU Frieder Meyer-Krahmer, FhG-ISI Erkki Ormala, Nokia Arie Rip, Twente (chair) The research team ... Erik Arnold, James Stroyan, Paul Simmonds, Sarah Teather, Ben Thuriaux, Alina Östling, Technopolis (leader) Johan Hauknes, Marianne Broch, Per Koch, Heidi Wiig, STEP Stefan Kuhlmann, Sybille Hinze, FhG-ISI Egil Kallerud, Randi Søgnen, Liv Langfeldt, Magnus Gulbrandsen, NIFU Barend van der Meulen, Frank van der Most, Twente The evaluation team

3. Contents … • Why have research councils? • Was the research council reform a good idea? • Is RCN doing a good job? • What are the research policy challenges? • What next?

4. RCN’s mandate (§2) - and that’s just the written bit • RCN shall develop its research policy and administer grants for research, based on guidelines drawn up by the government and parliament. RCN shall further provide advice as a basis for the development of the government’s general research policy. RCN shall • Initiate research, which promotes the development of Norwegian industry and society • Contribute to the development of knowledge of humanity’s common problems, especially problems related to the environment and development • Support researcher-initiated research, among other things by acting as a supplementary source of funding for the institutions, which perform basic research • Work to achieve a good balance between long-term basic research and use-oriented research • Work to ensure that user concerns are taken into account in applied research • Promote international research co-operation • Promote quality, efficiency and relevance in the research system • Initiate and use the results of evaluations of research and research institutions • Take strategic responsibility for the research institute sector in Norway • Promote the exploitation of the results of research by the state, industry and the general public

5. Our mandate (leaving out the details …) • The evaluation of the Research Council of Norway is to cover the period from its establishment on 1 January 1993 to 31 December 2000, both dates inclusive. It shall give an overall evaluation of the Research Council in the light of the principal objectives laid down in report to the Storting St. meld. nr. 43 (1991-92), recommendation to the Storting Innst. S. nr. 231 (1991-92), and the statutes (articles of association) of the Research Council. • The evaluation shall analyse the connection between the Research Council’s framework conditions, organisation and instruments, and the objectives laid down for its activities. Assessments shall be empirically grounded, among other things in the experiences of central groups of actors in the Ministries, research institutions, the commercial sector and the Research Council itself. In the light of this analysis, the evaluation shall consider what the Research Council’s framework conditions should be, how the Research Council ought to be organised, and what steps the Council itself should take, so that one may be as well equipped as possible to meet the future challenges confronting Norwegian research.

6. Structuring Observing Judging Janus is the god of evaluation. He is a judge, not a researcher Analysing Choosing evaluation questions, effects to be evaluated, defining criteria Defining indicators or suitable alternatives Defining the field of observation Collecting data in the field Analysis of data Analysis of causality and attribution of effects Judging effects individually Formulating a synthetic judgement

7. Given that evaluation tools are poor, we look for convergence Railway Research Group, KTH

8. Detailed mapping of work steps onto terms of reference

9. Framework Conditions Financial environment; taxation and incentives; propensity to innovation and entrepreneurship; mobility Demand Consumers (final demand) Producers (intermediate demand) Education and Research System PoliticalSystem Industrial System Large companies Professional education, training Government Intermediaries Institutes; Brokers Higher education and research Governance Mature SMEs New, technology- based firms Public sector research RTD policies Infrastructure IPR & information Innovation & business support Banking, venture capital Standards and norms Our evaluation leant heavily on domain theory. We made this explicit, focusing on the idea of innovation systems..

10. … and the Mode 1/2 discussion

11. Drivers Growth in Mode 2 production Changes in the nature of technologies ‘Hyphen-technologies’ Dematerialisation Appropriation via intellectual property Changed industrial organisation of knowledge production Globalisation De-integration Acceleration: concurrent science Changed social contract Increased relevance The state as a growing user Trends Global knowledge markets Emergence of superuniversities Reorganisation of the Research Institute sector End of the ‘3-hump model’ Industrial ‘observatories’ and new PPPs at the university/industry interface IPR as a constraint as well as an income generator In the judging phase, we also tried to take account of current changes - some of which go beyond theory

12. Results. The research council reform was a good idea in 1993. It still is • The old structure was not only messy, but was unable to cope with the newer policy challenges, such as the Main Target Areas • The issues in the Norwegian research system are about adequate research funding and the role of research in economic and social development development • Changes in the way knowledge is produced and used require both holism and diversity in research policy • Some other countries are moving towards more integrated solutions - Denmark, England, Finland …

13. RCN NAVF NTNF NLVF NORAS NFFR IE NT BF MH KS MU NMF RSF RMF RHF RNF History matters. The merger was a triumph of political turf over policy vision. The forces that destroyed the action areas won again

14. Executive Board Government Director General Division Boards (6) Strategy, Admin Division Directors (6) Programme & Discipline Boards Ministries Divisional Staff RCN’s three ‘steering levels’ became a battlefield in the early days and continue to provide means to fragment the Council

15. The organisation and its context lead to rigidities • Vertical links are weak • No-one is present at more than one level • Links to overall strategy are hard to forge • Higher levels tend to bless what goes on below, rather than engaging strategically • Structures are locked in, owing to their tight and detailed links with external funders • Little or no slack is available for change-agent or arena roles • There is quite a lot of administrative diversity, but some of it is in the wrong places

16. RCN does a solid job where it can, but has more responsibility than authority • Making progress towards its six (summarised) goals • Administration is cost-effective • Starvation rations do not encourage dynamism • The sector principle reinforces the barriers between the Divisions • RCN needs to change its character from planning-system to research and innovation policy arena • cp FUGE, Demo 2000, OG21 ...

17. Goal fulfilment is almost surprisingly good. Testimony to the power of muddling through? • RCN shall produce useful national and sectoral research policy advice to the government, based on an holistic national perspective • RCN shall fund research to meet social and industrial needs, taking account of users’ needs and promoting the uptake of results • RCN shall fund the high-quality basic and applied research needed in the national system of knowledge production, seeking to integrate the two as far as is appropriate while securing the place of basic research • RCN is tasked with strategic responsibility for the research institute sector in Norway • RCN is tasked with promoting the interaction of Norwegian knowledge production with the international knowledge production system • RCN shall use appropriate and efficient processes (including evaluation) and organisational structures in performing its tasks Improving Yes, but … Q/A and procedures in place for both. Integration? Well done. Consequences? Exploiting EU. Integration into world research Use of diversity should be more strategic

18. RCN’s ability to act as an holistic council reflects its funding

19. Norwegian research policy faces major challenges • The ‘value-creation gap’ (cp Norman, Reve) • Raising national R&D/GDP, especially through the creation of new industry • Move from the so-called ‘three-hump model’ with a strict division of labour among universities, institutes and ‘users’ to one where different modes of knowledge production interpenetrate … • … and to take the consequences by extending and modernising the roles of the institutions • Find an appropriate balance between the number of researchers and the amount of ‘bottom-up’ funding available in the system. This is really a question about funding, not about basic research

20. R&D/GDP

21. Governments have cut SND and walked away from innovation policy. IE’s budget fell 16% 93-00, and has been cut by another 150 MNOK

22. Norway needs a stronger Innovation Agency function, cp TEKES. Getting the needed political understanding is hard, in a rich country

23. Strategy and Foresight Response mode/ ‘free’ research Strategic Programmes, Institutes and Infrastructures Strategic Innovation Agency SND Thematic R&D areas Absorptive Capacity What are the functions that would need to be encompassed in an holistic research council?

24. Structural issues Cultures Contested nature of the social contract Sectoral principle and organisation Few convincing examples Integrated approaches Change agencies Capture of principal-agent systems by client communities Education Ministry Industry Ministry Research Council Innovation Agency Industrial Community The struggle to integrate disparate cultures will continue to be hard Research Community

25. After the evaluation, political path-dependency re-asserts itself • Those who already ‘knew’ the answer before we started work carried on telling everyone what they already ‘knew’ • The ‘basic research’ faction still wants NAVF back. (Perhaps the collective amnesia of this faction is liked with its average age…!) • Industry’s representatives want NTNF back • My good friend Hans Skoie’e campaign against a single council continues. It’s about time he was awarded a medal - at least for persistence! • The education minister seems to have blocked the idea of raising research and innovation to the national (prime ministerial) level • Politically understandable … • … but this has important negative symbolic value, cp Finland, Sweden, Ireland • The idea of continuing with a single council is accepted … but what that actually means remains fluid