Monitoring R&D as a Methodological Tool for Impact Studies

1. Monitoring R&D as a Methodological Tool for Impact Studies Stig Slipersæter Norwegian Institute for Studies in Research and Higher Education Centre for Innovation Research

2. Background • Lack of international standards for output and impact indicators • Small steps to integrate outcome data relevant for impact studies into Norway's national R&D statistics

3. Monitoring R&D • An example on monitoring • An example on use of data • Lessons learned

4. Example: Research institutes • 65 independent research institutes • Annual budget about 1 billion US $ • Approximately 7000 employees • Monitoring scheme run by NIFU on behalf of the Research Council of Norway • Purpose: Accountability and policy analysis • Integrated with regular R&D statistics

5. Input to research activities • Financial resources split on: • Government funds • Normal research contracts: • Programme allowances from research council • Public sources, split on various ministries and their agencies, local and regional authorities etc. • Domestic industry, split into three categories • Contracts from abroad, split into seven categories • Detailed data on human capital

6. Output from research activites • Indicators for output: • Enlarged set of variables for printed dissemination • Articles in scientific journals • Different kinds of reports • Books and chapters in books • Articles in magazines of professional associations • Popularization • Patents applied for and granted • Spin-offs • Partcipatons at conferences, seminars etc

7. Indicators for diffusion • In- and outgoing mobility • Shared positions (with universities, industry and others) • Cooperative research projects • Supervising of students • Visiting students • Visiting scholars (in- and outgoing)

8. Advantages of the scheme • Details on income • An extended set of variables on outcome • Variables on different diffusion processes

9. Utilising the institute monitor • Users evaluation of applied social research at 31 research institutes • Analysis of financial data showed: • 12 institutes serving regional and local authorities • 19 institutes serving national authorities and international organisations • How is differences in customer base affecting the outputs of the institutes?

10. Differences in written output

11. What purposes were R&D used for? User survey shows two main categories of use: • Instrumental: New facts and policy support • Conceptual: New ideas, concepts, methods and ways of understanding Output from national and academically oriented institutes considered being of largest instrumental value for users

12. Lessons learned for indicators 1 • There are many ways of transferring scientific results: Broad set of transfer mechanisms should be included • Use of output is far from obvious: Impact can come in unexpected ways Ask the user of scientific results what is important before making the indicator.

13. Lessons learned for indicators 2 When setting up a system of indicators, consider: • A broad set of types of output • Include at least these groups in the process: • Producers of output (i.e. scientists) and the ones reporting (i.e. administrators) • Users of output • Users of indicators • The importance of time series