SAI’s role in development and use of key indicators for R&D evaluation

1. SAI’s role in development and use of key indicators for R&D evaluation Timo Oksanen 3.4.2012

2. About SAI’s role in indicator development • Depending on the national mandates, the SAI’s role can be active or passive – or something in between • However, an active role in indicator development can endanger SAI’s independency and objectiveness • The NAO of Finland has not participated in Finland’s KNI development • Therefore, we have kept an outsider’s view to Finnish KNI-system

3. Findicator: The place to find up-to-date information on: • demographic developments in Finland • international crisis management • public expenditure • obesity among Finns • income differences • fish catches Available to all at www.findicator.fi

4. Findicator is a comprehensive databank • Includes approximately 100 indicators of social progress • Up-to-date and relevant information on important societal issues • Indicators selected in consultation with user groups and data providers • Grouped thematically and by policy issue in line with Government Programme

5. R&D key indicators • In the Finnish KNI collection (findicator.fi), there are only two indicators that measure R&D outcome and effort: • Time series of patent applications • Time series of R&D expenditures, by sector

6. Patent applications in Finland, 1972 - 2010

7. R&D expenditures by sector, 1971 - 2010

8. On the basis of our R&D audits it can be said that… 1. A lot of work has been done (Slides 7-9) http://www.tekes.fi/u/Better_results_more_value.pdf 2. Some things are just happening (slide 10) 3. A lot of things has still to be done….(slides 11-18). To put it shortly: ”Systemic change is a highly topical issue both in Finland and internationally. Climate change, aging of the population, advancing sustainable development as well as structural transformations in the economy and the related need to find new growth areas are current grand challenges. Responding to these challenges is not possible through individual small-scale reforms but calls for system-level changes”. http://www.tekes.fi/u/systeemisen_muutoksen_haasteet.pdf

9. Indicators related to economy and economic renewal 9

10. Indicators related to well-being 10

11. Indicators related to knowledge, education and culture 11

12. Levels of the implementation Indicators • Visual level = Graphics and visualisation of indicators • Activity level = e.g. availability of the information behind the indicators • Technical level = platform and connections, technical implementation of administration • Definition level = Detailed indicator definition • Data level = sourcing and updating of data • Administrative level covers all the levels above defining the responsibilities and funding Visual level Activity level Technical level Administrativelevel Definition level Data level Data 12

13. Concluding remarks (1) 13 There is a clear need for discussion on research and innovation and the related societal development and challenges The framework should allow for the decision makers to quickly assess the Finnish research and innovation development against the main international trends and key benchmark nations The question of phenomena selection contains a political aspect The set of indicators (including the impacts chains) needs continuous update on the basis of newest knowledge As well, it should adapt to the changes in societal priorities and goals International cooperation is needed to make existing and forthcoming efforts comparable More knowledge needs to be gained on analyzing the impacts of research and innovation on well-being and the society

14. Concluding remarks (2) 14 Factors that need to be considered particularly when setting target values to indicators include: Accumulation factor: how much emphasis is put on the current level of standing, what has been achieved and built over the past years or decades Improvement and development factor: as compared to relation to peers or selected benchmarks Systemic factor: how much emphasis is given to the existence and functioning of the research and innovation system (or ecosystem) as a whole, instead of its elements alone. Global factor: how much emphasis is given to Finnish position and impact in global value chains, EU frameworks or in addressing global challenges, as compared to our success and impact within the country Relevance factor: how well is the current research and innovation activity tuned for, prepared or adaptive to address the societal challenges, such as ageing of population, economic recession, etc.

15. A step to wider scope: where are we now?(Fred Gault: Social impacts of the development of the STI Indicators, 2011) Indicator development is a dynamic activity. The OECD Innovation Strategy included a Measurement Agenda which is now being implemented. It includes intentions to: i. Improve the measurement of broader innovation and its link to macro-economic performance; ii. Invest in high-quality and more comprehensive data infrastructure to measure the determinants and impacts of innovation; iii. Recognize the role of innovation in the public sector and promote its measurement; iv. Promote the design of new statistical methods and interdisciplinary approaches to data collection; and, v. Promote the measurement of social goals and social impacts of innovation”.

16. Special issues…(1) Policy impacts (Gault, 2011) • As the indicators expand and policy makers recognize that innovation is not an isolated event, more attention is being given to the framework conditions and the policy mix that helps the system to work better. As more micro data analysis is done, the important result that the propensity to innovate in firms is higher than the propensity to do R&D will have more influence on policy.

17. Special issues…(2) Funding of Finnish universities 2012 • The model comprises in three main parts: education, research, and other education and science policy objectives. A total of 75% of the core funding will be allocated on the basis of a formula for education and research, of which 41% is based on educational factors and 34% on research factors. The remaining 25% of the core funding is based on education and science policy objectives. Education-based funding criteria (as % of core funding): Master’s degrees awarded by the university (15 %), Bachelor’s degrees (9 %), the number of students completing a minimum of 55 credits (11 %, of which 3 % based on data produced by the student feedback system from 2015), credits completed in open university and non-degree studies (2 %), the number of degrees awarded to foreigners by the university (1 %), incoming and outgoing international student exchanges in the university (2 %) and the number of job-holding graduates (1 %).

18. … universities…(3). • The research-based funding criteria: doctoral degrees awarded (9 %), publications (13 %, of which 10 % international refereed publications and 3% other scientific publications - from 2015 the number of the Finnish Publication Forum classification levels 2 and 3 publications instead of international refereed publications and the number of level 1 publications instead of other scientific publications); competed research funding (9 %, of which 3 % international competed research funding and 6% other competed research funding); doctoral degrees awarded to foreigners (1 %); and foreign teaching and research personnel (2 %).

19. SAI’s role in development…? • Attribution/contribution/independence/capacity-building/resources: • So far the NAO of Finland has not participated in Finland’s KNI development • The Future: it is difficult or even impossible to separate the impact of SAI from other inputs and activities.

20. i. besserwisser? ii. Facilitator? iii. Risk Manager? iv. Measurement Technician? v. Capacity Builder? (workshops 2011, brainstorms 2012) vi. Problem solver? vi. Link between R&D actors and desicion making? from measurement of results for contributor of the knowledge management of the whole R&D from peaces to the whole (the life cycle of the knowledge and knowledge management) from analyses to synthesises, from outputs to processes and capabilities of actors from mechanical synthesises to real problem solving …by co-operation with the R&D actors The Role of the SAI with the STI-indicators?

21. The last steps at 2012: Findicators: An official network to coordinate the development of indicators (1) • to make propositions about developing the services and infrastructures of indicators (and knowledge management) in the government of Finland • to follow-up international development in the field • to co-operate with other developmental projects in this field, etc.

22. The spesific lines of developing R&D indicators in near future 2012 (2) • http://www.tekes.fi/u/Better_results_more_value.pdf • The Academy of Finland and Tekes will report until September 2012 to the Research and Innovation Council of Finland about the main lines of implementing the Framework (slides 9-11 above) (resources, administrative responsibilities and task, pilots (focusing single indicators) etc. • are politics able to utilise this kind of information? • how? when? why? where? how much it costs?