Evaluation in R&D sphere in Ukraine: Real practice and problems of transition to new standards

1. Evaluation in R&D sphere in Ukraine: Real practice and problems of transition to new standards Igor Yegorov Centre for S&T Potential and Science History Studies, National Academy of Sciences of Ukraine and the Institute of Statistics, State Service of Statistics of Ukraine e-mail: igor_yegorov1@ukr.net Paper presented at the International workshop Science evaluation as a prerequisite for promoting excellence in research 6 November 2012 Chisinau, Moldova

2. Key objects of evaluation in R&D and innovation sphere in Ukraine • Individual researchers • Research organizations (institutes, design bureaus and so on) • Research and innovation projects • R&D programs and R&D sections of development programs

3. Main sources of data Expert opinion (individual experts or expert panels) State statistics (State Service of Statistics of Ukraine) Department of Intellectual Property Institute of S&T and Economic Information Internal statistics of different ministries and agencies (for instance on functioning of technoparks) Information from special surveys and databases Foreign sources (internationally recognized publications, awards and so on)

4. Problems with utilization of official statistics • Rigidity in approaches and difficulties in changes of indicators and questionnaires • Deviation from international standards (examples: system of scientific degrees, calculation of number of publications and so on) • The willingness to serve ‘political’ interests (example: number of research projects and other indicators, which are working ‘in favour’ of old system)

5. Key regulatory acts Law on Scientific and S&T Activities (1992) Law on Innovation Activities (2002) Law on S&T Expertise (2006) Law on Higher Education (1993) Law on State S&T Programs (2004) Total number of laws and other regulatory acts on R&D and innovation exceeds 80. Almost all of them contain clauses on evaluation at different levels.

6. Important features of the Ukrainian research system • High level of centralization in Ukrainian R&D system: administrative power of rectors and directors of the research institutes is ‘excessive’ (including full control over all financial transactions) • System of the evaluation of research results with clear domination of internal publications, not international publications as indicators of success • The procedures of evaluation and selection of R&D projects are often not transparent and fair for potential participants.

7. Recent changes in R&D and Innovation statistics • Introduction of the CIS-type survey in Ukraine (2009-2010, data for 2008-2009). New survey is expected in 2013 • Switching (partial) to international classification of sectors - since 2009 • Implementation of new NACE-type classification (new version of KVED) – since 2012 • Implementation of indicators of international publications in some universities and research institutes

8. Publications of Ukrainian authors in scientific journals in 1996-2005, tous. (Thomson Reuters)

9. Number of scientific publications according to the State Statistical Service of Ukraine, 1991-2011, thous.

10. Evaluation of individual researchers • Method: Panels of experts from the same institute (as a rule: at the level of department and at the level of the institute) discuss the work of researcher and vote openly for the decision. This decision has to be confirmed by the scientific council of the institute. • Periodicity: Every five years • Indicators are used indirectly (number of publications, participation in conferences and seminars, awards, preparation of the thesis (if any), participation in a work of different scientific councils and commissions and so on)

11. Evaluation of research organizations A) Panel of external experts from the same research area prepares a report on activities (every four-six years). Indicators include number of publications, number of patents and so on. B) Evaluation with implementation of the special metrics (2012) Evaluation of potential (within the projects or programs) varies according to the goals of evaluation (academic programs, STCU and so on)

12. Evaluation with implementation of the special metrics (2012) ‘Objective’ indicators (60% of final assessment) Expert evaluation (40% of final assessment) ‘Objective’ indicators: a) personnel (5 indicators) b) financial situation and research equipment (8 indicators) c) research in priority areas (3 indicators) d) publication activities (2 indicators) e) innovation activities (4 indicators) f) international activities (3 indicators) g)’visibility’ at the international level (2 indicators)

13. Example1: ’Visibility’ at the international level Number of references on the research organization in the Google searching system Number of publications in domestic and international scholar journals with high impact-factor per one employee of the research organization

14. Example 2: Publication activities Number of publications in domestic scholar journals per employee of research organization Number of publications in foreign scholar journals from international databases of scientific publications per employee of research organization (data bases include information from Scopus and some other databases on medical publications)

15. Expert evaluation of research organization Level of research activities Relevance to the national priorities Potential for further development Indicators, which reflect specific features of research area Note: all indicators have their own weights, depending on the type of organization

16. Evaluation of research projects Ex- ante evaluation Evaluation of the process (if the project has long duration or it is divided into several stages) Ex-post evaluation Different types of indicators are used, depending on the type of the project Special forms are used by different agencies and ministries, not less than two experts take part in the evaluation procedure Key idea is to reach the goal of the project (must include, if possible, description of quantitative indicators, which have to be achieved) within certain time and financial constraints

17. Example of indicators for selection of the project for the Ministry of Economic Development and Trade (October 2012) Proposed price (60% of the final assessment) Other indicators (40%): a) Number of Doctors and Candidates of Sciences in the research team (30 points, if more than 7: 3D+4C) b)Number of articles in scholar journals (30 points, if more than 10) on the topic of the project c) Monographs on the subject (15points per monograph) d) Participation in similar projects with the Ministry during the last 3 years (10 points per project)

18. Evaluation of the programs In accordance with to the normative acts and existing laws at the state and ministry level Problems: Lack of quantitative criteria and utilization of ‘artificially constructed criteria’ Non-transparent expert procedures (as a rule, no foreigners take part in the evaluation, excluding Key Laboratory initiative and some joint programs with foreign partners) Lack of money for financing of R&D programs, which is used as an argument for justification of the level of results achieved within the program

19. Conclusion Ukraine has not yet created relevant institutions and proper instruments necessary for effective transformation of R&D sector, including effective system of research assessment. Internal practice of evaluation differs substantially from the international one. Unfortunately, every year the possibilities for the implementation of effective transformation policy are shrinking, as the number of researchers is declining and the research centres are losing their ability to conduct research.

20. Conclusion With time running out, remaining research institutions will face more problems with the creative capabilities needed to compete internationally Reforms of evaluation of R&D sector have to be an integral part of general transformation of R&D system EU could contribute to the reforms by stimulating introduction of new standards of R&D project assessment and implementation

21. Thank you for your attention !