Some shortcomings of current ranking systems

1. The importance of national ranking of universities and research institutions: Experience of “small” countries MajaJokić, KrešimirZauder & Marina Matešić Institute for social research in Zagreb, Agency for Science and Higher Education, Croatia

2. Some shortcomings of currentranking systems Current university and research institution ranking systems (ARWU, THE, SIR, Leiden, Ranking Web of Universities ): use data from commercial citation databases WoS and Scopus, e.g. Affiliation the main problem is in unique identification of institutions, recognition the institutions, e.g. University of Zagreb has 29 different constituent units, autonomous faculties with own constituent units such as departments; additionally, English and Croatian affiliation variants all Croatian universities have as their constituent parts social sciences, arts and humanities faculties - which are not equal with STM in current ranking, especially in citations count

3. Some shortcomings of current ranking systems The Leiden Ranking: offers statistics only at the level of science as a whole does not take into account arts and humanities publications does not take into account non-English language publications ARWU: scientific periphery countries have small chances of being included in ARWU the weights of the six indicators are arbitrary and questionable THE: Our key complaint is that Thomson Reuters has the choice of the scientists to be included in survey. What are the criteria?

4. Representation of 11 Eastern Europe EU countries in the current university ranking systems The EU countries of Eastern Europe and Croatia are represented in ARWU with a total ratio of 1.2% included universities: Hungary (2), Poland (2), Slovenia (1) and Czech Republic (1) According to Leiden Ranking, that ratio is 1.6% and includes countries: Poland (3), Hungary (2), Czech Republic (1), Slovenia (1) and Croatia (1). In THE ranking, which includes only 400 universities, the ratio is 1%: Poland (2), Czech Republic (1) and Estonia (1)) Only SIR included all Eastern Europe EU countries and Croatia. According to SIR data for 2011, among 3042 institutions the mentioned countries were present with 136 institutions (4.47%)

5. National approach to university and research institutes ranking Based on analysis of strengths and shortcomings of current ranking systems, a more wholesome approach could be developed. To keep the ranking from becoming pure competition! core scientific values, such as the excellence in science and education is the driving force for positive intellectual, social and technological change, need to be incorporated in the very ranking system. The national approach could include : independent data collection from multiple relevant national and international sources, and take into account the specificities of individual universities especially as relating to social sciences and humanities

6. National approach to university and research institutes ranking Data collections/databases: The data should represent reliable information about productivity (author books, editor books, book chapters, articles in national and international journals, conference proceedings, patents, etc.) which is not done by existing ranking systems. In addition to international sources (e.g. WoS, Scopus) the national bibliography or more specialised bibliographies and other sources (e.g. institutional repositories) should be used. Additional sources, such as the national citation indices - to support the productivity analysis while observing the differences between sciences, social sciences and humanities.

7. National approach to university and research institutes ranking To support interpretation, the methodological approach should include some sophisticated indicators for productivity and impact used by the global rankings (e.g. Leiden Ranking iSCImago Institution Rankings) or develop own indicators based on current positive examples. An addition to bibliometric, national approach needs to be enhanced with such dimensions as human potential and material resources. These dimensions should be developed further to be fully usable, e.g. the indicators used by the THE ranking or U-Multirank could be used, or other wholesome indicators including university activity, specificities and national and international recognisability could be developed.

8. National approach to university and research institutes ranking A significant concept missing from most ranking systems, which is relevant for both universities and institutes, is the concept of "outcome”. Defining that outcome in a measurable way is no easy task with no wholesome methodology readily available. Outcome is also more usable on national than international level as it will be easier to both define and estimate in local context and should thus be the concern for national rankings of scientific institutions.

9. National approach to university and research institutes ranking Croatian Agency for Science and Higher Education (AZVO) developed the criteria (including outcome indicators) for purposes of reaccreditation of scientific institutions: impact on development/transfer of research results in the economy transfer of research results in higher education research results as the basis for introduction of social change international impact - impact of research results on international recognition of Croatia Similar view of the problem is expressed by U-Multirank through success in knowledge transfer (such as partnerships with business and start-ups), and contribution to regional growth.

10. Conclusion Universities and other research institutions should develop own strategies and criteria which will make them recognizable in the relevant area. Whether they choose local focus or broader international academic community, a usage of lingua franca (i.e. English) is necessary to be visible and comparable across national borders including scientist and student mobility. Multidimensional ranking (output and outcome indicators, human potential and material resources) allows taking into account the specificities of universities and research institutions.

11. Conclusion Although rankings seem to not only be in trend but also an important part of current organization of science, we agree with colleagues (Adler & Harzing 2009) that the most crucial question is “What is our scholarship actually contributing?” and this question should guide the ranking systems rather than vice versa.

12. Thankyou 