BIBLIOMETRICS – USE AND LIMITATIONS

1. BIBLIOMETRICS –USE AND LIMITATIONS Wolfgang Glänzel KU Leuven, Belgium ISPR, HAS, Hungary

2. STRUCTURE OF THE PRESENTATION • Introduction • Scope and structure of bibliometrics • Data sources of bibliometric research • “Perspective shift” in use of bibliometrics • Bibliometrics as a tool in research evaluation • Bibliometric research and services at the Steunpunt O&O Statistieken Bibliometrics. Use and limitations

3. 1. INTRODUCTION Bibliometrics. Use and limitations

4. 1. Introduction • What is bibliometrics? • The terms bibliometrics and scientometrics were introduced by Pritchard and Nalimov & Mulchenko in 1969, and are nowadays used almost synonymously. • Pritchard • “bibliometrics isthe application of mathematical and statistical methods to books and other media of communication”. • Nalimov & Mulchenko • Scientometrics is“the application of those quantitative methods which are dealing with the analysis of science viewed as an information process”. Bibliometrics. Use and limitations

5. 1. Introduction • Gorkova (1988) • Informetrics stands for a more general subfield of information science dealing withmathematical-statistical analysis of communication processes in science. • A quite broad definition of the scope of bibliometrics given by Glänzel& Schoepflin (1994)includes “all quantitative aspects and models of science communication, storage, dissemination and retrieval of scientific information”. Bibliometrics. Use and limitations

6. 1. Introduction • These notions also integrate many presently existing orientations such as applications to science policy, library science, technometrics, Web analyses and information retrieval. Prominent methodological issues are related to • subject delineation • subject evolution and structure • publication activity/scientific productivity • scientific collaboration, co-authorship networks • citation impact and citation networks • bibliographic coupling, co-citation analysis • computational linguistics; text mining, co-word analysis • visualisation techniques • datamatching • science-technology linkage Bibliometrics. Use and limitations

7. 1. Introduction • Common misbeliefs on bibliometrics • Main task of bibliometrics should be the expeditious issuing of “prompt” and “comprehensible” indicators for science policy and research management. • Bibliometrics is only an auxiliary tool, research on methodology is unnecessary. Instead, bibliometricians should elaborate guidelines explaining the use of their indicators. • Bibliometrics might be reduced to simple counting activities in order to replace/supplement qualitative assessment by quantitative indicators and to set publication output off against funding. Bibliometrics. Use and limitations

8. 1. Introduction • Facts about bibliometrics • Bibliometrics is a powerful, multifaceted endeavour encompassing sub-areas such as • Structural scientometrics came up with results like the re-mapping of the epistemological structure of science. • Dynamic scientometrics constructed sophisticated models of scientific growth, obsolescence, citation processes, etc. • Evaluative scientometrics developed arrays of indicators to be used to characterise research performance at different levels of aggregation and come up with benchmarking solutions. Bibliometrics. Use and limitations

9. 2. SCOPE AND STRUCTURE OF BIBLIOMETRICS Bibliometrics. Use and limitations

10. 2. Structure of Bibliometrics • Three “components” of present-day bibliometrics according to its three main target-groups • Bibliometrics for bibliometricians • The domain of bibliometric “basic research”. • Bibliometrics for scientific disciplines • A large but also the most diverse interest-group. Due to the scientists’ primary scientific orientation, their interests are strongly related to their speciality. • Bibliometrics for science policy and research management • The most important field of application. Here the assessment and comparative analysis of research performance are in the foreground. Bibliometrics. Use and limitations

11. applied Scientometrics basic 2. Structure of Bibliometrics Links of bibliometrics with related research fields and application services Science policy Scientific information Research management Librarianship Services for Research in Economics Sociology of science History of science Library and Information Science Life sciences Informetrics Mathematics/Physics Webometrics Bibliometrics. Use and limitations

12. 3. DATA SOURCES OF BIBLIOMETRIC RESEARCH Bibliometrics. Use and limitations

13. 3. Sources of Bibliometrics Data sources of bibliometrics are bibliographies and bibliographicdatabases. Large scale analyses can only be based on bibliographic databases. Despite objections to journal coverage and data processing policy of the ISI, the databases of the Institute for Scientific Information (Thomson - ISI), above all, the Science Citation Index (Expanded) have become the most generally accepted data source. Recently several countries have developed their own national citation indexes which are also used (as supplementary) data sources. Bibliometrics. Use and limitations

14. 3. Sources of Bibliometrics • Bibliographic databases are generally designed for the retrieval of information. However, their versatility is often limited in bibliometric use. • Originally, bibliometrics was developed for studying communication in basic research. Present-day tasks: • How to extend bibliometrics towards application to engineering, social sciences and humanities? • How to measure and evaluate aspects of technologylinkage? • Since non-periodicals are important conveyors of information in engineering, social sciences and humanities, journal based data-sources are accepted by scientists only with certain reservations. Bibliometrics. Use and limitations

15. 4. “PERSPECTIVE SHIFT” IN USE OF BIBLIOMETRICS Bibliometrics. Use and limitations

16. 4. Perspective Shift • Bibliometrics deals with the quantitative analysis of the (social) network of scientific communication. • From the historical perspective, information scientists and bibliometricians proceeded from the same approach. First citation analyses were designed to aid the decision which periodicals should best be purchased by small libraries. Many tools and indicators developed by the ISI were designed for use in scientific information, information retrieval and libraries. • The application to science policy has brought a new perspective, and resulted in re-interpretation of bibliometric conceptions. Bibliometrics. Use and limitations

17. 4. Perspective Shift Example: The notion of citations in information science, bibliometrics and science policy Citations became a widely used measure of the impact of scientific publications, but the notion of what citations actually express sheds light on different aspects. These different views might results in conflicts caused by re-interpretation and “perspective shift”. Bibliometrics. Use and limitations

18. 4. Perspective Shift • Some examples: • According to Merton, citations are intimately connected with the reward system of science. • Cozzens: “Citation is only secondarily a reward system. Primarily, it is rhetorical-part of persuasively arguing for the knowledge claims of the citing document.” • Cronin: Citations are "frozen footprints in the landscape of scholarly achievement … which bear witness to the passage of ideas“. • Glänzel and Schoepflin: Citations give “a formalised account of the information use and can be taken as a strong indicator of reception at this level.” Bibliometrics. Use and limitations

19. 4. Perspective Shift The process of re-interpreting the notion of citation and its consequences interpretation citation Bibliometrics/Information science Information use uncitedness: unused information frequent cite: good reception self-cite: part of scient. communication repercussion (possible distortion of citation behaviour) re-interpretation uncitedness: low quality frequent cite: high quality self-cite: distortion of impact Reward system (quality measure) Research evaluation/Science policy Bibliometrics. Use and limitations

20. 5. BIBLIOMETRICS AS A TOOL IN RESEARCH EVALUATION Bibliometrics. Use and limitations

21. 5. Research Evaluation • Present-day use of bibliometrics • The evaluation of scientific research performance is the most important application of present-days bibliometrics. • Growing, often controversial, policy interest to use bibliometric techniques as output measures of research productivity and efficiency. • Thus bibliometrics evolved to a standard tool in science policy and research management. • Bibliometric components are used, among others, in national science reports, bibliometric indicators appear in the daily press, benchmarking results are used for research management and for allocation of research funding. Bibliometrics. Use and limitations

22. 5. Research Evaluation • What can bibliometrics not be responsible for? • Bibliometrics can be used to develop and provide tools to be applied to research evaluation, but is not designed to evaluate research results. • Bibliometrics does not aim at replacing qualitative methods by quantitative approaches. • Consequently, bibliometrics is not designed to correct or even substitute peer reviews or evaluation by experts but qualitative and quantitative methods in science studies should complement each other. Bibliometrics. Use and limitations

23. 5. Research Evaluation • Distorted behaviour based on policy use and misuse of bibliometric data • One issue concerns the changes in the publication, citation and collaboration behaviour of scientists (both positive and negative) that the consistent policy use of bibliometric indicators might potentially induce. • Uninformed use and misuse are not always beyond the responsibility of bibliometricians. Unfortunately, bibliometricians do not always resist the temptation to follow popular, even populist, trends in order to meet the expectations of the customers. • Any kind of uninformed use or misuse of bibliometric results involves the danger of bringing bibliometric research itself into disrepute. Bibliometrics. Use and limitations

24. 5. Research Evaluation • The problem of inappropriate use ranges from uninformed use, over selecting and collecting ‘most advantageous’ indicators to the obvious and deliberate misuse of data. • Uninformed use • incorrect presentation, interpretation of bibliometric indicators or their use in an inappropriate context caused by insufficient knowledge of methodology, background and data sources • Misuse • intentionally incorrect presentation, interpretation of bibliometric indicators or their deliberate use in inappropriate context • tendentious application of biases or tendentious choice of (incompatible) indicators Bibliometrics. Use and limitations

25. 5. Research Evaluation • But even correct use might have undesired consequences • Author self-citations are shown in an unfavourable light. Authors might thus feel urged avoiding self-citations – a clear intervention into the mechanism of scientific communication. • Indirect repercussionscan be observed when bibliometric tools are used in decision-making and the scientific community recognises the feedback in terms of their funding. • Butler (2004)showed on the example of Australia what might happen when funding is linked to publication counts. She found that this has stimulated an increased publication activity in the lower-impact journals. Bibliometrics. Use and limitations

26. Schematic visualisation of the feedback of policy use of bibliometrics on the scientific community 5. Research Evaluation Bibliometrics. Use and limitations

27. 5. Research Evaluation • Possible positive effects • Scientists might recognise that scientific collaboration and publishing in high-impact or even top journals pays off. Also their publication activity might be stimulated. • Possible negative effects • Exaggerated collaboration, even trends towards hyper-authorship, inflating publication output by splitting up publications to sequences, inflating citation impact by self-citations and forming citation cliques, etc. • Trend towards replacing quality and recognition by visibility at any price or towards preferring journals as publication channels in social sciences and humanities might be among these effects. • A certain “champions league” mentality is spreading among scientists (“Shanghai ranking”, H-index, “IF Filter”). Bibliometrics. Use and limitations

28. 6. BIBLIOMETRIC RESEARCH AND SERVICES AT THE STEUNPUNT O&O STATISTIEKEN Bibliometrics. Use and limitations

29. 6. The Steunpunt O&O Statistieken The Steunpunt O&O Statistieken (Research Centre for R&D Statistics – SOOS) was created in January 2002 after open calls for proposals. SOOS is a unique and dedicated research center to support its science and innovation policy in Flanders. This center has to develop an appropriate system of quantitative indicators to quantify R&D efforts at Flemish universities, research institutes and industry that can be used by policy makers to support the development of appropriate regional science and innovation policy for Flanders. Bibliometrics. Use and limitations

30. 6. The Steunpunt O&O Statistieken • ACTIVITY OVERVIEW • The activities of SOOS can be classified into four categories: • 1. Developing an appropriate IT-infrastructure to support S&T indicator development; • 2. Providing the Flemish government with appropriate S&T indicators on which to base its science and technology policy resource allocation decisions; • 3. Executing specific studies on science and technology domains or science policy relevant topics for the Flemish government; • 4. Doing original research in the field of S&T studies. Bibliometrics. Use and limitations

31. 6. The Steunpunt O&O Statistieken LONG- AND MEDIUM-TERM SERVICES FOR THE FLEMISH GOVERNMENT I. The BOF-key One of the major funding mechanisms for basic science in Flanders is the BOF-mechanism. The Bijzonder Onderzoeksfonds is a research fund of the Flemish government that is distributed to the Flemish universities on the basis of a specific allocation model. This allocation key comprises both input and output measures on the research performance of the universities involved. Part of the key is based on publication and citation data as derived from WoS by SOOS. Bibliometrics. Use and limitations

32. 6. The Steunpunt O&O Statistieken BOF-funding for Flanders amounts to 100 million EUR annually. This amount is distributed in basic science funding amongst the six Flemish universities (LUC/UHasselt, KUB, KUL, VUB, UA and UGent). II. The IOF-key The IOF-key is a distribution key to determine the annual allocation of 12 million EUR (to be augmented in the year 2006) amongst the six Flemish universities in order to support applied research. The IOF-key is partially based on patent statistics derived from the USPTO, EPO and PCT databases. SOOS has been instrumental both in the development of and the data delivery for those distribution instruments. Bibliometrics. Use and limitations

33. 6. The Steunpunt O&O Statistieken III. The Flemish Indicatorbook The Flemish Indicatorbook on Science, Technology and Innovation appears biannually, and does not only support government questions on R&D-proficiency in Flanders, but also aims to supply adequate and up-to-date information on the European R&D-potential of the Flanders region. Two editions of the Indicatorbook have been published by SOOS so far: 2003 and 2005. Bibliometrics. Use and limitations

34. 6. The Steunpunt O&O Statistieken IV. Domain Studies (here bibliometrics only) Aim of these studies is to conduct and to provide analyses of Flemish research and technology activity in the international context in highly important areas. Expected outcomes are relevant information about performance and competitiveness in Flanders. Three domain studies have been prepared and published by SooS: Nanoscience and –technology, Biotechnology and molecular biology (both in 2003) and Stem-cell research (2004). The studies are based on both publication and patent data extracted from the WoS, the USPTO and EPO databases. Bibliometrics. Use and limitations

35. 6. The Steunpunt O&O Statistieken BIBLIOMETRIC RESEARCH PROJECTS AT SOOS Methodological and applied research is closely linked to the service tasks, for instance, in the context of methodological validation, of improvement of the efficiency of indicators or of the subject classification. ‘Basic research’ at the Steunpunt is structured in a systematic way to support policy-relevant research and services and to contribute to the advancement of the field of bibliometrics.   Similarly to the services, research is done at the national/regional, institutional and research-group level. Bibliometrics. Use and limitations

36. 5. The Steunpunt O&O Statistieken • Development of a hierarchic subject classification scheme for evaluation purposes • The role of author self-citations in scientific communication • The myth of delayed recognition • Analysis of possible extension of bibliometric methods to social science and humanities • Studying the network of science communication • “Indicator research” (improving journal and subject impact measures and science indicators) Bibliometrics. Use and limitations

37. 6. The Steunpunt O&O Statistieken • Bibliometrics beyond the WoS (e.g., proceedings literature) • Bibliometric approach to social sciences and humanities • Combining text-mining and bibliometrics • Mapping academic mobility and scientific collaboration in Europe • Research group dynamics and their impact on performance and visibility • Bisociation analysis of scientific communication Bibliometrics. Use and limitations