Application of bibliometric analysis Advantages & pitfalls Thed van Leeuwen

1. Application of bibliometric analysisAdvantages & pitfallsThed van Leeuwen Workshop on Research Evaluation in Statistical Sciences , Bologna, 25th March 2010

2. Introduction of bibliometrics Bibliometrics can be defined as the quantitative analysis of science and technology performance and the cognitive and organizational structure of science and technology. Basic for these analyses is the scientific communication between scientists through (mainly) journal publications. Key concepts in bibliometrics are output and impact, as measured through publications and citations. Important starting point in bibliometrics: scientists express, through citations in their scientific publications, a certain degree of influence of others on their own work. By large scale quantification, citations indicate influence or (inter)national visibility of scientific activity, but should not be interpreted as synonym for ‘quality’.

3. CWTS data system CWTS has a full bibliometric license from Thomson Reuters Scientific to conduct evaluation studies using the Web of Science. Our database covers the period 1981-2009. Some characteristics: Over 31.000.000 publications. Over 350.000.000 citation relations between source papers. 100.000.000 authors (incl. variations), 15.000.000 ‘unique’ names. Over 60.000.000 addresses, some 90% cleaned up over the last 10 years. Contains reference sets for journal and field citation data.

4. Bibliometric indicators produced by CWTS

5. Some basic indicators are … P: number of publications in journals processed for the Web of Science. C: number of received citations, excl. self-citations. CPP: mean number of citations per publication, excl. self- citations Pnc: percentage of the publications not cited (within a certain time-frame !!!) % SC: percentage self-citations related to an output set.

6. Important indicators are… CPP/JCSm: ratio between real, actual impact, and mean journal impact. CPP/FCSm: ratio between real, actual impact, and mean field impact. JCSm/FCSm: ratio between journal impact, and field impact, indicative for the ‘quality’ of the journal package in the field

7. Various types of analysis focus on … Research profiles: a break down of the output over various fields of science. Scientific cooperation analysis: a break down of the output over various types of scientific collaboration. Knowledge user analysis: a break down of the ‘responding’ output into citing fields, countries or institutions. Highly cited paper analysis: which publications are among the most highly cited output (top 10%, 5%, 1%) of the global literature in that same field(s). Social network analysis: how is the network of partners composed, based on scientific cooperation.

8. Journal & Field Normalization

9. Calculating the JCSm & FCSm ---------------------------------------------------------------------------------------------- Type publ. Journal Journal # citations year category until 1999 ----------------------------------------------------------------------------------------------  I review 1996 CANCER RESOncology 17 II note 1997 J CLIN ENDEndocrinology 4 III article 1999 J CLIN ENDEndocrinology 6 IV article 1999 J CLIN ENDEndocrinology 8 ----------------------------------------------------------------------------------------------  

10. Calculating the JCSm & FCSm 2 ----------------------------------------------------------------- CPP JCSFCS ----------------------------------------------------------------- I 17 16.9 23.7 II 4 3.1 3.0 III6 4.8 4.1 IV 8 4.8 4.1 -----------------------------------------------------------------

11. Calculating the JCSm & FCSm 3 The mean citation score is determined as: 17 + 4 + 6 + 8 CPP = ------------------ = 8.8 1 + 1 + 1 + 1 Themean journal citation score as: (1 x 16.9) + (1 x 3.1) + (2 x 4.8) JCSm = -------------------------------------- = 7.4 1 + 1 + 2 CPP / JCSm (8.8 / 7.4) = 1.19 Themeanfield citation score as: (1 x 23.7) + (1 x 3.0) + (2 x 4.1) FCSm = -------------------------------------- = 8.7 1 + 1 + 2 CPP / FCSm (8.8 / 8.7) = 1.01

12. Citation Windows & Impact Measurement

13. Citation measurement and ‘windows’ Publication years, fixed citation ‘window’. Publications of 2002, with three citation years (namely 2002, 2003, and 2004), followed by 2003, with three years, etc. Blocks of publication years with a window decreasing in length. Publications of 2002-2005, with citation window of 4 years (2002-2005), 3 years (2003-2005), 2 years (2004-2005), and 1 year (2005).

14. Citation measurement with ‘fixed window’ Citation years 2002 2003 2004 2005 2006 2007 2008 2009 • 2002 • 2003 • 2004 • 2005 • 2006 • 2007 • 2008 • 2009 2002 2003 2004 2003 2004 2005 2004 2005 2006 2005 2006 2007 2006 2007 2008 2007 2008 2009 2008 2009 2009

15. Citation measurement with ‘year blocks’ Citation years 2002 2003 2004 2005 2006 2007 2008 2009 • 2002 • 2003 • 2004 • 2005 • 2006 • 2007 • 2008 • 2009 2002 2003 2004 2005 2003 2004 2005 2004 2005 2005 2003 2004 2005 2006 2004 2005 2006 2005 2006 2006 2004 2005 2006 2007 2005 2006 2007 2006 2007 2007 2005 2006 2007 2008 2006 2007 2008 2007 2008 2008 2006 2007 2008 2009 2007 2008 2009 2008 2009 2009

16. Methodological issues

17. Adequacy of citation indexes : implications for bibliometric studies

18. How to tackle this issue ? • We conduct analyses on the adequacy of the citation indexes across disciplines based on reference behavior of researchers themselves. • The degree of referring towards other indexed literature indicates the importance of journal literature in the scientific communication process.

19. Assessment of WoS Coverage Citing/Source Non-WoS WoS Non-Wos Journals Books Conference proceedings Reports Etc. ?% ?% Cited/Target Non-WoS WoS

20. Total ISI/WoS Database (2002) Citing/Source Non-WoS WoS 25% 75% Cited/Target Non-WoS WoS

21. The medical & Life sciences

22. The natural sciences

23. Statistical sciences

24. The engineering sciences

25. The social– and behavioral sciences

26. The humanities

27. Overall WoS coverage by main field

28. Conclusions on adequacy issue • We can clearly conclude that the application of bibliometric techniques, solely based on WoS (but very likely also Scopus) will not be valid for some of the ‘soft’ fields in the social sciences and the humanities. • That is why the tool box has to be extended !

29. The H-Index and its limitations

30. The H-Index, defined as … The H-Index is the score that indicates the position at which a publication in a set, the number of received citations is equal to the ranking position of that publication. Idea of an American physicist, J. Hirsch, who published about this index in the Proc. NAS USA.

31. Examples of Hirsch-index values • Environmental biologist, output of 188 papers, cited 4,788 times in the period 80-04. • Hirsch-index value of 31 • Clinical psychologist, output of 72 papers, cited 760 time sin the period 80-04. • Hirsch-index value of 14

32. Problems with the H-Index For serious evaluation of scientific performance, the H-Index is as indicator not suitable, as the index: Is insensitive to field specific characteristics (e.g., difference in citation cultures between medicine and other disciplines). Does not take into account age and career length of scientists, a small oeuvre leads necessarily to a low H-Index value. Is inconsistent in its ‘behaviour’.

33. Actual versus field normalized impact (CPP/FCSm) displayed against the output. Large output can be combined with a relatively low impact

34. H-Index displayed against the output. Larger output is strongly correlated with a high H-Index value.

35. Consistency: Definition Definition. A scientific performance measure is said to be consistent if and only if for any two actors A and B and for any number n ≥ 0 the ranking of A and B given by the performance measure does not change when A and B both have a new publication with n citations. 35

36. Consistency: Motivation Consistency ensures that if the publishing behavior of two actors does not change over time, their ranking relative to each other also does not change Consistency ensures that if the individual researchers in one research group X outperform the individual researchers in another research group Y, the former research group X as a whole outperforms the latter research group Y. 36

37. Inconsistency of the h-index h = 4 h = 6 h = 8 h = 6 Actor A Actor B 37

38. ISI Impact Factors: calculation and validity

39. Methodology: ISI’s classical IF The ISI Impact Factor (IF) is defined as the number of citations received by a journal in year t, divided by the number of citeable documents in that same journal in the years t-1 and t-2, • Or, as a Formula: Citations in year t Number of ‘citeable documents’ in t-1 & t-2

40. Share ‘citations-for-free’ for The Lancet Publications Citations 90+91 1992 Article 784 2986 Note 144 593 Review 29 232 Sub-total 957 (a) 7959 (b) Letter 4181 (d) 4264 (e) Editorial 1313 905 Other 1421 909 Total 7872 14037 (c) • ISI Method: • Citations in 2000 . • Citeable documents in ‘98 and ‘99 • 14037 (c) • 957 (a) IF=14.7 • CWTS Method: • Citations to Art/Not/Rev in 2000 . • Art/Not/Rev in ‘98 and ‘99 • 7959 (b) • 957 (a) • Citations to Art/Let/Not/Rev in 2000 . • Art/Let/Not/Rev in ‘98 and ‘99 • 7959+4264 (b+e) • 957+4181 (a+d) IF=8.3 IF=2.4

41. ISI Impact Factors From 1995 onwards CWTS has analyzed the uses and validity ISI Journal Impact Factor (IF). Most important points of criticism were: • Calculated erroneously. • Not sensitive for the composition of the journal in terms of the document types. • Not sensitive for the science fields a journal is attached to … • Based on too short ‘citation windows’.

42. Distribution of citations used for the calculationof the IF value of The Lancet • The red area indicates citations ‘for free’, while the blue area indicates ‘correct citations’ • The IF-score of The Lancet is seriously ‘overrated’ by the scientific ‘audience’ of the journal.

43. Impact Factors for Br. J. Clin. Pharm. and Clin. Pharm. & Ther. The graph shows the correct and erroneous impact factors of BJCP and CPT • In the case of CPT, citations to published meetingabstracts are included, while BJCP has stopped publishing of meetingabstracts !

44. Document types and fields Field Journal IF JFIS The IF is for ‘02, JFIS covers ‘98-‘02

45. Fields and Citation windows

46. Citation measurement of IF 2002 2003 2004 2005 2006 2007 2008 2009 • 2002 • 2003 • 2004 • 2005 • 2006 • 2007 • 2008 • 2009 20022003 2004 20032004 2005 20042005 2006 20052006 2007 20062007 2008 20072008 2009 20082009 2009

47. CWTS answer to the problems of the IF This indicator is the JFIS, the Journal-to-Field Impact Score. The JFIS solves the main objections against the Impact Factor, as the calculation of JFIS is based on equally large entities, document types are taken into account, JFIS is field-normalized, and finally, based on longer citation windows (1-4 years)

48. Citation measurement of JFIS Citation years 2002 2003 2004 2005 2006 2007 2008 2009 • 2002 • 2003 • 2004 • 2005 • 2006 • 2007 • 2008 • 2009 2002 2003 2004 2005 2003 2004 2005 2004 2005 2005 2003 2004 2005 2006 2004 2005 2006 2005 2006 2006 2004 2005 2006 2007 2005 2006 2007 2006 2007 2007 2005 2006 2007 2008 2006 2007 2008 2007 2008 2008 2006 2007 2008 2009 2007 2008 2009 2008 2009 2009

49. End of the presentation For questions regarding the contents of the presentation, mail to: leeuwen@cwts.nl