1 / 102

First the basic principles of bibliometric analysis

trista
Download Presentation

First the basic principles of bibliometric analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Measuring Quality and Impact of the Social SciencesConcepts, Opportunities and DrawbacksPre-Conference of the 10th International Conference on Science and Technology Indicators University of Vienna, September 17, 2008Anthony F.J. van RaanCenter for Science and Technology Studies (CWTS)Leiden University

  2. This presentation will highlight recent CWTS projects:* Benchmarking & Evaluation* HEFCE * Identification of Excellence

  3. From these recent studies we present empirical results for social science fields particularly concerning:* WoS coverage * Characteristics of WoS publications* Characteristics of n-WoS publications* Bibliometric results and peer judgments

  4. First the basic principles of bibliometric analysis

  5. Basic Concept: QualityScientific performance relates to achieved quality in the contribution to the increase of our knowledge (‘scientific progress’)(1) as perceived by others: peer review (2) as measured by advanced bibliometric analysis

  6. Basic issues for research assessment, also in the social sciences:* Objectivity* Transparency* How to handle interdisciplinarity, definition of fields* Different ways, prestige and intensity of publication* Role of co-authors in publications* Orientation of research: local vs. global* Language* Ageing of research results* PhD training* Time dimension of awards* Socio-economic impact

  7. Citing Publications Cited Publications All calculations are corrected for self-citations!

  8. What do citations measure? • Many studies showed positive correlations between citations and qualitative judgments • In principle it is valid to interpret citations in terms of intellectual influence which is an important aspect of scientific quality • Thus, the concepts of citation impact and scientific quality do not coincide ‘automatically’

  9. GoogleScholar Total publ universe non-WoS publ: Books Book chapters Conf. proc. Reports ArXiv Scopus WoS sub-universe 8,000 j; 1,000,000p/y LNCS Source expansion Compendex *CWTS is in license agreement negotiations with Scopus *CWTS currently compares Scopus- vs. WoS coverage *CWTS bibliometric algorithms can be applied to Scopus data Medline Refs > nWoS Target expansion

  10. Network of publications (nodes) linked by citations (edges) Lower citation-density Higher citation-density e.g., applied research, e.g., basic natural social sciences medical research FCSm JCSm Expected values for normalization Absolutely necessary but……are they appropriate? CPP

  11. CWTS applies two types of field definitions: Field = set of journals ‘established fields’ scientific medium-grained structure Journal + reference-based re-definition (expansion) of fields

  12. Main field: Social and Behavioral Sciences Major field, e.g. Economics & Business All publication titles + abstracts (~30,000,000) have been grammatically parsed to enable bibliometric analysisby themes/concepts/ instruments and to create word-correlation based maps of science journals fields

  13. Field = clusters of concept-related publications new, emerging often interdisc. fields scientific fine-grained structure cluster

  14. Social Sciences Top-50 EU universities, their top-10% publications in this field Now specific sub-field CPP/FCSm values can be calculated, for instance for research on democracy But, obviously, the finer grained, the more ‘noisy’

  15. Basic Performance Indicators • POuput: Number of publications in internationally refereed CI-covered journals • C Absolute Impact:Number of (self-ex) citations to these publications • H Hirsch-index • CPPOutput-normalized Impact: Average number of cits/pub of the institute • JCSm Average number of cits/pub of the journal set used by the institute • FCSm Average number of cits/pub of all journals of a specific field in which the institute is active (FCSm) • p0 Percentage of not-cited publications

  16. CWTS Key Research Performance Indicators: • JCSm/FCSm Relative impact of the used journal set • CPP/JCSm Internat. journal-normalized impact • CPP/FCSmInternat. field & doc-normalized impact • Pt/ΠtContribution to thetop-5, 10, 20,..% • P*CPP/FCSmSize & Impact Together: Brute Force

  17. Basic research high FCSm High CPP high FCSm, but low JCSm low FCSm, but high JCSm low CPP low FCSm Up to factor ~20 Applied research, engineering

  18. Internal WoS-coverage of social science fieldsresults from HEFCE and Benchmark projects

  19. Internal WoS coverage of main fields of science

  20. What is the internal WoS coverage and how is it calculated? Example: EUR 2000-2004

  21. Internal WoS coverage (%) of submitted publications per UoA From: Moed, Visser, Buter, 2008

  22. Internal WoS coverage for all main fields of science

  23. 1991-2006 purple: non-WoS ref light blue: CI ref

  24. 1991-2006 purple: non-WoS ref light blue: CI ref

  25. External WoS-coverage of social science fieldsresults from HEFCE and Evaluation projects

  26. What is the external WoS coverage and how is it calculated? Example: Uppsala 2002-2006

  27. From: Van Leeuwen 2006

  28. 84% of the total number of publications submitted to the 2001 RAE from science-related departments were published in WoS-covered journals. For Mathematics publications the WoS coverage is only slightly lower (82%), It is substantially lower for Social Sciences and Humanities (25%) From: Moed, Visser, Buter, 2008

  29. External WoS coverage (%) of submitted publications per UoA

  30. What is the correlation between internal and external WoS coverage?

  31. Characteristics of WoS publications insocial science fieldsresults from HEFCE and Benchmark & Evaluation projects

  32. time lag & citation window Publications from 1991,….1995

  33. Main differences with the natural and medical sciences: *Lower numbers (more than 1 order of magnitude….) *Slower rise , broader peak and much slower decay (less hectics…)

  34. EUR 200-2006 Benchmark Study

  35. Characteristics of non-WoS publications insocial science fieldsresults from HEFCE and Benchmark & Evaluation projects

  36. Top-10% (of impact) of EU publications in Political Science, Economics, and Psychology 1997-2003, 4-y citation window (to calculate their impact) From references all WoS-references removed, only non-WoS references(with freq > 2)have been analyzed Total about 28,000

  37. From: Nederhof, van Leeuwen, van der Wurff 2008 From these:

  38. (> 1980)

  39. Top-50 non-WoS >1980 references by document type

  40. Bibliometric results and peer judgmentsresults from HEFCE and Benchmark & Evaluation projects

More Related