ISI Research Services Group:

1. ISI Research Services Group: Analytical tools for Research Evaluation

2. Outline • Research evaluation: • Unique features of Web of Knowledge • Journal Selection • Quantitative tools • Cui bono: What sectors use these analyses • Citation Analysis – Issues and Caveats • Free Stuff!! ISI Research Services Group Nancy K. Bayers

3. The Web of Knowledge is… • An integrated, web-based environment providing high quality content and the tools to access, analyze, and manage scientific and scholarly information • Incorporating new technologies (linking, searching, and personalization) to integrate, extend, and enhance the researchenvironment ISI Research Services Group Nancy K. Bayers

4. Web of Knowledgesm E-Journal Full-Text A Free Resource!! Evaluated, Selected Web Content E-Patent Full-Text ISI Research Services Group Nancy K. Bayers

5. ISI Web of Knowledge • A. Journal Selection Process • “A cost effective index index must restrict its • coverage, as nearly as possible to only those • items that researchers are likely to find useful.” • -Eugene Garfield, Ph. D. ISI Research Services Group Nancy K. Bayers

6. ISI Web of Knowledge • … Journal Selection • 1. Basic Journal Publishing Standards • 2. Editorial Content • 3. International Diversity • 4. Citation Analysis ISI Research Services Group Nancy K. Bayers

7. ISI Web of Knowledge … Journal Selection 1.) Publishing Standards a.) Timeliness b.) Editorial Conventions c.) English Language Bibliographic Information d.) Peer Review ISI Research Services Group Nancy K. Bayers

8. ISI Web of Knowledge ... Journal Selection 1.) Basic Journal Publishing Standards a.) Timeliness of Publication • A journal must be published according to its stated publication schedule to be considered for coverage in the ISI database. • ISI must receive three on-time issues in sequence. ISI Research Services Group Nancy K. Bayers

9. ISI Web of Knowledge • ... Journal Selection • 1. Basic Journal Publishing Standards • a.) Timeliness of Publication • b.) International Editorial Conventions • To optimize retrievability: • - Informative Journal Titles • - Descriptive Article Titles • - Full Address Information for Every Author • - Complete Bibliographic Information for • All Cited References ISI Research Services Group Nancy K. Bayers

10. ISI Web of Knowledge ... Journal Selection 1.) Basic Journal Publishing Standards a.) Timeliness of Publication b.) International Editorial Conventions c.) English Language Bibliographic Information - Article Titles - Author Names and Addresses - Cited References - Abstracts and Keywords ISI Research Services Group Nancy K. Bayers

11. ISI Web of Knowledge ... Journal Selection 1.) Basic Journal Publishing Standards a. Timeliness of Publication b. International Editorial Conventions c. English Language Bibliographic Information d. Peer Review Application of the peer review process is another indication of the journal’s standards, the overall quality of research presented, and the completeness of cited references. ISI Research Services Group Nancy K. Bayers

12. ISI Web of Knowledge • ... Journal Selection • 2.) Editorial Content • Do we need this new journal in the database, or is the category already well covered? • Does this journal offer something new? ISI Research Services Group Nancy K. Bayers

13. ISI Web of Knowledge • ... Journal Selection • 3.) International Diversity • Are the authors, editors, and editorial advisory • board members from around the world? • Does this journal reflect the global context • in which scientific research takes place? ISI Research Services Group Nancy K. Bayers

14. ISI Web of Knowledge • ...3. International Diversity • Regional Journals: • To properly reflect the global context in which scientific research takes place, and to provide balanced coverage in each category, ISI also seeks to cover the best regional journals as well. ISI Research Services Group Nancy K. Bayers

15. ISI Web of Knowledge • ... Journal Selection • 4. Citation Analysis • Expert use of citation data help ISI identify • important, influential, and useful publications. • ISI measuresuse by analyzing citation data. ISI Research Services Group Nancy K. Bayers

16. ISI Web of Knowledge • ... Journal Selection • ... 4. Citation Analysis • New Journals • Has the past work of authors and editorial • advisory board members received citations? • Established Journals • Impact Factor ISI Research Services Group Nancy K. Bayers

17. ISI Web of Knowledge ... Journal Selection ...4. Citation Analysis Impact Factor: The average number of times recent articles in a journal were cited in a particular year. Nature Genetics: 2000 Impact factor: 30.910 ISI Research Services Group Nancy K. Bayers

18. ISI Web of Knowledge Journal Citation Reports (JCR) -- 2001 Science Edition Impact Factor Calculation Journal: Nature Genetics Impact Factor: 29.600 Cites in 2001 to articles published in : 2000 = 6,167 1999 = 7,538 00+99 = 13,705 Number of articles published in: 2000 = 253 1999 = 210 00+99 = 463 Calculation: Cites to recent articles 13,705 = 29.600 Number of recent articles 463 ISI Research Services Group Nancy K. Bayers

19. Research Evaluation • Web of Knowledge: unique features • cited references -- footnotes • all author names • all author addresses • Count • papers -- output • citations -- influence or impact • Baselines ISI Research Services Group Nancy K. Bayers

20. Cui bono? • Policymakers • Research directors & administrators • Researchers • Publishers • Information specialists ISI Research Services Group Nancy K. Bayers

21. Words to the wise... • Citation analysis is a fair evaluative tool in those fields where publication in the journal literature is the main vehicle of communication. • Need multiple indicators. Citation analysis is only a quantitative supplement to other measures of research evaluation, including peer review • There are always exceptions to everything. ISI Research Services Group Nancy K. Bayers

22. Citation analysis: issues and caveats • Field publication behavior differences • How to handle... • Self cites and negative cites • Reviews and methods papers • English language bias • Skewed distributions • Relative measures/baselines • Confidence levels • What citations do not reflect ISI Research Services Group Nancy K. Bayers

23. Field citation rates: average cites 1997-2001 • Cell & Developmental Biology 15.9 • Molecular Biology 10.6 • Biochemistry & Biophysics 9.1 • Neuroscience 7.3 • Chemistry & Analysis 5.9 • Physics 4.4 • Earth Science 3.5 • Materials Science 2.1 • Mathematics 1.3 ISI Research Services Group Nancy K. Bayers

24. Field publication rates/size Field % papers in world • Clinical Medicine 24% • Chemistry 14% • Physics 12% • Biology & Biochemistry 8% • Plant & Animal Science 6% • Materials Science 4% • Neuroscience 4% • Geosciences 3% • Mathematics 2% • Computer Science 1% ISI Research Services Group Nancy K. Bayers

25. Self-cites and negative cites SELF CITES • Cumulative nature of individual research • a natural and acceptable procedure if not excessive • Approximately 25% in biomedicine • 16% in plant physiology • 17.5% in neurobiology NEGATIVE CITES • Only 7% ISI Research Services Group Nancy K. Bayers

26. Reviews and methods papers • Highly cited • Influence subsequent research • Lay foundation ISI Research Services Group Nancy K. Bayers

27. English language bias • English is the language of science • Representative journals from each region and nation • Not significant for most analyses ISI Research Services Group Nancy K. Bayers

28. Skewed distribution in bibliometrics • 80/20 rule • 20% of papers get 80% of cites • 50% of papers are never cited ISI Research Services Group Nancy K. Bayers

29. Relative measures--baselines • Need context -- what does X cites mean? • Normalize by size -- average citation rates • Normalize by field -- average field citation rates ISI Research Services Group Nancy K. Bayers

30. What citations to not reflect • Clinical importance • Public policy influence • Regional or local interest ISI Research Services Group Nancy K. Bayers

31. Confidence levels Large data sets (nations, regions, large institutions) high moderate Engineering & Applied Sciences Fundamental & Basic Sciences low moderate Small data sets (small departments, individual researchers) ISI Research Services Group Nancy K. Bayers

32. In-Cites: www.in-cites.com ISI Research Services Group Nancy K. Bayers

33. In-Cites: www.in-cites.comEditorial Commentary by Highly Cited Scientists ISI Research Services Group Nancy K. Bayers

34. In-Cites: www.in-cites.comHot Paper Commentary in Medicine ISI Research Services Group Nancy K. Bayers

35. Special Topics: www.esi-topics.com ISI Research Services Group Nancy K. Bayers

36. Special Topics: www.esi-topics.com ISI Research Services Group Nancy K. Bayers

37. Special Topics: www.esi-topics.com ISI Research Services Group Nancy K. Bayers

38. Special Topics: www.esi-topics.com ISI Research Services Group Nancy K. Bayers

39. Science Watch: www.sciencewatch.com ISI Research Services Group Nancy K. Bayers

40. ISI Research Services Group: Thank You!! Analytical tools for Research Evaluation