1 / 31

Patricia Brennan Thomson Scientific January 10, 2008

Patricia Brennan Thomson Scientific January 10, 2008. How do we assess value? Current Academic Evaluation Market Role of citations, citation metrics Possibilities with new metrics – Usage Factors, H Index, Irish Research output: Summary Metrics.

trory
Download Presentation

Patricia Brennan Thomson Scientific January 10, 2008

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Patricia Brennan Thomson Scientific January 10, 2008

  2. How do we assess value? • Current Academic Evaluation Market • Role of citations, citation metrics • Possibilities with new metrics – Usage Factors, H Index, • Irish Research output: Summary Metrics Research Publication and Performance - Impact Factor Imperative?

  3. Stakeholders in Academic Evaluation

  4. Drivers for increased evaluation and assessment

  5. Scholarly communications are changing: what is the output ? Scholarship is increasingly Collaborative Source: National Citation Report Ireland 2003-2003

  6. Scholarship is Global >5 countries Source: Web of Science®

  7. Scholarship is increasingly collaborative Source: Web of Science®

  8. Collaborations are getting broader and more complex Largest Collaboration in 2006: 2512 authors, a “collaboration of collaborations”

  9. What to measure? When? • Total Papers • Total Citations • Citation Impact (cites per paper) • Percent Cited Paper • Impact Relative to Field • Percentile Rank in Field • Collaboration Indicators • Expected Citation Count • Ratio of Citations to Expected citation count • Expected Citation Rate for Category • Mean / Median Citation • H Index • Citation Frequency Distribution • Time Series Trends Authors Institutions Nations Topics Fields

  10. Metrics: Individual papers • 1. Cites: Total citation count for selected paper. • Cites2: 2nd generation cite count based on total citations received by the citing articles. • Expected Citation Rate: An average rate of citation for all the papers of that document type (articles, reviews, letters, etc.), in that journal, for that selected year. This is a metric to evaluate citation counts. • Ratio: Ratio of expected cites to actual cites • Field: Subject area for the journal in which the paper appeared. • %: Percentile position of the paper based on citations in the same field.

  11. Metrics: Individual papers • 1. Cites: Total citation count for selected paper. Cites: 85

  12. Metrics: Individual papers 2.Cites2: 2nd generation cite count based on total citations received by the citing articles. Cites: 85 Cites2: 574

  13. Metrics: Individual papers • Expected Citation Rate: An average rate of citation for all the papers of that document type (articles, reviews, letters, etc.), in that journal, for that selected year. This is a metric to compare peer journal papers. Cites: 85 Cites2: 574 Expected Citation Rate: 20.8 (All Articles from European Journal of Neuroscience in 2001 received on average 20.8 cites through year-end 2006.)

  14. Metrics: Individual papers • 4. Ratio: Ratio of expected cites to actual cites Cites: 85 Cites2: 574 Expected Citation Rate: 20.8 Ratio: 4.1 [85: 20.8 = 4.1]

  15. Metrics: Individual papers • 5. Field: Subject area for the journal in which the paper appeared. Cites: 85 Cites2: 574 Expected Citation Rate: 20.8 Ratio: 4.1 Field: Neuroscience [Note: For the multidisciplinary journals Science, Nature and PNAS, all articles and reviews are reassigned based on the primary category to which the article’s citing and cited journals are assigned.]

  16. Metrics: Individual papers • 6. Percentile: position of the paper based on citations in the same field and year. Cites: 85 Cites2: 574 Expected Citation Rate: 20.8 Ratio: 4.1 Field: Neuroscience Percentile: 3.5% [The 85 cites to this Neuroscience paper places it in the top 3.5% based on the citation distribution to all papers published in this field in 2001. ]

  17. Metrics for groups of papers • Total # papers and total # cites:combined numbers for the set • Mean times cited:Total cites divided by total papers. [average impact] • Median times cited:Midpoint for citations • H-Index:Number of papers (N) in a given dataset having N or more citations. • C-Index:Sum of all actual citations divided by sum of all expected citations. • Average Percentile: average of the field percentile measures which are based on field and year of publication • Disciplinarity: reflects the level of multidisciplinarity in a set of papers, ranging from 0 to 1, where the lower the number, the greater the multidisciplinarity. (Herfindal Index)

  18. H-Index: Number of papers (N) in a given dataset having N or more citations. 14 papers in this set had 14 or more citations

  19. C-Index:Sum of all actual citations divided by sum of all expected citations. C index = 1.98 357180.46

  20. Average Percentile: average of the field percentile measures which are based on field and year of publication Average Percentile = 31.62

  21. Tools for multiple workflows: Author, Researcher • Promotion and Tenure • Author Finder • The Distinct Author Identification System (DAIS) • Citation Report • ResearchID.com “.. I would prefer to see something like a unique identifier for people (so I can still get all of an author's work even if I don't know his professional history) ..”

  22. Tools for multiple workflows: Administrators, Researchers • Evaluating Research Teams Initial Views and Exploration • Analyze Tool • Citation Report Profiling Within Institutions • Journal Use Reports A Global View • Essential Science Indicators Detailed and Precise Analysis • Custom Analysis

  23. Tools for multiple workflows: Librarian, Publisher • Evaluating a journal collection: • Journal Citation Reports, • Journal Use Reports

  24. The Impact Factor Imperative IF Calculation

  25. Irish Research Output: Summary Metrics Source: National Citation Report Ireland 2003-2003

  26. Irish Research Output: Comparative Output Source: National Science Indicators 06

  27. Irish Research Impact: Comparative Impact Source: National Science Indicators 06

  28. Irish Research Output: Summary Metrics • Most Cited Author: Wolf K H • Cited 6143 times Source: National Citation Report Ireland 2003-2003

  29. Understanding Context: Average Impact per Field

  30. Establishing Best Practices Use relative measures, not just absolute counts Obtain multiple measures Recognize the skewed nature of citation data Confirm data collected are relevant to question Compare like with like • Consider whether available data can address the question • Choose publication types, field definitions, and years of data • Decide on whole or fractional counting • Judge whether data require editing to remove “artifacts” • Ask whether the results are reasonable

  31. Thank You

More Related