1 / 18

What is Evidence ?

How good is the research base? New approaches to research indicators Colloque de l’Académie des sciences "Évolution des publications scientifiques" 14-15 mai 2007. What is Evidence ?. Research performance analysis and interpretation Founded 2000, grew from government and HE research management

anakin
Download Presentation

What is Evidence ?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. How good is the research base?New approaches toresearch indicatorsColloque de l’Académie des sciences "Évolution des publications scientifiques" 14-15 mai 2007

  2. What is Evidence? • Research performance analysis and interpretation • Founded 2000, grew from government and HE research management • Studies for e.g. UUK, HEFCE, OST, Defra, EC, universities • Annual OSI PSA target indicators for UK science and engineering • Research funding and impact studies across research base • Current work for Austria, New Zealand, Sweden • Quantitative research analysis products • Overview of complete research process • Funding, activity and outputs in detailed, structured and mapped databases • Data reconciled to subject areas and institutions • Higher Education Research Yearbook (5th ed’n) • Indicator applications, Publication databases, Research profiling

  3. Linking indicators to management information • ‘Average impact’ is a good bibliometric index but not sufficient • A great tool for reporting but not for action • Average is a metric; distribution is a picture • Data are skewed, so average is not central • Many papers are uncited and a few papers are very highly cited • New approach looks at where the spread of performance falls • Activity is located within distribution by more than a single metric • Thresholds help in describing peak of performance • This improves descriptive power, information content and management value

  4. Traditional impact indicators are excellent for international reports

  5. Bibliometrics track increase in UK share of world citations in response to research assessment

  6. Impact index is coherent across UK grade levels - data are for core science disciplines, grade at RAE96

  7. Chemistry – alternative bibliometric indices, both correlate with our mapping to RAE grade Each data point is an institution

  8. Bibliometric impact (1996-2000) is related to RAE2001 grade for UoA14 Biology

  9. Assumed distribution of “research performance”

  10. Actual distribution of data values The variables for which we have metrics are skewed and therefore difficult to picture in a simple way

  11. Simplify the data picture • Scale data relative to a benchmark, then categorise • Could do this for any data set • All journal articles • Uncited articles (take out the zeroes) • Cited articles • Cited less often than benchmark • Cited more often than benchmark • Cited more often but less than twice as often • Cited more than twice as often • Cited more often but less than four times as often • Cited more than four times as often

  12. Categorise the impact data This grouping is the equivalent of a log 2 transformation. There is no place for zero values on a log scale.

  13. UK Impact ProfileTM[10 years; 680,000 papers] MODE (cited) AVERAGE RBI = 1.24 MODE MEDIAN THRESHOLD OF EXCELLENCE?

  14. Implications • Is the UK research base as good as we thought? • YES - the average is unchanged • What lies beneath just became apparent • The ‘peak’ of high impact is very concentrated • Evaluate Impact ProfileTM methodology • Do other countries look similar? • Yes, we profiled the USA as well • Does it work by year and by subject? • See Scientometrics, Vol. 72, No. 2 (2007) 325–344 • How can we apply it?

  15. Impact ProfilesTM for subjects & sites – molecular biology

  16. Impact ProfilesTM for international institutes Location USA EMBL UK France Japan

  17. Where does this take us? • New metrics would advance utility and application • ‘Average impact’ is not indicative of distribution • Should we also use e.g. median, mode? • Index proportion of activity at thresholds of excellence? • Above world average, More than 4 x world average, etc • Weight the categories to produce a single metric • A single metric is still useful for reporting • How do we treat uncited? • How much do we value the truly exceptional? • A picture has value in itself • It has descriptive power beyond a simple index • It enables rapid and transparent comparisons for the less expert • It helps us locate specific activity within a profile

  18. www.evidence.co.ukHow good is the research base?New approaches toresearch indicators

More Related