How good is our research? New approaches to research indicators

1. How good is our research?New approaches toresearch indicators

2. Average is a metric; distribution is a picture • ‘Average impact’ is a good bibliometric index but not sufficient • A tool for reporting but not for action • Data are skewed, so average is not central • Many papers are uncited and a few papers are very highly cited • New approach looks at where the spread of performance falls • Activity is located within distribution by more than a single metric • Thresholds help in describing peak of performance. • This improves descriptive power, information content and management value

3. Traditional impact indicators

4. Distribution of “research performance”

5. Distribution of “research performance” A good ‘indicator’ should capture and reflect this in some meaningful way Do current metrics do this?

6. Distribution of data values - income Minimum Maximum

7. Distribution of data values - impact The variables for which we have metrics are skewed and therefore difficult to picture in a simple way

8. Simplifying the data picture • Scale data relative to a benchmark, then categorise • Could do this for any data set • All journal articles • Uncited articles (take out the zeroes) • Cited articles • Cited less often than benchmark • Cited more often than benchmark • Cited more often but less than twice as often • Cited more than twice as often • Cited less than four times as often • Cited more than four times as often

9. Categorising the impact data This grouping is the equivalent of a log 2 transformation. There is no place for zero values on a log scale.

10. UK ten-year profile 680,000 papers MODE (cited) AVERAGE RBI = 1.24 MODE MEDIAN THRESHOLD OF EXCELLENCE?

11. Implications • Is the UK as good as we thought? • YES - the average is unchanged • What lies beneath just became apparent • The effective peak is very concentrated • Other countries would probably look similar • New metrics are needed • Average impact not indicative of distribution • Need to add median, mode • Proportion of activity at thresholds of excellence • Above world average, More than 4 x world average, etc • Evaluate methodology • Does it work by year and by subject? • How can we apply it?

12. Time profile

13. Subject based curves

14. Subject & site profiles – molecular biology

15. HEIs – 10 year totals - 1

16. HEIs – 10 year totals - 2

17. HEIs – 10 year totals – 4.1 Smoothing the lines would reveal the shape of the profile

18. HEIs – 10 year totals – 4.2 Absolute volume would add a further element for comparisons

19. HEIs – 10 year totals – 4.3

20. What next? • Profiles • Create a view of the distribution of performance • Provide more information useful to management • Require a change in metrics • Applications • Disaggregate the components of the research base • Track institutional profiles against benchmark • Evaluate the link between platform and peak • Track papers through time: e.g. leaders vs. climbers

21. How good is our research?New approaches toresearch indicators