THE REF AND BIBLIOMETRICS Presentation at Northampton University, 3/2/09

1. THE REF AND BIBLIOMETRICSPresentation at Northampton University, 3/2/09 Charles Oppenheim Loughborough University C.Oppenheim@lboro.ac.uk

2. MY CREDENTIALS • Have undertaken research on the links between RAE results and bibliometrics since the mid 1990s • Member of the Committee advising HEFCE on the use of bibliometrics in the REF, and the pilot use of it to compare to 2008 RAE

3. ONE IMPORTANT QUESTION • Is the RAE a way of evaluating past performance, predicting future performance, or a way of working out how much QR money to dish out? • The three are not identical, yet the RAE tries to be all three • Evaluating past output (+ PhD completions, research income achieved, etc.) does the first; evaluating RA5, future research plans, does the second

4. THE REF • Announced by Gordon Brown when he was Chancellor of the Exchequer (so it is clear that the motivation is cost-cutting) • To be metrics based – details left to HEFCE et al to sort out • HEFCE itself evidently surprised by the announcement

5. THE REF • HEFCE commissioned expert advice on the use of bibliometrics and consulted the community on key elements of the REF • Large number of responses • Consultation outcomes published on HEFCE website • Significant modification announced in April 2008: • combination of metrics-based indicators, including bibliometrics where appropriate, as well as input from expert panels for all subjects

6. THE PILOT • Trial run of the bibliometrics approach using RAE2008 data • Ongoing right now • Main purpose of pilot is to assess two things: do the bibliometrics results correlate with actual RAE results? What are the administrative and technical burdens on HEIs in doing the pilot? • Broad results will be published; participating HEIs will get detailed results, to be retained for a short time period and only for the purpose of feeding back to HEFCE any errors or issues

7. THE REF PILOT IN PRACTICE • Collect ALL papers written by staff submitted to 2008 RAE by selected HEIs in the selected subject areas • Assign the papers to somewhere between 100 and 250 subject categories (probably two runs, one with the smaller and one with the larger number of subject categories) • Calculate: average no. of citations per article; again, but ignoring top and bottom 25% results; % uncited • Calculate: world average number of citations per article in chosen subject area over chosen time period • Calculate: % of articles from HEI that are above the world average • N.B. Subject areas based on journal title and where it is assigned by Thomson-Reuters; ignore non-journal articles (for the Pilot only hard sciences and life sciences are being examined)

8. FURTHER CALCULATIONS • Do the same, BUT: • Ignore all review articles (identified by algorithm) • Add in/exclude papers published in any previous employment not in this HEI • Exclude papers by Category C staff (medicine) • Restrict to 6 papers with the highest number of citations

9. FINALLY • See which of the combinations provides the best correlation with actual RAE results • HEFCE will digest the results and will then probably follow the best combination in running the real REF

10. HOW WILL THE REAL REF WORK? • Department submits (probably) all papers published by (probably) all staff over a certain time period for review (time period will depend on subject area; shorter for fast-moving subjects) • Issues regarding checking who is employed by the HEI, master list of publications – all of this will force HEIs to get their management information in order

11. NEXT STAGE • HEFCE counts the numbers of citations to all the papers and totals them up using WoS and/or SCOPUS (for pilot, it’s just WoS) • HEFCE assigns papers to subject area • HEFCE does a world calculation of the average number of citations per paper per year for that subject area • A profile, along the lines of RAE2008, will then be created of proportion of papers from Dep’t uncited, below world average, at world average, above world average; maybe by percentiles. • Decisions yet to be made about excluding certain publications, e.g., in “popular” outlets, review papers (characterised by number of citations in that article) from these calculations • Followed by a round of peer review (“light touch” for STM, heavier touch for arts/humanities) to amend profiles in light of particular circumstances of Department/subject area • The profile still forms just one component of final REF assessment of UoA – PhDs, research income, etc., still get considered

12. WHY BIBLIOMETRICS? • Civil servants clearly felt that this would provide a cheap and reliable method of evaluating research • But, following up the One Important Question, it is backward looking only and does not evaluate future research strategy • There are other issues as well, as we shall see!

13. CHEAP AND RELIABLE? • I’m partly to blame for this • In a series of articles published since 1997, I have demonstrated the statistically significant correlation between RAE results and citation counts – and have argued that citation counting could and should be used as a cheap and reliable substitute for expensive and subjective peer review • It’s possible (I don’t know) that Treasury civil servants read my articles and were persuaded by them

14. IF THIS IS WHAT THE CIVIL SERVANTS DID….. • …then they were being naïve • I made it clear that to reliably undertake such studies, you needed subject experts to carry out the analyses manually • Instead, the Treasury instructed HEFCE to go for a purely algorithmic approach

15. THE EVIDENCE • All studies carried out so far have shown a statistically significant correlation between RAE scores and citation counts • Subjects evaluated include: archaeology; business studies; genetics; library and information management; engineering; music; psychology • So, the whole gamut of pure science, engineering, social sciences and humanities – but not medicine yet

16. THE CORRELATIONS ARE HARDLY SURPRISING • Citation counts are a measure of impact • And impact is closely related to quality • Nonetheless, the two concepts are not synonymous • We don’t really know what the RAE peer panels were evaluating; “international standard” research = international impact?

17. BUT IF THE CIVIL SERVANTS WERE NAÏVE, SO ARE CRITICS OF CITATION ANALYSIS A long familiar catalogue of criticisms, aptly called “fairy tales” by Ton van Raan, head of CWTS in Leiden, the organisation managing the REF Pilot: • ISI’s Web of Knowledge has poor coverage of the humanities, computer science, conferences, monographs….. • Poor coverage of non-English language sources • Co-authors only included post-2000 • People with the same surname and initials • Same person using different names, e.g., after marriage

18. MORE FAIRY TALES There are also the issues of… • Clerical errors by ISI • Citing for the wrong reasons, e.g., to impress referees, because material is conveniently to hand…. • Not all influences are cited • Mistakes in citing, e.g., title, author surname… by the author • Deliberately controversial or erroneous articles designed to attract negative citations • Self-citation • Mutual citation within a group (“citation clubs”) • Deliberately choosing high Impact Factor journals to improve citation counts • Journal editors forcing authors to cite references from their journal

19. TYPICAL OF THE NAÏVE/UNINFORMED COMMENTS • Ron Johnston, former VC of Essex University, in THE, 8/5/08, p. 24 • “ISI data cannot be readily downloaded to be normalised to produce reliable measures” • “No evidence that citation scores and RAE scores are correlated” • “Evaluation can be done only by peer review”

20. YES, IT IS TRUE THAT… • WoS is not strong in its coverage of humanities journals • Not strong on non-English sources • The humanities, engineering, computer science are less dependent on journals than other subject areas • But the correlations are still there!

21. WHAT ABOUT THE REST? • Citing for the wrong reasons: rare and not statistically significant • Mis-citing: a fairly constant problem in all subject areas – no impact overall • Deliberately controversial articles: no increase in overall citations • Self-citation: no statistically significant effect • Mutual citation within a group: no evidence of this • Choice of high Impact Factor journals: article quality counts, not IF

22. POSSIBLE ALTERNATIVE SOURCES • SCOPUS – a serious contender; better coverage than WoS in engineering, conferences, etc., and more global in coverage Easier to analyse the data as well, for various technical reasons – less cleaning up needed. Main downside – currently untested; database does not go back that far • Likely to be a global deal (as with Web of Knowledge) so that HEIs can access SCOPUS at reasonable cost • Google Scholar – data is very dirty and there is duplication; data structure not suited for citation analysis; these points ruin its great potential for a wide range of subjects

23. A KEY POINT • No matter how convincing the objective arguments might be, if people don’t “buy into” the concept, there will be problems • Most academics simply don’t believe citation counts are an adequate substitute for peer review • So the current approach to the REF, combining bibliometrics with peer review, makes a lot of sense

24. WHERE WE HAVE ENDED • Civil servants were naïve to think simple citation counts would do the trick • Many academics are naïve in believing that citation counts cannot work in their subject area • The proposed new REF gives us the best of both worlds • But what weighting for bibliometrics and peer review? • Will a new Government scrap the REF altogether??

25. REF VERSUS RAE • REF – all data is in the public domain, so anyone can replicate and check if they’ve been calculated correctly; numbers are “objective” • RAE – decisions taken behind closed doors • HEFCE knows use of bibliometrics is controversial, and is determined to involve stakeholders at all stages of the pilot and implementation of the REF

26. ANY QUESTIONS?