Humanities and Metrics – a UK Perspective

1. Tony McEnery, Lancaster University Humanities and Metrics – a UK Perspective

2. Introduction • Political background • Regulatory background • The spark • The reaction • The counter reaction • The HEFCE/AHRC group • Possibilities • Findings • What does it all mean?

3. Political Background • The UK context – dual support • Both sides conduct peer review checking – this talk will focus on the generic support of research (QR) rather than project specific support • Desire to shift from a quality blind to a quality informed generic research support system • The desire to do this in a way which allocated on the basis of a fairly detailed review

4. Consequences £ UK Poland Australia Finland US Netherlands, Germany Depth of review

5. Regulatory Background • Assessments in 1986, 1989, 1992, 1996, 2001 and 2008 – primarily outputs based, peer review heavy • These assessments have teeth – in 2005-6 nearly £0.25B was allocated to A&H departments in the UK through this process • With each UK RAE there have been changes to how research quality is assessed and rated • What changes have been wrought over the years? A key change is that the system has downplayed the importance of quantity in research productivity in favour of measures of quality in research production over time.

6. The funding bodies are clear about what this has led to – an ever improved system which has become ever more “transparent, comprehensive and systematic”. Indeed, HEFCE claimed that the 2001 RAE exercise was “the most rigorous and thorough exercise to date” • We should be cautious, however – government bodies rarely admit mistakes, and often engage in policy based evidence making. With that said, members of academic community have become accustomed to the RAE to the extent that they believe it is indeed a good system, as will be seen.

7. However, prior to the system being threatened with radical change we should also recall that the system was widely vilified. • What caused an academic community that was dissatisfied with a research quality exercise system to change its mind so? Metrics.

8. The Spark • The RAE in the UK is undoubtedly a major burden on the University system. • It is also, however, a key way in which the funders can influence the behaviours of academics. • What if the behaviours had become so stable that one were able to measure compliance with them while removing the burden of assessment? • Lord May, a fierce critic of the RAE, argued that there was a positive correlation between the two arms of research support – research council funding corelated with QR funding.

9. So why bother? If one type of income can predict another, use one to allocate the other. • There were a variety of problems with May's argument. Crucially, it did not encompass the A&H subjects. • However, had the Philosopher's stone been found? • The government thought so.

10. The Reaction • After consulting a small number of Vice chancellors, the then Chancellor of the Exchequer (Gordon Brown) announced a shift to metrics based allocations in early 2006 • In the announcement the possible exclusion from this system of the A&H subjects was raised. This in turn raised the issue of how to deal with these subjects.

11. The HEFCE/AHRC Group • Due largely to the efforts of Sir Brian Follet a team was set up to answer two main groups of questions. • What are the distinguishing characteristics of excellence in research in the Arts and Humanities, and how might these be recognised and reinforced through any system of quality assessment and funding allocations to deliver the policy aim of government and the public funding bodies? What behaviours, and what types of research activity, should assessment and funding systems consequently seek to incentivise?

12. What metrics based approaches to assessing quality and allocating funding in the arts and humanities are possible now or could become so in the next few years? To develop a robust and effective approach, how broad a field of potential metrics and related indicators of quality should be considered?

13. The Counter Reaction • As the group set to work a general counter reaction began • Universities decided they did not like metrics • Academics decided they did not like metrics • What were their motives?

14. Universities • Financial – their calculations revealed that what might be an acceptable correlation at a high level of aggregation could translate into a significant shift of resource – it has been calculated that using research grants as a metric to drive QR income would have cut the grant of Nottingham by 12% but boosted that of Warwick by 26%. Such volatility was deeply unattractive to Universities.

15. Academics • The change of system may have required a change of behaviour, especially in the A&H subjects – grants are new for the A&H subjects. The previous change had become normalised and entrenched. Talk of 'perverse incentives‘ arose. • A perception that peer review was valuable – this has grown in strength, across all disciplines, since the shift to metrics was proposed. • Subject associations were influential in the current system – they would be less so in the new system. • An allergy to number? Cling to nurse for fear of worse?

16. Possibilities • The group swiftly concluded that one metric alone would be unsatisfactory for the A&H subjects – the diversity of the subjects was great. No clear, stable, correlation of one single X with research quality was visible across the whole of the A&H subjects. • Consequently we explored a wide range of metrics which we could viably use, and mapped those to where they had been used by others. • All might work to a degree for any given discipline. In what follows I note the measures proposed and note where such a measure is already in use elsewhere for these purposes

17. Research inputs Operating income - Germany, Netherlands Staff numbers - Germany, Netherlands, Poland Research income Third-party income (excluding RC income) - Germany, Poland, Australia Income from international sources (EU etc.)‏ - Germany, Poland, Australia RC income - Germany, Poland, Australia

18. Internationalisation of research Co-operation in RC-funded national networks - Germany, Finland Visiting lecturers/ incoming researchers - Germany, Poland Incoming graduate students - Germany Numbers of researchers in international networks - Finland, Poland, Slovenia

19. Researcher productivity Numbers of monographs - Netherlands, Finland, Poland, Australia, Slovenia Numbers of journal articles - Netherlands, Finland, Poland, Australia, Slovenia Publications in leading international journals - Germany, Slovenia Indexed in international bibliographic database - Slovenia Book chapters - Australia Published conference proceedings (refereed)‏ - Australia Bibliometric analyses, citations - Netherlands, Belgium Patents - Netherlands, Finland, Poland, Slovenia

20. Development of databases etc. - Slovenia Organisation of significant national or international conferences - Poland PhD completion rates - Netherlands, Finland, Poland, Australia Masters degrees - Finland, Poland, Australia

21. Measures of research esteem Number of RC reviewers - Germany Membership of national evaluation bodies - Poland Invited lectures/keynote speeches - Netherlands Awards and prizes - Poland

22. KT Measures Integration of research into teaching - Slovenia KT/relevance to industry - Finland, Poland, Slovenia Expert reports commissioned - Poland, Slovenia Wider dissemination of research findings - Poland

23. Problems range across all of these measures – the most essential one of which relates to the assessment of quality. How does one sift the wheat from the chaff? This is the key problem – otherwise one may simply stimulate the oversupply of low grade outputs etc. • The problem impacts less on some measures because they have peer review underlying them – notably RC income. • Yet even a measure such as journal publication may not be one which allows one to assume peer review has taken place on the output – different disciplines take quite different approaches to peer review at times.

24. The problem of quality dogs questions related to esteem in particular. They can become overwhelming in the arts.

25. The Findings • It is for reasons such as these that the group develop the distinction between a metrics led and a metrics informed approach to research quality assessment. Metrics informed uses metrics but moderates the findings based upon peer review. • This view appears to be in the ascendant in the UK – the solution appears to be applicable in the sciences as much as in the arts and humanities.

26. What Does it All Mean? • Measuring research quality is a difficult – and perhaps foolish – task. Typically it is an individual judgement. It is certainly a task which is political (at the macro level and the micro level). • Measuring research quality is about influencing research behaviours as much as measuring research quality in any objective sense – indeed measurement may at times be the antithesis of this. • However, in an environment in which public funds are dispensed in aid of research, we should expect those that give to want to influence behaviours and reward those behaviours they desire.

27. However, as an academic community we should always bear in mind a healthy scepticism about such an enterprise. • As individual academics we should value our academic freedom and, mindful of the potential consequences, do what we think is right rather than what the system asks us to do when needs be. • With that said, what the systems often ask us to do is so reasonable .... • Perhaps I have been habituated!