REF Impact Pilot: Challenges and Benefits of Measuring Research Impact at University of Glasgow

1. REF Impact Pilot Laura Tyler Marketing & New Media Manager University of Glasgow

2. Glasgow’s involvement • Clinical Medicine (17 case studies) + Earth Systems (4 case studies). • Writing team formed • Reporting to a Steering Group (SG) of senior academics chaired by VP R&E. • Process was: • Trawl for stories led by SG • Interviewed researchers where possible • SG inputted to the stories • Staff survey (Clin Med) for impacts of indicators • External reading team from user community was extremely helpful

3. The exercise • Extremely positive, if challenging, experience. • Created excellent examples of the benefits of our own research. • Raised (really raised!) awareness. • Created a strong academic/administrative team. ….there were of course some challenges

4. Practical issues with the pilot Impact statement • Questions were open to interpretation Case study template • Split sections caused confusion – not logical flow and caused repetition Duplication • Across all three elements – impact statement, case studies and research context document

5. Labour intensiveness • Labour-intensive for the staff involved (Academics, R&E, Faculty Contacts). • Collection and collation of the material we required to submit to HEFCE • the iterative nature of the drafting process. • It will be important to manage colleagues’ expectations from the outset.

6. External support • Challenging to engage external contacts when preparing case studies and gathering evidence: - identifying the people - changes of staff in users/lack of corporate memory - this includes the engagement of academic staff who have now left GU.

7. Identifying/quantifying impact • Capturing the extent of the impact and the precise time when that impact occurred. • Meeting the pilot definition of impact – the academic understanding of impact was often not aligned with the HEFCE criteria. • We need to support colleagues in identifying and describing the impact of their research once the final HEFCE criteria are known. • Difficult to quantify the impact of research when technologies have been sold on to third parties.

8. Commercial sensitivities • We were allowed to flag case studies as being commercially sensitive, however we cannot guarantee confidentiality. • In a number of cases, research users would not provide us with information.

9. HEI control • To a large extent, Universities are reliant on another party to create the Impact (companies/other organisations). • We are limited in the control we have on the nature and extent of the impact achieved.

10. ALL SORTS OF OTHER FACTORS Jobs New Products New services Turnover Policy Changes Health Impacts Cultural Impacts Profit R&D expenditure % turnover from new products/ services Networking/ Events Consultancy CPD Collaborative Research Contract Research Licensing Company Creation Teaching Publications Start-up Spin-out Society Government Policy-makers Small Companies Big Companies Processes New Knowledge Materials Knowledge Research Researchers Technology £ $ Know-how Innovation Skills ALL SORTS OF OTHER FACTORS KE USERS / BENEFICIARIES IMPACTS RESEARCH OUTPUTS KE CHANNELS

11. HEFCE guidance • Guidance was not always entirely clear and sometimes open to interpretation, with definitions being iterated as we went (as expected in a pilot). • Important for the final REF impact criteria to be as clear and unambiguous as possible, to minimise the scope for confusion and hopefully the pilot has moved us towards this.

12. Subjectivity • How will the panel weight different ‘types of impact’? • Quality of story writing will also be a major factor • Polarised views within our working group and our reading group “A fantastic example of work” “I’m not sure it’s eligible”

13. In summary • We found the exercise extremely useful in gathering impact stories, which are hugely valuable in themselves. • We found that the exercise helped to educate the broader community regarding impact. • We are concerned that impact has more to do with our users than this exercise recognises – to some extent we are trying to measure the performance of others. • The subjective nature of comparing stories makes us a little nervous. • The results of the Pilot were not surprising

14. Thank you laura.tyler@glasgow.ac.uk