Impact AND THE RESEARCH EXCELLENCE FRAMEWORK

1. Impact AND THE RESEARCH EXCELLENCE FRAMEWORK

2. Incorporating impact in the REF______________________________________________________________________________________________________________________________ • The aim is to identify and reward the contribution that high quality research has made to the economy and society: • Making these explicit to the government and wider society • Creating a level playing field • Encouraging institutions to achieve the full potential contribution of their research in future

3. Impact: initial consultations______________________________________________________________________________________________________________________________ • Widespread acceptance of the principle of incorporating impact in the REF, and agreement that the impact assessment should: • Be based on expert review • Review historical impacts, not predict future impact • Focus on the impact of submitted units’ research, not individual researchers • Be underpinned by high quality research • Take a wide view of impact, inclusive of all disciplines

4. The impact pilot exercise______________________________________________________________________________________________________________________________ • Tested and developed a case study approach to assessing the impact of research • Five units of assessment (UOAs) • 29 UK higher education institutions each submitting to 2 UOAs • Each submission included: • An ‘impact statement’ for the submitted unit as a whole • Case studies illustrating examples of impacts achieved (a total of one case study per 10 research staff) • Impacts that occurred during 2005-09, underpinned by research since 1993

5. The pilot panels______________________________________________________________________________________________________________________________ • Membership drawn from academia and research users from the private, public and third sectors • The panels tested the methodology by: • Assessing the case studies in terms of ‘reach and significance’ of the impacts • Considering the wider ‘impact statements’ • Producing impact profiles • Reflecting on the process, identifying issues and making recommendations on how to improve the process

6. Key findings______________________________________________________________________________________________________________________________ • The process makes explicit the benefits that research in each discipline brings to society

7. Benefits of research______________________________________________________________________________________________________________________________

8. Key findings______________________________________________________________________________________________________________________________ • The process makes explicit the benefits that research in each discipline brings to society • It is possible to assess the impact of research, through expert review of case studies • A number of refinements are needed for full implementation • A generic approach is workable, with scope for REF panels to tailor the criteria as appropriate to their disciplines • The weighting should be significant to be taken seriously by all stakeholders, and needs careful consideration

9. Decisions on impact______________________________________________________________________________________________________________________________ • Published decisions available online at: www.ref.ac.uk • Weightings of the three elements: outputs, impact and environment • Broad framework for the assessment of impact

10. The REF Framework______________________________________________________________________________________________________________________________

11. Broad framework ______________________________________________________________________________________________________________________________ • Based on expert review of case studies (1, + 1 in 10) • May include any social, economic or cultural impact or benefit beyond academia, arising from excellent research, that has taken place during the assessment period • Also include information about how the unit has supported and enabled impact during the assessment period • Assessment of reach and significance • Involvement of research-users • Further guidance: July 2011 and January 2012

12. BIBLIOMETRICS IN THE REF Royal Society of Medicine University Health & Medical Librarians’ Group 7 March

13. Contents ______________________________________________________________________________________________________________________________Contents ______________________________________________________________________________________________________________________________ • Background to bibliometrics • Bibliometrics pilot • Preliminary feedback from sub-panels • Next steps

14. Background to bibliometrics ______________________________________________________________________________________________________________________________ • Count the number of citations gained by each item under assessment • Time dependence • Subject dependence • Benchmarking of raw counts

15. Bibliometrics pilot ______________________________________________________________________________________________________________________________ • 22 UK HEIs • All UOAs with reasonable coverage in the bibliometric databases • Trialled several approaches • Address based model • Author based models • Just RAE submitted people • Selected papers

16. Conclusions from the pilot ______________________________________________________________________________________________________________________________ • Fully inclusive models difficult • Institutional data verification • Defining subject areas • Cannot use formulaically - though significant potential to support assessment process • Equalities implications

17. REF Sub-panel feedback ______________________________________________________________________________________________________________________________ • Interest from main panels A and B • Little interest from C and D • Range of options • Simple counts and guidance on use • Fully normalised / benchmarked

18. REF Sub-panel feedback ______________________________________________________________________________________________________________________________ • Not for use formulaically • As a means of informing peer review • Cross-check/tie break • Reflects academic significance only • Supporting evidence only • Concerns about: • Differential coverage • Discouraging applied/translational work • Timeliness of data • Clarity of guidance on submissions • Normalisation of scores • Time lag on indicators

19. Next steps ______________________________________________________________________________________________________________________________ • Procurement of citation data • Discussions with several suppliers • REF steering group will take a decision informed by • Procurement discussions • Panel feedback • Decision published in July

20. Thank you______________________________________________________________________________________________________________________________ Kim Hackett Policy Adviser HEFCE 0117 931 7267 k.hackett@hefce.ac.uk David Mawdsley Analyst HEFCE 0117 931 7365 d.mawdsley@hefce.ac.uk