The Research Excellence Framework: Assessment Framework, Guidance, and Panel Criteria

1. These slides have been produced by the REF team, and were last updated on 30 January 2012 • They provide a summary of the assessment framework and guidance on submissions and the REF panel criteria • They may be used freely for the purposes of briefing any interested parties on the REF • Should anyone other than the REF team edit these slides, then any slides that you add or amend with your own interpretation of the REF guidelines, should not be presented with the REF logo and design. It should be clear to the audience that such slides have not been produced by the REF team.

2. The Research Excellence FrameworkAssessment framework, guidance on submissions and panel criteria

3. Presentation outline • Overview • REF panels • Staff • Outputs • Impact • Environment

4. Overview

5. Overview: Purpose of the REF The REF replaces the RAE as the UK-wide framework for assessing research in all disciplines. Its purpose is: • To inform research funding allocations by the four UK HE funding bodies (approximately £2 billion per year) • Provide accountability for public funding of research and demonstrate its benefits • To provide benchmarks and reputational yardsticks

6. Overview: Principles of the REF The REF is a process of expert review. The assessment is founded on the professional judgement of discipline-based expert panels. The conduct of the REF is guided by the principle of: • Equity: All types of research and forms of output in all disciplines shall be assessed on an equal basis • Equality: HEIs are encouraged to submit the work of all their excellent researchers • Transparency: The assessment criteria, procedures and outcomes to be published in full

7. Overview: Key changes since 2008 RAE • Inclusion of assessment of impact • Fewer UOAs/panels, operating more consistently • Strengthened equality and diversity measures • Revised eligibility criteria for staff • Addition of (limited) use of citation data in some UOAs • Removal of ‘esteem’ as a distinct element • Revised approach to ‘environment’ and data collection • Increased ‘user’ input on panels; and an integrated role for additional assessors • Publication of overall quality profiles in 1% steps

8. Overview: Guidance and criteria Comprehensive information and guidance is set out in: • Assessment framework and guidance on submissions (July 2011): • Sets out the information required in submissions and the definitions used • Panel criteria and working methods (Jan 2012): • Sets out how panels will assess submissions • Further supplementary guidance will be published during 2011-12 on technical and procedural matters

9. Overview: The assessment framework 65% 15% 20%

10. Overview: Submissions • Each submission in a UOA provides evidence about the activity and achievements of a ‘submitted unit’ including: • Staff details (REF1a/b/c) • Research outputs (REF2) • Impact template and case studies (REF3a/b) • Environment data (REF4a/b/c) • Environment template (REF5) • A submitted unit may, but need not, comprise staff who work within a single ‘department’ or organisational unit

11. Overview: Multiple and joint submissions • Institutions will normally make one submission in each UOA they elect to submit in • Joint submissions are encouraged where this is an appropriate way of describing collaborative research • Multiple submissions only byexception and with permission from the REF manager: • Where an HEI also makes a joint submission in that UOA • Where HEIs have merged • In Sub-panel 28 where one submission is in Celtic studies • Where a sub-panel considers there is a case given the nature of the disciplines covered. These sub-panels are listed in the panel criteria statements.

12. Overview: Timetable

13. Overview: Publication of results • The primary outcome of the REF is an ‘overall quality profile’ to be awarded to each submission: • Using the same scale as RAE2008, but in steps of 1% • To be published in Dec 2014 • Further reports and feedback during 2015: • An overview report by each main panel, including observations by each of their sub-panels • Concise feedback on submissions, to the heads of HEIs • The output, impact and environment sub-profile for each submission • A report by the Equality and Diversity Advisory Panel • Submissions will be published (except for confidential or sensitive information)

14. Overview: Example of a quality profile

15. REF panels

16. REF panels: Main and sub-panel roles There are 36 sub-panels working under the guidance of 4 main panels. Membership is published at www.ref.ac.uk

17. REF panels: Main Panel A

18. REF panels: Main Panel B

19. REF panels: Main Panel C

20. REF panels: Main Panel D

21. REF panels: Additional assessors Additional assessors will be appointed to extend the breadth and depth of panels’ expertise: • Both ‘academic’ assessors (to assess outputs) and ‘user’ assessors (to assess impacts) will be appointed • Assessors will play a full and equal role to panel members, in developing either the outputs or impact sub-profiles. They will be fully briefed, take part in calibration exercises and attend meetings: • Some appointments in 2012, where the sub-panel has already identified a clear gap in expertise required to assess outputs or impact • Further appointments in 2013, in the light of the survey of institutions’ submission intentions

22. REF panels: UOA descriptors and boundaries • UOA boundaries are not rigidly defined and all panels expect submissions to cross boundaries with other UOAs • Sub-panel membership is broad and includes members with interdisciplinary expertise • Assessors will be appointed to extend their breadth and depth of expertise. Where there is a significant overlap between UOAs, ‘joint’ assessors may be appointed to work with more than one sub-panel. • The sub-panels prefer to assess submitted work within the sub-panel but may, exceptionally, cross-refer parts of submissions to other sub-panels for advice. The original sub-panel remains responsible for recommending the outcome

23. REF panels: Main panel working methods • Each main panel will work with its sub-panels to apply common assessment criteria • Panels will undertake calibration exercises and reviews of emergent outcomes to make sure overall assessment standards are applied consistently

24. REF panels: Sub-panel working methods • The sub-panel chair will allocate work to members/assessors with appropriate expertise • All outputs will be reviewed in sufficient detail to contribute to the outputs sub-profile • Each case study will normally be assessed by at least one academic member and one user member/assessor • Sub-panel members may raise audit queries with the REF team for investigation • The panel secretariat will minute panel procedures

25. Staff

26. Staff: Category A and C staff • HEIs select which staff to include in submissions: • Category A staff: Academic staff with a contract of at least 0.2 FTE, on the payroll of the HEI on 31 Oct 2013, with a primary employment function of either ‘research only’ or ‘teaching and research’ • Category C staff: Staff employed by an organisation other than an HEI with a contract or job role including research, and whose research is primarily focused in the submitting unit on 31 Oct 2013 • (Category C staff will contribute to the quality profile but not the volume measure for funding purposes) • Research assistants are eligible only by exception

27. Staff: Codes of practice on staff selection • The funding bodies encourage HEIs to submit the work of all their excellent researchers, and HEIs have legal obligations affecting their staff selection procedures • Each HEI is required to develop, document and apply a code of practice on the fair and transparent selection of staff for the REF: • Must be signed off by the head of the institution and submitted to the REF team by 31 July 2012 at the latest • The REF Equality and Diversity Advisory Panel will examine these for adherence to the guidance • They will be published at the end of the assessment process

28. Staff: Codes of practice on staff selection • Codes should demonstrate fairness to staff by addressing the principles of: • Transparency: clearly setting out the procedures for staff selection, and communicating these to all eligible staff • Consistency: applying consistent procedures across the institution • Accountability: clearly defining responsibilities for decisions, with appropriate training for those involved • Inclusivity: promoting an inclusive environment, with robust procedures for staff to disclose individual circumstances • An Equality Impact Assessment should inform the code and be kept under review at key stages of the selection process

29. Staff: Individual staff circumstances • Up to four outputs must be listed against each member of staff • This can be reduced without penalty where circumstances have constrained an individual’s ability to work productively or produce four outputs during the assessment period: • A wide range of circumstances will be taken into account • With as much clarity as possible about how many outputs may be reduced without penalty • To be treated consistently across the exercise • The allowances for maternity, paternity and adoption leave have been revised following consultation

30. Staff: Individual staff circumstances

31. Staff: Clearly defined circumstances • ‘Tariffs’ define the number of outputs that may be reduced without penalty • These will be applied consistently by all REF sub-panels • Circumstances may be combined up to a maximum reduction of three outputs • Where an individual has a combination of clearly defined andcomplex circumstances, these should be submitted collectively as ‘complex’

32. Staff: Clearly defined circumstances:Early career researchers Staff eligible for selection who started their careers as independent researchers on or after 1 August 2009

33. Staff: Clearly defined circumstances:Part-time working, secondments and career breaks For part-time working, the equivalent ‘months absent’ should be calculated

34. Staff: Clearly defined circumstances:Maternity, paternity and adoption leave • Changes were made to these arrangements, following consultation • Individuals may reduce the number of outputs by one, for each discrete period of: • Statutory maternity leave or statutory adoption leave taken substantially during the period 1 January 2008 to 31 October 2013, regardless of the length of the leave.  • Additional paternity or adoption leave lasting for four months or more, taken substantially during the period 1 January 2008 to 31 October 2013. • Constraints relating to pregnancy, maternity, paternity, adoption or childcare that fall outside of – or justify the reduction of further outputs in addition to – these allowances may be submitted using the arrangements for complex circumstances.

35. Staff: Clearly defined circumstances:Other circumstances in UOAs 01-06 • Outputs may be reduced by up to two for the following: • Category A staff who are junior clinical academics (and have not gained a CCT or equivalent prior to 31 October 2013. • Category C staff who are employed primarily as clinical, health or veterinary professionals

36. Staff: Complex circumstances • EDAP will consider all cases of complex circumstances • EDAP will make recommendations to the Main Panel Chairs, who will decide • Sub-panels will be informed of the decisions and will assess the remaining outputs without penalty • Information will be kept confidential to EDAP, Main Panel Chairs and the REF team • ECU will publish a range of worked examples – including EDAP’s rationale

37. Outputs

38. Outputs: Eligibility • Outputs must be: • a product of research (as defined for the REF) • first brought into the public domain between 1 January 2008 and 31 December 2013 • Authored/co-authored by the member of staff against whom it is listed (regardless of where they were employed prior to the census date)

39. Outputs: Outputs ‘pre-published’ before 2008 • A change in the guidance on pre-publication of outputs has been made, following consultation: • An output first published in its final form during the REF publication period, but ‘pre-published’ in 2007 – is eligible for submission to the REF, provided that the ‘pre-published’ output was not submitted to the 2008 RAE. Examples may include: • An online first article or preprint • A preliminary version or working paper

40. Outputs: Range of output types • Outputs may include but are not limited to: printed or electronic publications, materials, devices, images, artefacts, products, buildings, confidential or technical reports, patents, performances, exhibits or events • All forms of outputs shall be assessed on a fair and equal basis

41. Outputs: Co-authorship • Co-authored outputs can be listed against more than one co-author when returned in different submissions • An output can be listed no more than twice within the same submission andmust be accompanied by a statement explaining the substantial and distinctive contribution of each of the submitting authors

42. Outputs: Double-weighting • Institutions may request ‘double-weighting’ for outputs of extended scale and scope • Requests for double weighting should be accompanied by a supporting statement • Sub-panels will assess the claim for double-weighting separately from assessing the quality of the output • If a sub-panel accepts a request for double weighting, the output will count as two outputs in both a submission and in the calculation of the outputs sub-profile

43. Outputs: Additional information on outputs • Panels may make use of additional information in the form of citation data and details provided by the HEI • Sub-panels have requested further information on the research process or content where this is not evident from the output • Sub-panel will not make use of journal impact factors, rankings or lists, or the perceived standing of the publisher, in assessing the quality of research outputs

44. Outputs: Citation data • The following sub-panels will make use of citation data: • Main Panel A: Sub-panels 1-6 • Main Panel B: Sub-panels 7-11 • Main Panel C: Sub-panel 18 • These sub-panels will use only the citation data provided by the REF team, sourced from Scopus (except for Sub-panel 11 that will, in addition, use Google Scholar) • All other sub-panels will neither receive nor make use of citation data • None of the sub-panels will use journal impact factors, journal rankings or other forms of bibliometric analysis

45. Outputs: Citation data • Where used, sub-panels will consider citation data as follows: • As one indicator of ‘academic significance’. Expert review remains the primary means of assessing outputs • Panels will assess all outputs on an equal basis regardless of the availability of such data. They recognise the limitations of citation data – especially for recently published outputs – and will have due regard to potential equality implications • In the relevant UOAs only, citation counts from Scopus will be provided to panels on a consistent basis. HEIs will be able to verify the outputs have been matched correctly, and view citation counts on the submission system • The funding bodies do not sanction or recommend that HEIs rely on citation data to inform the selection of staff or outputs for their REF submissions

46. Outputs: Assessment criteria * Each main panel provides a descriptive account of the criteria

47. Impact

48. Impact: Definition of impact for the REF • An effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia • Impact includes an effect, change or benefit to: • The activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding • Of an audience, beneficiary, community, constituency, organisation or individuals • In any geographic location whether locally, regionally, nationally or internationally • It excludes impacts on research or the advancement of academic knowledge within HE; and impacts on teaching or other activities within the submitting HEI

49. Impact: Range of impacts • Panels recognise that Impacts can manifest in a wide variety of different ways, may take many forms and occur in a wide range of spheres • Examples of impact may include: • Impacts on public policy and services, • Impacts on society, culture and creativity, • Impacts on practitioners and services, • Impacts on the environment, • Impacts on the economy

50. Impact: Examples of impacts: Main Panel A