Reflections on RAE 2008

1. Reflections on RAE 2008 Richard Thorpe Business & Management Sub-panel (i36) r.thorpe@lubs.leeds.ac.uk

2. General Issues RAE: Research Assessment Exercise Research quality over period 2001–07 to be assessed a longer period than usual Criteria and working methods published – lesson, read the exam question! A new grouping of sub-panels into main panel areas main panel 36 Funding to be affected from 2009-10

3. Make up of the panel • Panel members plus 3 – sub panels • Accounting and Finance • Economics and Econometrics • Information Management • Employment Relations • Entrepreneurship • Innovation • Leadership • Management Education and Management Learning • Management Science • Organisational Psychology • Public Management and Administration • Service Operations • Small Business • Strategic Management and Technology • Technology Management

4. Additional specialist advice • Corporate Social Responsibility • International Business • Tourism • The more mathematic aspects of Operational Research • The panel recognised that outputs couldn't always be characterised as falling neatly into disciplines and that in considering how best they might be distributed and assessed we were sensitive to other categorisations such as theme, sector or function

5. David Blackaby Jane Broadbent Ray Paul Structure of main panel I 34: Economics & econometrics (David Greenaway, Nottingham) 35: Accounting & finance (Andy Stark, MBS) Main panel I: Chair David Otley 36: Business & management (Mike Pidd, Lancaster) 37: Library and information mgt (John Feather, Loughborough)

6. Some history • Previous proper RAEs in 1992, 1996, 2001 • Applied a single rating to the whole of a submission • 2001: each output rated as • International, National or Sub-national quality • 5* implies • >50% output of International standard • Very little output rated below National level • Grade applies to all staff submitted, however

7. Main differences from 2001 • R monies 2001 • Same amount for all staff in department • Leads to cliff edge funding • R monies post-2008 • No cliff edge • Profile funding • Main panel I • Accounting & finance • Economics & econometrics • Business & management Studies • Library & information management • Staffing rules: no overlap period

8. Dual funding system for research HEFCE R monies Based on RAE Pays for research time & infrastructure Other research income, including research councils Pays for research projects & programmes Research income

9. RAE 2008 quality profile • Quality level: 4*, 3*, 2*, 1*, unclassified • Unclassified = zero stars • Percentage of research activity in each category • Based on FTE staff submitted • No requirement to state % submitted • To apply to • research output, • the research environment • indicators of esteem & impact

10. RAE 2008 quality criteria for outputs * To knowledge, theory, policy or practice

11. Judgements of quality • Originality • e.g. innovation or distinctiveness of the methodological approach • Data sets used • Research questions posed • Underlying hypotheses or theoretical framework • Significance • Insight and scope of coverage of the work • Impact on the discipline in the UK or internationally • Extent to which has opened up new areas of research • Current or potential impact on policy and practice • Rigour • Contextualisation of the work • Strength ,appropriateness and intellectual coherence • Extent to which the research outcomes are supported

12. Research outputs: 4 per person (70%) Research environment (20%) Esteem & impact indicators (10%) Quality profile Elements of assessment e.g. Research income PhD students Staff development Weighted and aggregated across each submission

13. RAE 2008: calculation of R monies e.g. Univ of North Midlands enters 50 FTE staff We do not know what the R values will be

14. RAE 2008 important dates • 2008 • 17th December profiles given to University Vice Chancellors for their institutions • 18th December Profiles published in the press for all institutions • 2009 • 4th January institutional feedback given to UoAs and output, environment and esteem indicators released • March Full submissions published for all institutions which will include papers submitted

15. Research outputs • For established staff: 4 per person expected • Unless work is exceptional • Or time out from research (apply pro rata rule) • Part-timers (apply pro rata rule) • Multi-authored work: avoid joint submission from same department unless work is exceptional • Different for early career researchers • Should flag up (possibly) • Exceptional work (e.g. potential 4* but not in top-ranked outlet) • Early career researchers • People who’ve had time out or part-time (equal opps) • Category C ??

16. The numbers • 90 submissions 97 • 3300 category A FTE staff (3500 on headcount) 3000 • 700 category B • 50 category C • 12,600 outputs 10,000 • 70% outputs 20% environment 10% esteem

17. Managing the process • Of the 12,600 about 10% were cross referred to i 34, 14% to i 35 and 4% to i 37 • All outputs were read in detail • Each panel member selected 4* outputs and these were discussed to ensure a common process of calibration • Some submissions were clearly not for Business and Management • For environment and esteem sub panel members were asked to read submissions and profiles were decide by the whole panel • A user member was a member of the panel and where submissions proved evidence of a research environment that was geared to policy and practice appropriately high marks were awarded

18. Research Income • In 2001 (a 5 year period) total research income was £200m • In 2008 (a 7 year period) total income was £360m OST/OSI funding £90m • In 2001 the no of research associates was 500+ • In 2008 the number of research associates was 470 • In 2001 PhD students numbered 2,600 • In 2008 PhD students numbered 3,450, 4.87 per research active FTE – so room for growth! Some high numbers (e.g. 12) were associated with low quality profiles

19. Issues for HRD • Issues relating to ‘field journals’ and their quality in respect to building a knowledge base • Other sub disciplines in a similar position • Hospitality and tourism • Entrepreneurship and small business • Issues relating to judging impact in relation to policy and practice • Theory vs Empirical vs Applied • Issues relating to Journal ‘quality’ impact factors and citations • Listings of journal quality • Published impact factor data • Location of journals • Quantitative verses Qualitative

20. Final Thoughts – does HRD need to ‘up its game’? • Something about the development of theory in the field – so what is being added to what? We need to be better connected to the knowledge base • Something about research design. Currently a great deal relies on the use of cases – how well do we theorise from the cases and can we be more innovative • Something about the originality of method – how can we be more rigorous in the methods we use and be able to make claims for policy and practice • Something about being picked up in citations and how we write, e.g. Jeff’s - ‘what a load of bollock’ paper: a story of the hairdresser and his suit’

21. Early career researchers • Central RAE definition • “Entered the academic profession on employment terms that qualified them for submission to RAE 2008 as Category A staff on or after 1st August 2003.” • Submission requirements: Business & Mgt • Appointed 1/8/03 to 31/7/05: normally 2 outputs • Appointed 1/8/05 or later: normally 1 output • Working papers may be submitted • The denominator will be adjusted so there is no point submitting more than this