1 / 25

RAE Review: Interim feedback from facilitated workshops

RAE Review: Interim feedback from facilitated workshops. Jonathan Grant and Steven Wooding. Plan. Introduction Who we are and what we do Where we went and who we spoke to What we did and why we did it Results from 6/9 workshops Task 1: What is quality research? How should it be assessed?

marja
Download Presentation

RAE Review: Interim feedback from facilitated workshops

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. RAE Review:Interim feedback from facilitated workshops Jonathan Grant and Steven Wooding

  2. Plan • Introduction • Who we are and what we do • Where we went and who we spoke to • What we did and why we did it • Results from 6/9 workshops • Task 1: What is quality research? How should it be assessed? • Task 2: Assessing model systems • Task 3: Building a better system • Task 4: Implementation and implications • Some emerging observations • Preference for expert review, but needs refining • Increased transparency and clarity of process • Tension between comparability and appropriateness • Need structures to support change

  3. Who we are and what do we do • RAND Europe • Independent not-for-profit public policy think-tank • ‘Cousin’ of RAND; US based independent think-tank employing 1600 researchers • RAND Europe. Established in Leiden (NL) in 1992; Cambridge in 2001 • In UK programmes include Transport, Information Society, Health and R&D Policy • Current projects include VFM study on government department research for NAO and on scientific mobility for WHO

  4. Where we went and who we spoke to • Based on 6/9 Workshops • 93 people Positions • 26 Administrators • 38 Senior academics • 13 Academics • 8 Research Fellows • 8 Unclassified Fields • 32 Medicine, science & engineering • 16 Social Science • 15 Arts and Humanities • 29 Not research active • 1 Unclassified *Excludes Cambridge, Reading & Belfast (n=c50)

  5. What we did and why we did it • Facilitated workshops • Provide framework for structured thinking • Captured outputs in a standard and comparable form • Allows comparison between mixed and like groups of people (e.g., administrators only vs. mix of HoDs, fellows, and research officers) • Purpose is to listen, not evaluate

  6. Agenda • Task 1: What is quality research? How should it be assessed? • Task 2: Assessing model systems • Task 3: Building a better system • Task 4: Implementation and implications

  7. Task 1: What is high quality research? How should it be assessed? • Purpose • Stimulate wide ranging thinking on the most important aspects marking out high quality research and research assessment systems • Task 1 • Introductions • Identify 5 characteristics of high quality research • Identify 5 characteristics of research assessment system • Vote (5 votes for each; allocated as seen fit)

  8. ‘Top 10’ characteristics of high quality research

  9. ‘Top 10’ characteristics of research assessment systems

  10. Task 2: Assessing models systems • Purpose • Evaluate the strengths and weaknesses of 4 ‘model systems’: Expert review; Algorithms/metrics; Self-assessment; and, Historical Ratings • Task 2 • Split into 4 groups of c4-5 • 2 groups look at ‘good’ aspects of 2 systems each • 2 groups look at ‘bad’ aspects of 2 systems each • Also identify questions that need answering

  11. Algorithms / metrics ‘Good features’ • Transparent (4) • Objective (3) • Cheap (3) • Simple (2) ‘Bad features’ • Not suitable for all subjects (5) • Spurious objectivity (3) • Open to ‘game playing’ (2) • Metrics are proxy measures (2) Selected questions • How to recognise novelty? • What do you count? • How to ensure comparability?

  12. Expert review ‘Good features’ • Acceptable to community (5) • Based on specialised knowledge (3) ‘Bad features’ • Not comprehensive (3) • Not transparent (3) • Perceived bias (3) • In consistent (2) • Expensive (2) Questions • Who are the experts and how are they selected? • How do you build in an appeals mechanism • How do you recognise innovation?

  13. Historical ratings ‘Good features’ • Light touch (3) • Cheap (2) • Ability to plan (2) ‘Bad features’ • Inhibits change (3) • Low credibility (3) • Perpetuates silos (2) Questions • How do you take account of changing performance? • Who makes the judgement? (HEI or individual) • How far back?

  14. Self-assessment ‘Good features’ • Sensitive to discipline (2) • Formative – considers self (2) • Ownership & Trust (2) ‘Bad features’ • No cross discipline comparability (3) • Open to game playing (3) • No confidence in system (2) • Effort could be large (2) Questions • How would you police it? • Who sets the goals? • How do you penalise inflated results?

  15. Task 3: Building a better system • Purpose • Design ideal research assessment system • Task 3 • Split in to 3 or 4 different groups of c4-7 • Select ‘seed’ system from Task 2 • Build on this using aspects of other system • Present back to plenary

  16. Starting point Expert review (16/18 breakout groups) Self-assessment (1/18 breakout groups) 1 ‘failed’ (to reach a decision) (remember based on 6/9 workshops) Identifying a base system

  17. Refining the expert system • Transparency and clarity of process • Establish rules at outset • Early (i.e., up to 5 years before assessment) • Don’t change rules during process • Provide feedback • Legal contract between FCs and HEIs (rules won’t change, in return for guarantee of no challenge) • Clarity of funding outcome

  18. Refining the expert system • Longer time period between assessments • Review every 8-10 years • ‘Triggering’ mechanisms for interim review at 4-5 years. • Self declaration for new or emerging areas • Metrics for decline areas • Sampling (selective or random) with other departments

  19. Refining the expert system • Broader panels • Mirror Research Councils/AHRB • Broad or supra-panels have broad freedom to establish discipline specific rules • Sub-panel operate to those rules, reporting to supra-panel • Aims to solve tension between comparability and appropriateness

  20. Refining the expert system • Continuous rating scale • More grades • Summation of individuals ‘scores’ • Possibly based on ranking • Continuous funding scale • No (or reduced) step changes in funding

  21. Refining the expert system • Conceptual (non subject) AoAs • Based on research outputs • Formal sciences Theorems • Explanatory sciences Laws • Design sciences Technological rules • Human sciences Artifacts and knowledge • ‘Users’ from relevant AoA

  22. Refining the expert system • ‘Lay‘ Panel members • Possible experience lay members (such as judges) as panel chair • Rolling review • One discipline every year • Transfer fees • Reward departments that nurture future high flyers • Return all staff

  23. Task 4: Implementation and implications • Purpose • Examine system would be put into practice and evaluate repercussions • Task 4 • Working in same groups with ‘devil’s advocate’ • Identify 5 steps for implementation • Identify 5 things that could go wrong with the system • Identify 5 changes to the UK research as a result of the new research assessment system

  24. Implication of change • Some changes to UK research: • Ensure research capacity is activated wherever it is found • Better support and funding for younger researchers • Good support for emerging areas of research • Equal opportunities are improved • Better recognition of interdisciplinary research • Funding directly following excellence • Less obsession with the RAE

  25. Emerging observations • Preference for expert review, but needs refining • Increased transparency and clarity of process • Tension between comparability and appropriateness • Need structures to support change But …. • Excludes Cambridge, Reading and Belfast • Not analysed by discipline and or profession

More Related