Presentation to Forum on “ Measuring Research Performance: What are the Options”?

1. Reflections on the Independent Strategic Review of the Performance-Based Research Fundby Jonathan Adams Presentation to Forum on “Measuring Research Performance: What are the Options”? 18 September 2008

2. Presentation Outline • Origins, Aims and Implementation • PBRF Evaluation Strategy – the review in context • Method and approach • Key review questions • Findings – overall effects; differential effects; unintended consequences; and suggestions for refinement • Strengths of the PBRF to protect • Dissemination and Utilisation • Acknowledgements

3. Performance-Based Research FundOrigins, Aims and Implementation Origin of PBRF – Key recommendation of the fourth Report of the Tertiary Education Advisory Committee, 2001; detailed design undertaken with the sector and set out in Investing in Excellence. Principles, Metrics, aims Aim – To encourage and reward research excellence within the NZ Tertiary Education Sector Implementation – funding available since 2003 and in 2007 - $232 million distributed.

4. PBRF Evaluation Strategy – the review in context • Ministerial directive to periodically evaluate PBRF when first introduced • 3-phased PBRF evaluation strategy developed in 2003 • Phase one – focus on implementation • Phase two – focus on emerging effects • Phase three – focus on outcomes and cost/benefit • Areas of focus in first and second phases of evaluation strategy centre on concerns raised during original policy design • Review completes the second phase of evaluation strategy

5. Method and Approach • Initial Design Phase – review of the lessons learnt from evaluation of UK RAE; intervention logic and symposium; consultation with sector on appropriate approach to evaluation • Review approach guided by principles • independence and credibility of reviewer • maximize use of existing secondary data • cognisant of participation costs for sector • Inclusion of all TEO types for participating in PBRF • Cognisant that it is too early to index the systematic impacts of the PBRF • Dr Jonathan Adams was the reviewer and his work was supported by a Review Advisory Group • Evidence collected via several complementary routes • Review report synthesised all material analysed by Dr Adams

6. What Questions did the Review Address? • Effects of PBRF on NZ research base in TEOs • Differential effects of PBRF on modes of research, disciplines, institutions and researchers • Improvements to consider

7. Findings – Effects of PBRF on NZ Research Base in TEOs • PBRF has been effective in delivering beneficial outcomes • Financial – directed more research resources to institutions delivering better research • Reputation – increased the quantity and quality of information about relative research quality at institutional and subject level in NZ TEOs • Formative – driven improvement in management, culture and awareness and priority given to research • These outcomes are expected to result in an overall enhancement in the performance of the NZ research base and in the country’s relative international standing

8. Findings - Differential Effects of the PBRF on Some Tertiary Sector – ITPs, Wananga and Small PTEs • Differential and distinctive missions of TEOs may mean that the PBRF is not the best mechanism for some sectors • Dr Adams suggests alternative research support for some tertiary sectors – ITPs, Wananga and PTEs

9. Findings - Differential Effects of PBRF on Subjects and Modes of Research • Does the PBRF model of assessment privilege research subjects, modes and outputs that most easily fit with the traditional western scientific paradigm? • PBRF model could work better for humanities / social science disciplines; professional schools and longer-term research with enhanced diversity of panel composition; panels continue to apply appropriate assessment methodology within their subject area; and conceptualising quality whatever the output mode • Risky and innovative research – evidence suggests that by rewarding excellence, the PBRF recognises and values such research

10. Findings - Differential Effects of PBRF on People • Available data suggests that the PBRF assessment process does treat different groups of researchers equitably • New and emerging researchers have a complex experience. At TEOs participating in both the 2003 and 2006 PBRF census the pool of staff under the age of 35 shrank by 14%. The management of new researchers may affect the long-term sustainability of the tertiary academic workforce and the future welfare of research quality • Maori researchers - disproportionately ‘new’ to research; and EPs lower average outcomes but many in fields that attracted lower subject weightings • Pacific Peoples’ experiences of the PBRF – data too sparse for sound conclusions

11. Findings – Unintended Consequences • PBRF may create undue focus on staff with established research programmes • Some TEOs inappropriately use PBRF quality evaluation results as a staff appraisal substitute • Early indications of frustration over lack of significant benefit after raising research quality. The balance between effort and reward could be adjusted to address this.

12. Findings - Operational Refinements to Consider • Who and what is evaluated? • Restrict eligibility to a core group of permanent academic staff around whom the research system pivots • Consider moving to group as unit of assessment post 2012 • Better recognise applied research such as evidence – based policy work • How evaluations are conducted – various improvements to panel processes suggested. • How different research activities are weighted to ensure emphasis on increasing quality • Increase the financial and scoring benefits of the “A” • Reduce the weighting for research degree completions • Review subject area weightings • How results are handled – dissociate scores from staff names and improve reporting.

13. Findings – Strengths of the PBRF to Protect • Additional funding is needed for the pace of improvement required to sustain and increase research excellence • PBRF must remain focused on identifying and funding research excellence – avoid adding potentially conflicting goals to its mandate. • Developing a mechanism of assessment that focuses on a healthy research environment as well as on excellent individual research

14. Dissemination and Utilisation of the PBRF Review Report? • PBRF Review report is an independently produced source of evidence for use by: • PBRF Sector Reference Group • Tertiary Education Commission • Minister for Tertiary Education • Tertiary Education Sector

15. Acknowledgements • Informants • PBRF Review Advisory Group • Dr Jonathan Adams • Contributors of secondary data