Excellence in Research for Australia

1. Excellence in Research for Australia ERA at UWA Presentation to the UMG, 19 November 2009 Lorna Robertson, Manager Research Assessment Unit, Research Services

2. Excellence in Research for Australia ERA is a research assessment exercise aimed at: • Evaluating a range of indicators of quality at the discipline level • Allowing for comparisons at an institutional level, national level, and international level • Identifying excellence in research performance • Identifying emerging research areas and opportunities

3. Units of Evaluation • Research assessment is discipline based: • Units of Evaluation, defined by Field of Research (FoR) codes, eg 0205 Optical Physics, 1110 Nursing • Allows for comparison across institutions • Disaggregates interdisciplinary research into component disciplines • The results of the ERA Trial will be publicly available at the national level • The results of the full ERA will be publicly available at the institutional level

4. 8 Clusters 1 Physical, Chemical & Earth Sciences (PCE) • FoRs: 02xx, 03xx, 04xx 2 Humanities & Creative Arts (HCA) • FoRs: 1201, 1203, 1205, 1299, 18xx, 19xx, 20xx, 21xx, 22xx 3 Engineering & Environmental Sciences (EE) • FoRs: 05xx, 09xx, 1202, 1204 4 Social, Behavioural & Economic Sciences (SBE) • FoRs: 13xx, 14xx, 15xx, 16xx, 17xx 5 Mathematical, Information & Computing Sciences (MIC) • FoRs: 01xx, 08xx, 1005, 1006, 1007, 1099 6 Biological & Biotechnological Sciences (BB) • FoRs: 06xx, 07xx, 1001, 1002, 1003, 1004 7 Biomedical & Clinical Health Sciences (BCH) • FoRs: 1101, 1102, 1103, 1105, 1107, 1108, 1109, 1112, 1113, 1114, 1115, 1116 8 Public & Allied Health Sciences (PAH) • FoRs: 1104, 1106, 1110, 1111, 1117, 1199

5. 22 Divisions, 157 Disciplines 22 Divisions 157 Disciplines • 0101 Pure Mathematics • 0102 Applied Mathematics • 0103 Numerical & Computational Mathematics • 0104 Statistics • 0105 Mathematical Physics • 0199 Other Mathematical Sciences • …

6. Indicators • Data collection and evaluation is based on four broad categories: Research quality, Research volume and activity, Research application, Research recognition • These broad categories translate to specific Indicators for which we have to submit data: • Eligible researchers • Research outputs • Research income • Applied measures • Esteem measures

7. Eligible Researchers • At census date (expected to be 31 March 2009), researchers must either: • Be employed at UWA; OR • Be a visiting, exchange, or seconded staff member employed elsewhere but working at UWA, AND show affiliation to UWA in their publication byline; OR • Be working for UWA in a non-salaried position (eg emeritus, honorary, adjunct, clinical, conjoint), AND show affiliation to UWA in their publication byline. • Data to be submitted for each eligible researcher includes up to three 4-digit FoR codes describing person’s research focus • Researcher and output data obligatory for all eligible T&R and RO staff • Honorary/Adjunct staff may be eligible, full criteria to be announced • Students are not eligible unless they meet the criteria (ie are also staff)

8. Research Outputs • Outputs must meet all criteria for eligibility (be research, published during the reference period, Include at least one eligible researcher as an author) • Each output must be assigned up to three 4-digit FoRs (one minimum) • Eligible outputs: • Books (A1) and revisions/new editions (A4) meeting criteria • Book chapters (B1) • Journal articles (C1) – only those in ARC ranked journal lists • Conference publications, full paper refereed (E1) – may only be those in ARC ranked conference list • Creative works (4 non-HERDC categories – UWA H and J1-J8): • Original Creative Works • Live Performance of Creative Works • Recorded/Rendered Creative Works • Curated or Produced Substantial Public Exhibitions and Events • Other non-print and non-HERDC categories to be advised in final Guidelines

9. Citation and Volume Analysis of outputs • All clusters will use volume analysis • Total number of research outputs against the FTE of eligible researchers by academic level and overall headcount (includes Honorary/Adjunct staff) • Most clusters will use citation analysis • Relative citation impact assessed against world and national benchmarks • Distribution of publications compared with world centile thresholds and the rest of the field • Distribution of papers against relative citation rate bands • All clusters will use ranked outlets • Ranked journals and/or ranked conferences

10. Peer Review of outputs • Some disciplines will use a peer review process to evaluate research outputs instead of citation analysis • Those in HCA, SBE, and possibly some disciplines in EE and MIC • Institutions identify a sample of outputs by 4-digit FoR to be peer reviewed • Peer review outputs sourced by the Library and held in the UWA digital repository • All creative works selected for peer review will be required to have a Research Statement • Addresses specific criteria, including an identification of the research component of the creative output

11. Income • Research income to be reported in a very similar way to HERDC (institutional attribution) • Australian Competitive Research Grants (Cat 1) • Other Public Sector Research Income (Cat 2) • Industry and Other Research Income (Cat 3) • Australian • International peer reviewed • International other • Cooperative Research Centre Research Income (Cat 4) • Tagged with 4-digit FoR codes

12. Applied Indicators • Applied measures: • Research commercialisation income • Patents • Registered Designs • Plant breeders’ rights • Possibly others • Must have an identifiable link to the research that generated them • Tagged with 4-digit FoR codes • Some attributed to the institution (eg Research commercialisation income, patents), others attributed to eligible researchers (eg Plant breeders’ rights)

13. Esteem • Esteem measures are being developed by ARC for ERA 2010 • Likely to involve selection of esteem measures from predefined list, for example: • Editorial role of A*/A journal • Editor of prestigious work of reference • Contribution to a ranked conference • Membership of a learned academy • Recipient of a nationally-competitive research fellowship • Membership of a statutory committee (PAH only) • Recipient of an Australia Council Grant or Fellowship • Esteem measures mostly get the relevant staff member’s FoRs • Esteem measures were not assessed in the Trial

14. Background Statements • Brief documents about research at UWA, providing context to the research data • Required for each 2-digit FoR code • Limited to 10,000 characters (approx 3-5 pages) • The statements address the following sections: • Overview – brief outline • Capacity – significant staff or resource changes • Environment – eg support for ECR and HDR • Collaboration – interdisciplinary, inter-institutional, international • Type – pure basic research, strategic basic research, applied research, experimental development • Other – anything else that should be taken into consideration • Only include information about research during the reference period, and not future directions

15. Evaluation • Research Evaluation Committees (RECs) will be formed for each cluster. • Comprised of internationally-recognised members with expertise in research evaluation • RECs will view a “dashboard” of indicator data reports by 4-digit FoR, including bibliometric data, (inter)national benchmarks, and research indicator data by FTE, year, level, etc • Some disciplines will be subject to a peer review stage in lieu of citation analysis • Institutions nominate a sample of outputs, to be made available for peer reviewers • Institutions will receive a rating from 1 (below average) to 5 (exceptional)

16. The ERA Trial: Two Clusters The ERA Trial assessed Cluster 1 and Cluster 2: Cluster 1 – Physical, Chemical and Earth Sciences (PCE) • 02 Physical Sciences • 03 Chemical Sciences • 04 Earth Sciences Cluster 2 – Humanities and Creative Arts (HCA) • 12 Built Environment and Design • 18 Law and Legal Studies • 19 Studies in Creative Arts and Writing • 20 Language, Communication, and Culture • 21 History and Archaeology • 22 Philosophy and Religious Studies

17. ERA Trial Reference Period • The staff eligibility census date was 31 March 2008 • The reference period for outputs was1 January 2002 – 31 December 2007 • The reference period for income and applied indicators was 1 January 2005 – 31 December 2007

18. Data Sources and Socrates • All data had come from difference sources, so it was necessary to bring the data together: • Outputs from UWA Publications Database, Research Services • Eligible researchers from HR • Staff FoRs from Planning Office • Income data from Research Finance Office • Applied data from Office of Innovation and Industry • The ERA data was loaded into Socrates for checking and correcting • Socrates underwent considerable development and testing to enable ERA data management and comparison • Academic Stewards were asked to check the data and advise changes • The final ERA data was loaded from Socrates directly into SEER (the online submission system)

19. ERA Trial Timeframe • The final Submission Guidelines were released by the ARC in March 2009 • The Evaluation Guidelines, outlining how the data would be assessed, were released by the ARC in April 2009 • Submission period for PCE was 1 June – 3 July 2009 • Submission period for HCA was 3 August – 21 August 2009 • The results of the PCE and HCA evaluations are due to be released in November and December respectively

20. ERA10: the “Big Bang” • The ARC has announced the submission period will commence in June 2010 • No information about closing date • The final Guidelines and Technical Specification documents are due to be released in December • This means a 6 month period to complete the submission: • Collect new data (eg ERA eligible staff outputs, new output categories, all esteem measures) • Check and correct data already held • Develop and test system functionality (developing and testing Socrates for esteem data collection, outputs, and other new indicators) • Select and source peer review items • Write research statements for creative works • Write background statements for all 2-digit FoRs

21. ERA10 • ERA reference periods are likely to be moved forward one year: • Staff census date: 31 March 2009 • Outputs: 1 Jan 2003 – 31 Dec 2008 • Income, applied: 1 Jan 2006 – 31 Dec 2008 • This has not been confirmed by the ARC • Greatly limits our ability to efficiently collect data • PCE and HCA largely complete • 2008 outputs, new indicators, reassess peer review sample • There will be 22-40 Academic Stewards, depending on the implications of the final Guidelines • Key staff with knowledge of the discipline, able to check data, identify gaps, draft background statements, and give advice for peer review selection where relevant • The RAU has 3 FTE staff for this project • Annual HERDC and other tasks ongoing during the period as well

22. Further information • ARC’s ERA website www.arc.gov.au/era • UWA’s ERA website www.research.uwa.edu.au/era • Contact Lorna Robertson lorna.robertson@uwa.edu.auor Lizzy Przywolnik elizabeth.przywolnik@uwa.edu.auin the Research Assessment Unit, Research Services