Understanding Research Performance The Evolution of UWA’s Research Management Tool, Socrates

1. Understanding Research PerformanceThe Evolution of UWA’s Research Management Tool, Socrates Academic Consortium 21 International Forum October 18-21, 2010. N.G. Mast, D.G. Glance & R.A. Owens University of western Australia Presented By Natalie Mast (natalie.mast@uwa.edu.au)

2. What is Socrates? • In late 2006 The University of Western Australia (UWA) launched Socrates, an online application designed to draw data from key research information systems, in order for the University to prepare portfolios for the Government’s Research Quality Framework (RQF).

3. What does Socrates do? • The system draws data from a number of core UWA databases, as well as two ISI databases and Scopus. • Socrates is then able to collate data for individual staff members to create a detailed research profile, showing an individual’s publications, citations, grant income and HDR load.

4. How does Socrates measure research Performance? • Having established a profile for individual researchers, it was then necessary to provide a measure, whereby the research performance of staff could be judged. • A Socratic Index was introduced to help the University gauge the level of research productivity for its staff.

5. The Socratic Index • The Socratic Index is calculated using data from a six year period (2004-2009). • Previously, there were 3 Socratic indexes in use within Socrates. • All three indexes relied on two forms of data, Research Outputs (ROPs), namely publications, and Research Inputs (RIPs), that is, competitive grant income (categories 1, 2, and 4 of the HERDC grants data collection). • The difference in the three measures rests in the manner in which publications are weighted.

6. The Original Socratic Indexes Government SI reflects the way in which income is gained via the Federal Government’s block grant schemes; It includes only HERDC publications and uses the same weightings for publications as the HERDC. UWA SI reflects the way in which internal funding flows according to research publication activity; a much wider range of publications are included, for example, scholarly reviews, musical compositions, and book reviews. There is a significant increase in the weighting of books, while the points attributed to conference papers are halved when compared to Government points. TISI SI relates only to performance in publications that are indexed through the Thomson Reuters ISI Web of Science.

7. Socratic Index II • In January 2009, UWA launched Socrates II, with a new, single Socratic Index (SI II) that focuses upon a six year research period, currently 2004-2010. SI II was designed to reflect the research aims of the University and measure the performance of our researchers accordingly. Overall, a more holistic measure of research performance was desired, so in addition to measuring research income and publications, higher degree by research (HDR) figures have been included. • To determine the SI II, Research Output Points (P), Research Input Points (I), and HDR completion points (S) for the five year are summed, and can be represented thus: • SI II= P1+I1+S1

8. SI II Indicators

9. Tabs for individuals within Socrates

10. Analysis Tab • Within the Analysis Tab, in addition to providing an individual score for each researcher, Socrates is also able to produce averages for: • The University • Academic Levels • Faculties • Schools

11. History Tab

12. Protecting Staff Privacy • Socrates has been designed with a number of differential access levels. • Individual researchers can see their own detailed record as well as all average figures. • Heads of School can see all the detailed results of their school members, as well as all averages. • The same privileges exist for Deans at a faculty level. • Members of the University Executive, the University’s Tenure and Promotions Committee and a small number of central administrative staff have access to all records.

13. Proving Assumptions about Research Performance • The University was for the first time able to conclusively verify a number of assumptions. Socrates shows that, at the University level, the productivity of staff members increases with their academic role level. The same holds true at the faculty level. • Socrates allowed us to chart the differences in performance by academic role. *LVLA = Associate Lecturer, LVLB = Lecturer, LVLC = Senior Lecturer, LVLD = Associate Professor, LVLE = Professor.

14. Using Socrates to chart the impact of R&D Policies • Feng et al. (2004, p. 182) claim that ‘the key factor which will bring out an increase in strength (in R&D activities) is the efficient management of R&D activities’. • Socrates plays a significant role in managing research performance activities, in part due to its adaptability. • When the University implements policies aimed at improving its research performance, the Socratic Index formulas can be recalculated to take into account the aims of the University. • The University Executive can then gain an understanding of its strengths and weaknesses in regards to its new aims, and staff can quickly begin to change their behaviour accordingly.

15. Change in the SI II from 2009 to 2010 *Scores for 2009 are based upon data from the period 2003-08, while scores for 2010 are generated using data from 2004-09

16. UWA Indexed Articles in the Web of Science* 2004-2009 • The introduction of the Socratic Index, has lead to a change in publication behaviour. • Staff members are now more aware of the influence of citation data, and understand that, as the University strives to improve its position in the Shanghai Jiao Tong Ranking, a greater importance is now placed on journal articles indexed by ISI. *SCI Expanded; SSCI and HCI only

17. Patterns of Publication

18. Benefits for Research Management • The ability to compare the research productivity of staff is highly valuable. • The University has been able to properly gauge areas of strength and weakness in its research performance. • We can identify research groups working well together. • This is particularly useful when nominating groups for schemes such as the ARC Centres of Excellence Programme.

19. Benefits to the University • Numerous scholars have described the difficulty in mapping the disciplinary fields of research groups and then measuring a group’s performance (see Bourke, & Butler, 1998). • By introducing a measurement, based on each individual’s performance, Socrates has managed to bypass many of the difficulties involved in measuring Research performance.

20. Additional Benefits • The application is used by the Promotions and Tenure Committee to chart the productivity levels of staff members and to determine their patterns of publication and grant application behaviour. • Anecdotal evidence suggests that individuals have begun to hold off on promotion until they exceed the average for their current level, and begin to perform at a level more consistent with the next step of the academic ladder. • Younger staff members have expressed satisfaction at having a gauge by which they can measure themselves, and many consider the average SI score for their level as the minimum score they should be aiming for. • Fundamentally, it is felt that Socrates has played an important role in clarifying for staff members the requirements of the University in terms of their research activity.

21. Benchmarking Opportunities • Socrates has provided UWA with the opportunity to benchmark our research performance against other Universities. • UWA is currently involved in a benchmarking exercise with one of our Group of Eight partners, the University of Queensland (UQ). • Once UWA and UQ have agreed which elements of their research performance data they wish to compare, a benchmarking index (BI) based on, for example, the level of research inputs, outputs and research training can be established, for comparison at the University, Faculty, research discipline (FoR code) and staff classification level.

22. What else is Socrates used for? • The ERA has been designed to inform the Government of the quality of the research being undertaken in Australian Universities. • The results of the ERA will significantly affect reputation and drive government funding to universities from 2011 onwards.

23. What does the ERA Measure • Publications from 2003-2008 for eligible staff present on the census date of 31 March 2009, including honorary and adjunct staff who use a university’s by-line, were required for the submission. • Research Income for the same period, but limited to income earned at the Institution. • The ERA required Research Outputs and Income to be divided across the eight clusters of disciplines using the 2008 Australian and New Zealand Standard Research Classification (ANZSRC) hierarchical structure. • For the ERA only the two digit Divisions, of which there are 22, and the four digit Groups, of which there are 157, ANZSRC Field of Research (FoR) codes were used. Outputs could be assigned up to three FoR codes, and Inputs could be apportioned to as many FoR codes as necessary.

24. Socrates' role in UWA’s submission to the ERA • Using the FoR codes, Socrates was able to draw together the some of the necessary data at the four and two digit level for the University’s submission to the ERA. However, a significant proportion of the FoRs were manually added or updated in Socrates. • As part of the ERA, the ARC also established a ranked list of 20,712 journals and a ranked list of 1950 conferences. Socrates was used to match journal articles and conference papers to the four point scale for journals and the three point scale for conference papers. Socrates was also used to match publications to their citation count in Scopus (the RQF had planned to use Thompson Reuter’s Web of Science but the ARC moved over to Scopus). • Having drawn data from the various databases, Socrates was also used as a collection tool, allowing researchers to input their indicators of recognition, and the appointed “stewards” from research areas were able to view and edit these submissions in Socrates. Background statements on each of the two digit FoR codes were also entered into Socrates, and the application was used to edit those statements. • The University’s XML submission was generated in Socrates and then uploaded to the ARC’s’ System to Evaluate the Excellence of Research’ (SEER) platform, which supports the submission, evaluation and administration workflows of the ERA initiative.

25. ERA summary data

26. Future uses of Socrates • Socrates should be able to provide teaching details for staff members, such as : • # units where they are course controller; • # units they teach per semester; • # lectures given each semester; • # tutorials given each semester; • # student satisfaction survey results • Such measures will also be useful in charting the University’s progression in University rankings that take factors beyond research performance into consideration. • Additionally, university committee service could also be recognised by Socrates, thereby allowing the Promotions and Tenure Committee with a more complete view of a staff member’s teaching, research and community service work. • We also hope to add additional data from the new product Incites, which will provide comparative citation data for both articles published in a particular journal and in specific fields.

27. Conclusion • Socrates has lead to a significant change in the way individuals view their research performance. • It has changed the way the University views research productivity at UWA. • Socrates has proven to be an effective tool for the measurement of research performance at the individual and group level. • It will also prove particularly useful in the long term as an application by which to chart the impact of research policy decisions upon the behaviour of researchers.