Measuring research performance at SA universities: Context, trends and consequences

1. Johann Mouton, CREST, Stellenbosch University SATN Conference 3 October 2013 Measuring research performance at SA universities: Context, trends and consequences

2. The thesis Research in any system is shaped by various factors . But at least three are important: • The material conditions • National and Institutional research “missions” and capabilities, and • Knowledge policies (national/ systemic/ institutional), strategies and plans that shape and steer R&D

3. 1. The material conditions of research: “The ontology of knowledge production” Our physical, social and geographical realities have historically been and continue to be major factors that influence the nature and shape of our knowledge production • Our geography – The clear skies drive our focus on astronomy and astrophysics • Our biodiversity explains our comparative strengths in fields such as ornithology, botany, marine biology and the ecological sciences • Our mineral resources have generated and demanded research and expertise in mining engineering and physical geology. • Our African origins - cradle of humankind - explains the long and significant production of knowledge in palaeontology and archaeology • South Africa has been and continues to be seen as a “laboratory of diseases” (even from the days of the rinderpest in the late nineteenth century to the current challenges of Malaria, TB and AIDS) that lends itself to experimental development and clinical trials • Our multi-culturalism and recent political history influences scholarship in ethnic and cultural diversity, inclusive education, human rights and reconciliation issues, etc.

4. 2. National (and institutional) missions The history of SA science can be presented as the history of different knowledge/science missions which coincided with specific national (and international) socio-political events Pre-mission • Knowledge production for its own sake: Science driven by curiosity: 1700 – 1850 (Amateur science) • Knowledge to solve industrial, health and social problems: deep level mining, rinderpest, white poverty: 1850 - 1940 (Mode 2 science) • Knowledge for the Afrikaner volk (“volksuniversiteite”): 1920 - 1950’s Mission • Big science: Science in the national interest (1946 +) • Strategic science: Science in defence of the apartheid state (military and energy research) (1960s to 1990) • Critical (social) science: new societies, new journals, critical research, critical scholars, new institutions (EPU’s, and many others) (1980s – 1994) • Science for innovation (Post 1994 – the adoption of the concept of a national system of innovation

5. Elaboration on “science missions” By “science mission” we refer to the recognition - first by the state, that it has a responsibility to fund and support (and also steer) science and second, by universities that they have a scientific research mission. The first is exemplified in events such as the following: • The establishment of a national Research Grants Board in 1918 (within the Department of Mines and Industry + Ministry of Education) • The appointment of Hendrik van der Bijl as the first science advisor to Prime Minister Jan Smuts in 1920 (He subsequently founded ESCOM in 1923 and ISCOR in 1926) • The post-war effort to establish science councils (according to the British CSIR-model) to undertake strategic research in high-priority areas Evidence of the second – science as a university mission – is illustrated by the emergence of university research centers and institutes late 1930’s and 1940s and especially in the 1950s and only much later with the establishment of research offices and appointment of research directors (1970’s).

6. 3. Knowledge policies and their impact on research (Knowledge) policies are high-level and long-term normative statements of desired end-states and require operationalisation through frameworks, strategies and plans. Such policies (and related strategies and plans) have differential impact: ranging from strong to weak, relatively direct to very indirect and both intended and unintended. Knowledge policies impact differentially on the different dimensions (nature, shape and volume) of knowledge outputs. There have been early examples of knowledge policies in the SA system: the first science policy goes back to 1982 (as far as we could establish) and the first Frascati-type surveys to measure the performance of the SA system to the early 1970’s. The introduction of the then DNE research subsidy scheme in 1987 and the establishment of the FRD rating system in 1986 are other examples. But it is especially since 1994 that we have witnessed a huge increase in the number of policy, framework, strategy and planning interventions.

7. A first mapping and examples • National knowledge policies • White Paper on Science and Technology (1996) • Education White Paper 3: Programme for the transformation of Higher Education (1997) • National strategies/ plans / frameworks • National Plan on Higher Education (2001) • National R&D Strategy (2002) • DoE Research Funding Framework (1987/ 2003) • HEQC Audit Framework (2004) • DST Ten Year Innovation Plan (2007) • National Development Plan (2012)

8. DHET Policy (2003) The 2003 policy listed the following journal categories as qualifying for subsidy purposes: (a) The Sciences Citation Index of the Institute of Scientific Information (ISI) (b) The Social Sciences Citation Index of the ISI (c) The Arts and Humanities Citation Index of the ISI (d) The International Bibliography of Social Sciences (IBSS) (e) The Department of Higher Education and Training List of Approved (Accredited) South African Journals

9. Coverage of SA journals in international databases • The latest updated list of SA Journals numbers 295 unique SA journal titles. Of these: • - 72 titles are indexed in the three WoS Indexes • - 73 in Scopus • - 39 in IBSS • Of these 295 unique title, 115 titles appear in at least one of the referenced databases, which means that there are 180 SA journals on the list that do not appear in any of these databases

10. The impact of knowledge policies and strategies on research (1) Knowledge policies and strategies have been devised to impact different dimensions of research. • The natureof research (basic, applied, strategic, Mode 1 and Mode 2): Steering research production into more collaborative, industry-linked research (e.g. THRIP, Innovation Fund grants) • The quality of research: Emphasis on quality-assurance systems and processes (HEQC Audits/ Assaf Journal Reviews/NRF Rating) • The shapeof research (scientific field distribution): Greater emphasis on SET fields, prioritising IKS, Space science, Astronomy, etc. • The volumeof research (magnitude of outputs): Increasing the volume of research outputs and knowledge outputs (Masters and doctoral) graduates

11. The impact of knowledge plans and strategies on research (2) • Research efficiency (or productivity): Whether individual (and institutional) productivity has increased: Are we getting more outputs given the inputs (funding/ etc) • Research collaboration: Whether scientists and scholars pursue more colaboration both nationally and internationally • The visibility and (international) impact of research: The extent to which research in SA is moe visible and recognised and cited internationally. • The transformation of research: The extent to which more female and black scientists and cholars participate in research production, i.e. whether we have expanded the human capital base of research.

12. Research output

13. Research output SA University research production - since the introduction of a national research subsidy scheme in 1987 – initially remained quite stable (ranging between 5000 and 5500 article units between 1988 and 2003) BUT then increased dramatically to reach more than 8000 units in 2010 and nearly 11000 in 2013. The best explanation for this dramatic increase is the introduction of the new research funding framework in 2003 (which came into effect in 2005) which provided much more significant financial reward for research units and clearly provided a huge incentive to institutions to increase their output

14. Total HE Research Output (subsidy units): 1987 - 2011

15. Outputs and monetary volumes R20k

16. Research output: UoT’s (1991 – 2011)

17. UoT’s research output (2005 – 2011)

18. UoT’s weighted research output (2005 – 2011)

19. Distribution of articles by main fiels The shape of knowledge production

20. Scientific output by field by University (1990 – most recent year)

21. Distribution of articles by Journal Index The shape of knowledge production

22. Example: The UP “Sciences” Faculties (80%) in Web of Science: 2005 – 2010

23. Example: The UP “Humanities” Faculties (31%) in Web of Science (2005 – 2010)

24. Research productivity measures

25. Example: UFS Research productivity by Faculty (2005 – 2011)

26. Example: Distribution of relative research productivity at UFS (2005 – 2011)

27. Comparison of WITS, UCT and UKZN i.t.o. research productivity

28. Example: The active human capital base: UCT and UP

29. Journal publication trends

30. Where do UFS authors publish?

31. Where do UCT authors publish?

32. Gender, race and age of author Demographics of research

33. Example UCT: Gender of author (1990 – 2011) The contribution of female authored papers increased from 17.6% in 1990 to 33.7% in 2011. This means that the contribution of female authors to the total research output at UCT has nearly doubled over the past twenty-two years.

34. Gender of author by UFS Faculty (2005 – 2011)

35. The age distribution of UP authors: 1990 - 2010

36. Example: UCT - Trends in the race of authorships: 1990 - 2011 From a low base in 1990 of 5%, the most recent data (2011) show that nearly 22% of all article outputs (measured in fractional counts) are produced by Black members of staff.

37. Consequences

38. Consequences of steering • On research agenda setting • DST agenda setting: Homogenization of demand • NRF agenda setting: Focusing of research • Extensive “research surveillance” at all levels (Auditing, monitoring and programme reviews) – setting of performance targets, annual scorecards and development of national M&E systems = the research audit society has arrived!)

39. Consequences of steering Emergence of non-virtuous research practices • Chasing the numbers: • Salami publishing • Involvement of incidental guest visitors as co-authors • Students required to publish from their theses – sometimes “at all costs”) • Unethical practices • Self-plagiarism • Co-authorship issues (Insistence on recognition) • “Protectionist” publishing in in-house journals

40. In conclusion The research performance of a university is a function of a multitude of historical, political, institutional and material forces. This is certainly true for SA universities and universities of technology. Policies and strategies to strengthen, incentivize and steer research production rarely are unequivocally good or bad. It is clear that SA policies of the past 25 years have resulted in significant increases and growth in research output volumes, in more international publications in the WoS and in more collaboration with foreign scientists and scholars. At the same time, these policies have clearly resulted in prioritising volume and quantities over quality of knowledge production. And, in addition, we are also witnessing more and more cases of practices in authorship and publication that are either unethical or border on the unethical. It is essential that we achieve a greater balance between concerns of production, productivity and efficiency on the one hand and quality and ethics on the other.

41. Thank you