Universities in the ‘Performance Age’

1. Universities in the ‘Performance Age’ Session two: Performance Management, Assessment and Rankings Maison française d’Oxford, Oxford, February 17-18th

2. Assessments and Rankings in French Higher Education: a Case Study Lise Gastaldi & Caroline Lanciano-Morandat The Institute of Labour Economics and Industrial Sociology (LEST) CNRS and Aix-Marseille University

3. Introduction • A presentation of the first works we realized within a larger research programme funded by the French ANR (National Agency for Research) • The Prestence project: From Prestige to Excellence, the fabric of academic quality • We investigate the consequences of a change in academic quality judgment devices with the increase of formal devices in addition to informal ones • With a focus on the devices dealing with higher education establishments

4. Introduction • Informal devices which are older • Produce global judgments about quality in terms of reputation or prestige • Judgments are formulated • With no formalized list of criteria • By various actors included non expert ones • Formal devices which are increasing currently • Evaluations are based on formalized criteria • With the aim to formulate more objective judgments by a more systematic evaluation of quality

5. Introduction • Informal and formal evaluation devices can produce different judgments on one establishment • The Prestence research programme aims to investigate how an establishment is succeeding in producing academic quality, given different regimes of quality evaluation exist • Putting the light on the organisational work of gathering and combining various resources • And investigating how the change of evaluation regime can influence the evaluation of the establishments and the way they react in terms of strategy and organisational choices

6. Introduction • A comparative research between several establishments in various countries* and scientific fields* • I and Caroline focus on the chemistry and physics field • We realized the first case study on an old and prestigious French institution • Characterising a singular organisational model and a focalisation strategy with the production of specific outputs • Through this French case study, we address the question of the sustainability of singularity in higher education when establishments face more developed formal evaluation devices

7. Content 1. The French context of evaluation in higher education 2. The case study: a singular establishment 3. Aims and issues 4. Singularity facing formal evaluation devices 5. From evaluation to change on strategy and organisational path? 6. Conclusion and research perspectives

8. 1. The French context of evaluation in higher education • We observe a strengthening of assessment • We focus on establishments and recurrent evaluations and we adopt a classification of these evaluation devices into two main categories • Evaluations leaded by national public authorities • With a focus on public establishments • Evaluations leaded by other actors (newspapers, higher education institutions…) • ARWU, THE, QS ranking, SIR…

9. The French context of evaluation in higher education • A classification that refers also to two main differences • The imperative nature of the evaluation devices (or not) • Public establishments cannot avoid assessments leaded by national public authorities • If we consider rankings • Some needs the establishments collaboration (it is the case of national ones and of the THE ranking for example) • The establishments can refuse to respond to the ranking producers requests, even it can be a risky strategy given the growing audience of rankings • Some doesn’t need the establishments collaboration and consequently the establishments are evaluated even if they are not agreed • The form and the ‘philosophy’ of the evaluation process and its result

10. Differences between the two categories we distinguish

11. The public assessment devices • Before 2007 • There were various devices leaded by different public entities • Some was in charge of assessment of the trainings • Some was in charge of assessment of the research activities and laboratories • There was no unified evaluation system

12. The public assessment devices reforms • In 2007, the AERES was created and took the place ofseveral former evaluation devices • Agency for research and higher education assessment • The AERES has the mission to evaluate all the research and higher education public institutions that are under a French Ministry authority • With a three levels evaluation: research laboratories, training programmes and establishments • An evaluation each 4/5 years by ad hoc experts committees compound by peers, higher education experts, industrial researchers…

13. The AERES assessments • First the entity evaluated make a report that presents itself, its activities, productions… with a self-evaluation • Then the experts committee visits the entity with meetings with the governance team, the academic and administrative staff and the students • They write an evaluation report, and the entity can respond • These documents are published on the AERES website • The committees mark laboratories and trainings on a A+, A, B, C scale • These evaluations can be a basis for the Ministry Higher Education and Research decisions as funding, trainings certification… • With nowadays a direct link with the SYMPA model that determines the research budget allocated to an university in relation with the marks obtained by its laboratories

14. National and international rankings • National rankings in France • Mainly dedicated to engineering and business schools • For engineering schools (that are part of universities or not) • Rankings are realized and published by newspapers • There are few old rankings and several new ones • A larger audience • With an Internet diffusion • With an increasing attention paid to rankings by students and their families • International rankings (Shanghai, THE…) • We also can observe a multiplication of rankings that enjoy a growing audience

15. A functionalist lens with a reference to socio-economics works (Karpik, 96; Cochoy,99…) • Various actors (students and their family, public authorities, partners, academic staff, employers…) who invest resources in higher education want to have some references about the quality and performance of research centres, trainings and establishments • Higher education can be considered as experience goods • It is difficult to have a precise idea of an entity quality when you are not inside • It is a more and more complex and larger system if we consider the increasing number of institutions and trainings in each country and if we consider the global ‘market’ of higher education • The stakeholders need quality references to do their choices among all these entities • And considering the new importance of accountability logics => All these facts lead to strengthen evaluation

16. 2. The case study: a singular establishment A French engineering school located in Paris • Created in 1882 in a specific context • East French regions, where chemistry schools were concentrated, passed into the hands of Germany • French chemical industrials wanted to create a new school to train engineers and researchers for national chemical industry • This project was presented to Paris city council and a new school was created under its authority • Nowadays • An engineering school that is a ‘grande école’ with several research labs on a same campus • Apluridisciplinarity both in teaching (chemistry, physics, mathematics and biology) and in research (chemistry, physics and biology)

17. Methodology: an in-depth case study • A collaborative research • We collect various data between February 2010 and November 2011 • 40 interviews • Public and internal documents • That we analyzed in a qualitative way and with an historical perspective • We organised several restitutions • to some actors in charge of the establishment governance • to our Prestence project colleagues

18. A singular establishment • Institutional status: one of the two public higher education institutions that are under a local authority and not under a Ministry authority • A small institution • 250 students • 400 academic and administrative staff • It is a small institution in France, and the gap is greater in comparison with foreign universities • A strong autonomy of the heads of laboratories behaving as CEO of small firms • Autonomy to conduct their research activities and manage their teams • Autonomy to lead valorisation activities with for example the capacity to take patents in their own name

19. A singular establishment • An organisational model characterised by strong interactions between research, teaching and industry • Research and teaching: lot of courses take place in the labs; the students spend 4 months in the school labs during the engineers training; the last year is dedicated to do a research Master degree; most of 60% of engineers do a PhD and the most of them will begin their career in industrial or academic research centres • Research and industry: researchers lead fundamental research but with a long term application preoccupation ; they have lots of research contracts and we can observe all the form of technology transfer: scientific consulting, patents, spin-offs… • Teaching and industry: with a six month stage in an industrial research department, with some courses linked to a specific firm needs and with various relations with an active network of industrial partners • An establishment that produces specific research and students prepared to work in the field of research and innovation • It is a singular establishment in the French context comparing to universities and also others engineering schools

20. 3. Aims and issues • This establishment presents a singular organisational model and produces specific outputs • It is a well-evaluated establishment for a long time • By informal judgment devices • Strong reputation and prestige in the academic world • Strong reputation in the industrial world • Some of its researchers are well-known by the public • By old formal judgment devices • Publications • French research institutions as CNRS or INSERM • However how is it evaluated by current formal evaluation devices?

21. Aims and issues • A main issue: Is singularity in the current higher education system sustainable? • Declined in two research questions with management and public management lens • How does singularity face the current formal evaluation devices? • Or in other words: how do the current formal evaluation devices deal with singularity? • Could these evaluations have some consequences on the singular establishments strategy and path? We address these questions through one case study

22. 4. Singularity facing formal evaluation devices • The establishment experiences some difficulties with current evaluation devices • With the AERES and the CTI • The CTI: the ‘commission for engineer diploma’ – a specific French instance that evaluates engineering schools on the training side and does recommendations to the Ministry that gives the right for an establishment to deliver engineer diploma • With national rankings and especially the international ones

23. Confronting singularity to national public assessments • Most of the research centres are well-evaluated but we studied the case of a strong controversy about one laboratory evaluation • It is an historical laboratory with a strong reputation and positive CNRS evaluations • But the AERES assessed it as a ‘B’ laboratory • All the laboratory members, the establishment directory and the head of the CNRS chemistry department protested • A case which illustrates some criticisms about the AERES evaluations • Difficulty to assess pluridisciplinary works • Subjectivity of the committees members • The short time allocated to visit each laboratory • Some tensions around the evaluation of application-oriented research

24. Confronting singularity to national public assessments • Tensions with the CTI • The CTI leads evaluations by comparing training characteristics with standards • The conformity with these standards is a condition to obtain accreditation • There are conflicts on • The training length: it is a 4 years training in this establishment compared to the 3 years standard • The training content: for the CTI students must follow courses in management and economics that are really low developed in this establishment

25. Confronting singularity to national public assessments • Due to its status there is no global evaluation of the establishment and no valorisation of its specific model with strong interactions between research, teaching and industry

26. Singularity facing rankings • First we should think that rankings can be more in favour of this kind of establishment • With the mechanical objectivity of the evaluation process that only counts the publications, there is no more tension with experts who can be subjective or not able to evaluate research in pluridisciplinary fields • Some rankings (some national or the THE ones) assert they evaluate all the dimensions and activities of an establishment • Nevertheless… • The establishment is ranked between the 10th and 20th positions in national rankings, consequently not among the most excellent ones • It is only ranked in the ARWU, between the 200th and 300th positions; it is not ranked in all the others rankings

27. Singularity facing rankings • When we compare the establishment profile and the rankings criteria • Strengths • An old institution • An historic and strong investment in research • A high ratio staff / students • A high quality of the students professional insertion • Weaknesses • The small size (with a lot of criteria in absolute terms, with the importance of reputation inquiry…) • The low internationalisation with few foreign students and academic staff • Dimensions that are not really included in rankings • The industrial relations and technology transfers • The interactions between research and teaching

28. Singularity facing rankings • If rankings are objective, they incorporate normative definitions of higher education and excellence with usually • A conviction of the superiority of a big size establishment • A focus on research outputs (evaluated with only two criteria: number and impact of publications) • A strictlyacademic definition of excellence • The importance of internationalisation both of students and scientific staff

29. Singularity facing various evaluation systems • Consequently we can ask if we would assist to a process of singularity reduction that will lead the establishments to more standard organisational models producing more standards outputs

30. 5. From evaluation to change on strategy and organisational path? • Such a process of ‘desingularisation’ could only occur if formal evaluations lead some changes in establishments strategy or organisational model, and this depends on several elements: • The way the establishment obtains crucial resources and the existence or not of a link between its evaluations and its capacity to gather these resources • The way the establishment’s members sense the evaluation in general and the different devices and also their consequences on their access to resources • The strategy of the establishment in relation to the various evaluation systems, given it can adopt • An active or passive resistance strategy • Or a strategy of adaptation; and in the case of we have to consider the capacity of the establishment for engaging reforms in order to improve next evaluation

31. Karl Weick (1977, 1995…) • Weick and the theory of ‘sensemaking’ • The process of sensemaking which is very important to understand how individual and collective actors in organisation act and react to environmental change is composed of 4 stages • An environmental change • An enactment process that leads the actors to take aware of the environment, but the environment is complex and various • The selection is the process by which actors try to reduce the environment ‘equivocity’ constructing interpretations and leading operations to give sense to the environmental change • The retention/memorisation: the actors retain some interpretations and act in consequence; and the interpretation of the environment which are memorized will influence the next processes of sensemaking when other environmental change will occur • Consequently it is important to adopt an historical perspective to analyse the actors reaction to a change in their environment

32. For a long time, a low sensibility to evaluation and especially to formal evaluations • On the research side • At the beginning: laboratories are not affiliated to external entities • Progressively some had been affiliated to universities or research entities as CNRS or INSERM • With positive evaluations by CNRS and INSERM • Nowadays it remains few laboratories that are not mixed research units and consequently that escape to formal evaluations • With the conviction that autonomy is a strength because it allows to lead application-oriented research and to exercise valorisation activities • On the teaching side • The establishment was in a good position to bargain with the CTI in order to maintain its singular characteristics

33. For a long time, a low sensibility to formal evaluations • The establishment enjoys positive informal evaluations by • The academic world, due to its research works and publications • The industrials which pay for collaborative research (financing projects and fellowships) and recruit engineers and doctors coming from the establishment • The students and their family, due to the establishment reputation and the very good conditions of professional insertion for the engineers coming out of it • The general public, due to its star scientists, 5 Nobel Prizes and its story (since Pierre and Marie Curie)

34. For a long time, a low sensibility to formal evaluations • Thanks to • These positive evaluations • The autonomy some laboratories enjoy because they are not affiliated to external institutions that allows them to lead valorisation activities (patents, spin-offs) without difficulties • Valorisation activities as sources of incomes • The historical support of the main public authority that allocates every year comfortable means to the establishment • The establishment did not face problems to access resources

35. For a long time, a low sensibility to formal evaluations • They have enough money and job positions to conduct their activities in good conditions • With funding of public authorities • With industrial research contracts • With the revenues of valorisation • They do not have problems to recruit scientific staff • They recruit high level students • The students do not experience difficulties for their professional insertion in higher education or in research and development departments in the network of industrial partners

36. Nowadays the context is changing • It is the end of the former victorious resistance against CTI orders: the establishment had to conform its engineering training to the 3 years standard length • Some evaluations by the AERES are reserved, with effects on internal hierarchy between laboratories and researchers • The fact the establishment does not appear in the international rankings except Shanghai one • The main public authority (the Paris city council) now heeds to these rankings and is concerned by the low international visibility of the establishment • With a stronger issue in terms of the return on the investment realized in financing this establishment

37. A progressive evolution of representations • A perception influenced by the establishment path • Some people who continue to refer only to informal judgments • A reference to few academic colleagues in prestigious institutions • Few persons mention rankings

38. The reaction to academic evaluations • Even the conditions of evaluation had changed with the AERES creation, this kind of assessment is usual and well-accepted • Peers assessment • Evaluation of the research programmes and the publications • Evaluation leaded by national public authorities (even in this case the Ministry is nor the main authority neither the main resources supplier) • Moreover their legitimacy, these evaluations condition a larger part of the resources (money and jobs position) • Directly in the case of laboratories associated to universities • In a more indirect way with the influence of the AERES marks on the capacity to obtain some financingson projects

39. The reaction to academic evaluations • In fact • Some of the criteria of formal public authorities assessments had been progressively integrated • More recent evolutions can be analysed as a response to the AERES evaluation process and norms • Some laboratories mergers to constitute more important units • The research activities and laboratories are now presented in a more intelligible way for external evaluators (or widely for stakeholders) • Through of a kind of map with different areas for each discipline that puts laboratories in relation to them • An internal reorganization of some labs to present a structure easier to understand in a fast way for external evaluators

40. Public evaluations and future of singularity • The evolutions are too recent to conclude but we can ask if some of them could conduce to a reduction of singularity • The aim to normalize the situation in integrating all the laboratories in research units affiliated to external academic partners, with an automatic loss of autonomy for them • The reorganisation of labs and teams are not neutral on the scientific activities

41. Public evaluations and future of singularity • Moreover public evaluation systems could lead to deeper transformations if we consider that pluridisciplinary teams and application-oriented research are not well evaluated by all the committee experts • Then the form of the assessment result can have a performative effect given the report formulates analysis and gives advices on the governance, the internal organization and the scientific field: orientations, programmes, collaborations, publication strategy…

42. The reactions to rankings • Few persons inside the establishment are preoccupied by rankings, except the director and the man in charge of communication • A low attention is historically paid to national rankings due to several elements 1- They are seen as non objective and non reliable because of their way of collecting data 2- They are considered as non legitimate because they are not realized by peers or experts and not controlled by academic community or public authorities

43. The reactions to rankings 3- National rankings have quite low impact on students recruitment • In the French higher education the engineering schools recruit by a special way: after a 2 or 3 years special training the students pass very selective exams • There are various exams to integrate the different engineering schools, and the establishment we study recruits since few years on the same exam that the most prestigious French establishment that is Polytechnic school • Whatever is their rank in rankings they succeed in recruiting excellent students

44. The reactions to rankings • When the international rankings appeared, most of the people in this establishment not paid attention to them • We may see here an effect of establishment path • Its members always have neglected national rankings • They have not anticipated the new audience of these international ones thinking the establishment prestige is sufficient to assure its durability • Indeed until now the establishment resources are not linked to its ranks in these rankings

45. The reactions to rankings • However the directory board in which members of Paris city council are part of took decisions that can be seen as reactions face to this growing audience of rankings • An increase of the students numbers and the aim to recruit foreign ones • A strategy to association with other higher education institutions • It is membership of ‘ParisTech’ that is a large association mainly of engineering schools located in Paris and its suburbs • It is also membership of ‘Paris Sciences et Lettres’ which is an association of institutions in sciences and humanities fields located in Paris

46. The reactions to rankings • We can see this strategy of association in the French context characterized by several mergers of universities with an evident preoccupation of increasing their size • When the size is a key factor in rankings • Given such mergers respond to Ministry pressure • So the establishment we study adopts an alternative strategy compared to merger in order to preserve its autonomy • We do not know now if this strategy is durable or if in the future this establishment will have to merge with university

47. The rankings and the future of singularity • For the moment rankings have low effects • But • The aim to increase the number of students will ask the question of the capacity to maintain all the activities on the same campus while it is a key factor in the high proximity between students and research • If the establishment will merge with other institution and especially with a big university, the question of the capacity to maintain its specific model with the heads of research centres high autonomy that is a factor of the establishment attractiveness for academic staff will be asked

48. The creation of a own assessment committee • The directory board decides to create an international scientific committee in order to evaluate the establishment in all its activities • This committee is composed of 10 international academic and industrial star scientists • Each year, they lead an overall evaluation on research, teaching and valorisation activities • They formulate qualitative assessment and numerous advices. For example they can highlight research topics to investigate, strategic scientific field to develop…

49. The creation of a own assessment committee • We can interpret this own international committee creation in two ways • As a way to say to national public authorities that the establishment acknowledges the importance of evaluation even if it is not evaluated by the AERES • As a buffer • To help the establishment to better understand the new international evaluation norms • And to help it to imagine some transformations that could improve its evaluations but in the respect with its singularity

50. Why this need for such a buffer? • Rankings only evaluate outputs and in some cases inputs (as budgets or staff), often with few indicators • These criteria are not obvious to turn into concrete actions • We have to more publish in higher level journals, ok… • It would be better if we have more money, ok… • We should have Nobel Prizes, ok… • Contrary to the AERES evaluations that give concrete practical and more easy-to-implement actions • The AERES evaluations are less normative but easier to understand and turn into action, whereas rankings are more normative and directive as they don’t give obvious ways of improvement