António Rendas Universidade Nova de Lisboa Portugal

1. "Are Medical Schools & Programmes a challenge?" António RendasUniversidade Nova de LisboaPortugal … mentioned in a (e)mail from EUA as possible speaker… AMSE Annual Conference Barcelona, June 2008

2. The challenge (1) “Thus we may conclude that six conditions have to be given in order to ensure QA is effective and worth the time and effort At the level of the individuals, there has to be: • trust in the benefit of the evaluation • readiness to expose one’s weakness • readiness to invest time and effort to improve one’s performance where need for improvement is identified.

3. The challenge (2) At the level of the institution, there has to be capacity to realise the outcomes of the evaluation, i.e.: • a sufficient degree of institutional autonomy • institutional leadership to orchestrate for reaching and difficult changes and • resources to support the change and incentivise corresponding initiatives

4. The challenge (2) “The more we undertake quality assurance without having taken care of these preconditions, the more we run the risk of letting it degenerate into a mere lip service, into a comfortable method for bureaucratic consciences to be soothed and for politicians to say they paid attention to quality without meaning it” Sybille Reichert Looking back - looking forward: Quality assurance and the Bologna process. In: Implementing and using quality assurance: strategy and practice, a selection of papers from the 2nd European Quality Assurance Forum, EUA Case Studies 2008

5. Personal Experience • Dean of a Portuguese medical school for almost a decade • Rector of a Portuguese university for the last sixteen months • "Victim" EUA evaluation process specially designed for Portuguese medical schools. •  "Spectator" Development of national guidelines for accreditation and quality assurance for helath care units used for teaching in undergraduate medicine. •  "Wise?Fool?" Preparing the Universidade Nova de Lisboa for a standard EUA evaluation starting in September. And the process starts again …

6. Experience as a “victim”EUA Institutional Evaluation Programme Between 2000 and 2002, EUA was invited to design and conduct a comprehensive review of medical education in the five established medical schools in Portugal. The team focused on strategic issues related to: • the organisation of the medical education in general • the link with clinical education • the management of research • the interaction between Faculties and hospitals • quality assurance in medical education • management of human resources

7. Experience as a “victim”EUA Institutional Evaluation Programme The team methodology was as follows: The Faculties were asked to perform a self-evaluation based on four aspects: • mission/objectives • implementation (processes) • assessment of past and present results and situation • strategies of improvement and change The main difference in the process, compared to a classical EUA institutional review, derived from the fact that the team had to evaluate five parallel Faculties in the same field.

8. Experience as a “victim”EUA Institutional Evaluation Programme For these five Faculties performing such a comprehensive self-evaluation in a European context and at institutional level, was a first experience. The most frequently expressed issues of concern were: • the reorganisation of relation between Faculties and hospitals • how to match academic and clinical careers • the organisation of research • improving quality and pertinence of teaching and curricula • the impact of national authorities of funding issues • advice on strategic management capacity

9. Experience as a “victim”EUA Institutional Evaluation Programme Main findings and diagnosis At Faculty level: • there are to many Committees and Councils, with little effective power, thus diluting responsibilities and hampering any new reforms • there are almost no quality assurance mechanism for teaching and efforts to promote pedagogical skills and innovation are too modest, and the team got the impression of minimal efforts to ensure quality in the practice of teaching • there is a lack of any consistent management information system and of monitoring performance At system level: • the recently adopted contract-based approach in the framework of which this evaluation is taking place, is very positive and effective and is to be welcomed.

10. Experience as a “victim”Partial implementation why? At individual level, there was: trust in the benefit of the evaluation (mobilisation) readiness to expose one's weakness (SWOT) But not so much: readiness to invest time and effort to improve one's performance where need for improvement is identified because…

11. Experience as a “victim”Partial implementation why? … although at intitutional level, there was: sufficient degree of institutional autonomy institutional leadership to orchestrate for reaching and difficult changes there were limited: resources to support the change and incentivise corresponding initiatives

12. Experience as a “victim”Partial implementation why? However, The Portuguese government signed in 2001 contracts with medical schools to financially support the implementation of the reccomendations Unfortunately, The problem was what happened in the following years

13. Experience as a “spectator“Development of national guidelines for accreditation and quality assurance for helath care units used for teaching in undergraduate medicine. Between June and October of 2002 a project team from the University of Porto worked with two international consultants to develop guidelines for an accreditation project. • The aim of the accreditation system was to improve and standardise the clinical learning experience for students by: • establishing agreed minimum standards for student learning in clinical environments, • improving communications and liaisons with health care units for teaching in undergraduate medicine.

14. Experience as a “spectator“Development of national guidelines for accreditation and quality assurance for helath care units used for teaching in undergraduate medicine. • The accreditation process focused on evaluating the quality of the FACILITIES and support provided for teaching. It considered the way teaching was organised and supported as well as specific issues relating to teaching and learning. • The process was to be implemented on a national basis.

15. Experience as a “spectator" What FACILITIES? • Those health care units used for teaching in undergraduate medicine and those who want to start teaching. A health care unit is defined as a small institution or a department within a large institution which delivers teaching to undergraduate students (hospitals and health centres).

16. Experience as a “spectator“Accreditation decision • The accreditation team will make a recommendation for an Accreditation Decision to be made by the Rectorate/Faculty (Medical School) based on whether they consider that health care units meets the minimum standards for accreditation. • The decision will be included in an Accreditation Report in which the accreditation team will detail its findings and conclusions in order to provide feedback the leadership of the organisation about its performance. The options will be: successful (usually for 3 years), conditional and failure (with opportunity for corrections).

17. Experience as a “spectator“Why no national implementation? At individual was there: • trust in the benefit of the evaluation? • readiness to expose one's weakness? • readiness to invest time and effort to improve one's performance where need for improvement is identified? yes maybe …

18. Experience as a “spectator“Why no national implementation? At institutional level was there: : • Sufficient degree of institutional autonomy? • Institutional leadership to orchestrate for reaching and difficult changes? • Resources to support the change and incentivise corresponding initiatives? Yes, maybe… but, above all, no further contracts were signed with the government.

19. “Wise? Fool?” Preparing for a EUA institutional evaluation of Universidade Nova de Lisboa, after having just changed the governance model according to the new laws of higher education in Portugal, which also includes a new evaluation process based on ENQA’s rules.

20. “Wise? Fool?” Will evaluation processes strengthen Institutional autonomy at a level that will decrease the risk of political intervention? Will evaluation processes strengthen Institutional autonomy at a level that will favour contract-based financial support?

21. Conclusion • Neave, 2004, considers that evaluation is not a neutral process and “evaluation systems are not independent of what a government’s intention are, nor from what its policies”. One of the risk is having the government using quality evaluation and performance indicators as instruments to ensure compliance of universities, despite rhetorical of increased autonomy. Alberto Amaral

22. Problem: …however:

23. Solution:

24. Solution: Evaluators do not implement solutions ?