260 likes | 404 Views
ENQA experiences with external reviews of agencies - Challenges -. Bruno Curvale President of the ENQA board Bologna expert Head of international affairs at AERES. 29th of May 2009 Vilnius. Outline of the presentation. Introduction I - European Quality Assurance Major steps
E N D
ENQA experiences with external reviews of agencies - Challenges - Bruno Curvale President of the ENQA board Bologna expert Head of international affairs at AERES 29th of May 2009 Vilnius
Outline of the presentation • Introduction • I - European Quality AssuranceMajor steps • II - The ESG and the ENQA membership criteria • Similarities and differences • III - ENQA experiences • ENQA coordinated reviews • The work of the board • IV - The issue of consistency • Interpretation • Notions • Cultural dimension • V - Improvements and remediation • Debriefing • Training
There are many challenges related to: The context The objectives of ENQA The link between the ESG and the ENQA membership criteria The complexity of the two stages evaluation process The nature of the judgements to be made by the review panels and the ENQA board The methodology itself
Bologna process and quality assurance developments Primary responsibility of HE institutions for quality Evaluation of EQAR Cooperation of QA agencies and HE institutions European cooperation in quality assurance European Standards and Guidelines European Register EQAR 04/2008 E4 Group 2009 Leuven Louvain-la-Neuve 1999 Bologna 2001 Prague 2003 Berlin 2005 Bergen 2007 London 2006 Reco. 2006-143-CE 1998 Reco. 98-561-CE Source : Colin Tück, E4, B. Curvale
II - The ESG and the ENQA membership criteriaENQA objectivesSimilarities and differences between the ESG and the ENQA criteria
ENQA objectives Promotion of agencies ENQA full membership as a recognised sign of professionalism A single external review of agencies for ENQA membership and EQAR listing
Differences and similarities between the ESG and the ENQA Criteria ENQA criteria Principle based or rules based? Agencies shall follow the ESG and have to comply with the membership criteria Correspond to part 3 of ESG Integrate the part 2 of ESG (and by consequence the part 1 of the ESG) Occasionally associate Standards (ENQA criteria 1) Occasionally mix Standards and guidelines but keep the requirement in the conditional (ENQA criteria 4) Occasionally go further than the standard or the guidelines (ENQA criterion 3, ENQA criterion 7) Occasionally split Standards in different criteria (ENQA criteria 6i and 6ii) Have additional requirements (ENQA criteria 8i, 8ii, 8iii) But are fundamentally identical to the letter and the spirit of the ESG, unless it is the opposite
III - ENQA experienceENQA coordinated reviewsThe work of the board
The experience • 7 ENQA coordinated reviews organised so far (ANECA, AQA, AQU, ACSUG, AGAE, ARACIS, EUA-IEP) • 2 to be done (SKVC for sure and ECCE) • 24 agencies (among 39 full members) are already externally reviewed (AAC, AQA, FHR, VLHORA, NEEA, EVA, ACQUIN, AQAS, ASIIN, FIBAA, GAC, ZEvA, HAC, HETAC, NQAI, NVAO, NOKUT, PKA, AGAE, ANECA, AQU, NAHE, OAQ, QAA) • 8 candidates members are preparing to the review
The types and the coordination of the external review process Type A (ENQA membership criteria / ESG) Type B (also other aspects) - statutory functions (national missions, …) - ECA Code of Good Practice - special context (e.g. bi-national character, NVAO) - follow-up on previous review … Co-ordination national ENQA(+ other non-national) Sources: Tibor Szanto, Barcelona 17/03/09
The work of the panel Is the agency… - fully compliant - substantially compliant - partially compliant - non-compliant … with the criterion/ESG standard Overall judgement not required (but can be made if the panel wishes to do so) Sources: Tibor Szanto, Barcelona 17/03/09
The work of the board To answer the question: Can it follows the analysis and thejudgements made by the panel and decide to grant the full ENQA membership. What the board expects from the report: ESG Parts II and III fully covered Each criteria discussed separately Structure (under each criteria): - evidence („description”, „facts and data”) - analysis („consideration”) - conclusion (judgement on compliance) - recommendation (if any) Sources: Tibor Szanto, Barcelona 17/03/09
Consistency of judgement is a challenge For the panels and for the ENQA board because of: Situations have to interpreted Notions used are difficult Language issues Cultural dimension Implicit knowledge
The experience is gathered Well organised review processes are determinant Internal processes and external processes are equally important ENQA is learning from what is done by the panels Debriefing/brainstorming meeting for the training of experts for agency reviews, Barcelona, Spain, 17 March 2009 ENQA has started to train external reviewers Training session/workshop for experts for agency reviews, 21-22 May 2009, Paris, France
The experience is shared ENQA develops its documentation and makes it available to all: The Guidelines for national reviews of ENQA member agencies The Briefing pack for review panel members of ENQA coordinated reviews The Principles for ENQA coordinated reviews are now merged in one single document: The Guidelines for external reviews for the purpose of ENQA membership ENQA will continue to develop the methodology of external reviews The board has now a “book of precedents”, regularly updated by the ENQA secretariat
Thank you for your attention bruno.curvale@orange.fr www.enqa.eu
The European parliament and the council of the European Union… Hereby recommend that member States: … 4. Enable higher education institutions active within their territory to choose among quality assurance or accreditation agencies in the European register an agency which meets their needs and profile, provided that this is compatible with their national legislation or permitted by their national authorities 5. Allow higher education institutions to work towards a complementary assessment by another agency in the European Register, for example to enhance their international reputation. … Extract of the Recommendation of 15 February 2006 on further cooperation in QA in HE
EQAR Criteria All agencies which comply substantially with the European Standards and Guidelines for Quality Assurance (ESG) can be admitted to the Register. Substantial compliance with the European Standards and Guidelines is to be evidenced through an external review by independent experts. Such a review is coordinated either by a national authority or another organisation that is independent from the quality assurance agency under review. Full ENQA membership, being also based on substantial compliance with the ESG, will normally constitute satisfactory evidence for inclusion in the Register. Extract from the EQAR website http://www.eqar.eu
Standard 3.3 Guidelines 3.3 Standard 3.1 ESG 3.1, 3.3 / ENQA Criterion 1: Activities “The Agency undertakes external quality assurance activities (at institutional or programme level) on a regular basis. These may involve evaluation, review, audit, assessment, accreditation or other similar activities and should be part of the core functions of the Agency. In undertaking its activities, the Agency should take into account the presence and effectiveness of the external quality assurance processes described in Part 2 of the European Standards and Guidelines (ESG” Extract from the Application form for membership of ENQA
Standard 3.5 Guidelines 3.5 ESG 3.5 / ENQA Criterion 4: Mission statement “The Agency should have clear and explicit goals and objectives for its work, contained in a publicly available statement. This statement should describe the goals and objectives of the Agency’s quality assurance processes, the division of labour with relevant stakeholders in higher education, especially the higher education institutions, and the cultural and historical context of its work. The statement should make clear that the external quality assurance process is a major activity of the Agency and that there exists a systematic approach to achieving its goals and objectives. There should also be documentation to demonstrate how the statement is translated into a clear policy and management plan.” Extract from the Application form for membership of ENQA
Standard 3.4 ESG 3.4 / ENQA Criterion 3: Resources “The Agency should have adequate and proportional resources, both human and financial, to enable it to organise and run its external quality assurance process(es) in an effective and efficient manner, with appropriate provision for the development of its processes, procedures and staff.” Extract from the Application form for membership of ENQA
Standard 3.8 Guidelines 3.8 “are expected“ in the ESG guidelines ESG 3.8 / ENQA Criterion 7: Accountability procedures “The Agency should have in place procedures for its own accountability. These procedures are required to include the following: i. a published policy for the assurance of its own quality, made available on its website; ii. documentation which demonstrates that: … … ii. a mandatory cyclical external review of its activities at least once every five years which includes a report on its conformity with the membership criteria of ENQA. Documentation could here either indicate that a review has taken place or is planned.” Extract from the Application form for membership of ENQA
Review AND evaluation of the review ENQA full membership Inclusion in EQAR Evaluation of: - the review process - the review outcomes ENQA board Register committee Review of the agency (ESG 3, 2 and 1) + specific ENQA criteria External review Procedures, “behaviour” Agency National regulations Purposes Q Standards Specific objectives 1 2