György BAZSA President, HAC ENQA

1. A possible option: grading scheme for judging compliance with criteria György BAZSA President, HAC ENQA Debriefing/brainstorming meeting for the training of experts for agency reviews Barcelona, 17 March 2009

2. Basic quantitative requirements - of a fine wine: ethanol 9 – 13 % sugar 0 – 100 g/l pH 3 – 4 - of a nice hostess: bust size 80 – 99 cm hip size 60 – 65 cm waistline 75 – 90 cm - of an excellent scientist: publications minimum 50 citations minimum 200 impact factor minimum 40 But still there is a need for the evaluation of its/her/his qualityby experts! Now, what about a good quality assurance agency?

3. i. Membership REGULATIONS OF ENQA (as from 26 September 2008) TITLE III. INVOLVEMENT IN ENQA a) Full Membership Full Membership of ENQA is open to quality assurance agencies that meet the relevant criteria described in Annex I Membership Provisions document. b) Candidate Membership Candidate Membership is available to quality assurance agencies that meet the criteria described in Annex I Membership Provisions document.

4. ANNEX 1 – Membership Provisions CHAPTER I. CRITERIA FOR FULL MEMBERSHIP a)Full Membershipof ENQA is open to quality assu-rance agencies in the field of higher education fromEHEA member states that have been operating andconducting actual evaluation activities for atleasttwo years. Before being accepted as a Full Member, an applicant agency must satisfy the Board that it meets the eight criteria, listed below.The applicant agency will thereby also meet the European Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG). - (8 or 15 criteria?)

5. b)An applicant agency may apply for Candidate Membership rather than Full Membership in the first instance. The Board will grant such member-ship if it believes • that the applicant demonstrates, through its application, substantialcompliance with amajority of the criteria and further believes • that the applicant will be able to demonstrate, through an acceptable external review submitted within no more than two years Candidate Membership, fullcompliance with the criteria for Full Membership.

6. QAAdocuments QAAexp. report ENQA non non Full member Candidate m. ENQA non non

7. ii. Evaluation of the compliances by the expert panel Dictionary and stages of compliance Compliant: in accordance with a specified body or rules (usually used in combination): Fully(compliant): entirely or wholly Substantially(compliant): of ample or considerable amount, quantity, size, etc. Partiallyorpartly(compliant): in part, to some extent or degree, not wholly; being such in part only, not total or general, incomplete Non-compliant: failure or refusal to comply, as with a law, regulation, or term of a contract.

8. ENQA National reviews(quide): • summarised conclusion stating whether the agency is • - fully or substantially compliant; • - partially compliant; or • - non-compliant.

9. ESU The study visit team believes that objectivity is best served by using a three level marking system. - Compliance, where QA implements the ESG in an effective manner; - Partial compliance, where QA practices are based on the correct interpretation of the ESG, positive aspects have been identified but the manner of implementation is not effective enough. - No compliance, where ARACIS practices are based on misinterpretations of the ESG …;

10. EUA - fully compliant - substantially compliant - partly compliant - (non-compliant) Summary (in the case of ARACIS): substantially compliant

11. March 2009!: Guidelines for external reviews of ENQA member and applicant agencies • - fullycompliant: the Agency adheres to the letter(literal interpretation) and spirit (intent) of thecriterion/ESG standard; • - substantially compliant: if the criterion/ESGstandard is not fulfilled to the letter, the Agency might still be considered substantially compliant if the principle/spirit of the stipulated criterion/ESG standard is followed in practice; • - partially compliant: the Agency does not fully or substantially comply with the criterion/ESG standard; • - non-compliant: the Agency does not comply with the criterion/ESG standard.

12. The way in which the membership criteria/ESG are to be interpreted by the review panel and the Board A possible option: grading scheme for judging compliance with different criteria The grading is proposed by the review panel

13. „The Review panel will need to exercise its judgement, in the light of all the evidence before it, and the intentions of the ENQA membership criteria / ESG, as to whether substantial compliance is being achieved, taking account of the circumstances in which the Agency is required to operate (see section 3.6.2). There is little point in adopting a ‘hard line’ position in respect of compliance with the ESG if, by doing so, trustworthy and credible agencies are prevented from gaining Full membership of ENQA through no fault of their own.”

14. „There may, as a result, be occasions where a strict reading of the ENQA membership criteria / ESG would indicate substantial rather than full compliance with one or more standards, but the panel may come to the conclusion that this should not jeopardise an overall judgement of compliance. This may occur when national legislation requires a different approach, or where an alternative way of achieving an objective is used, or because of other justifiable reasons.”

15. „Where an agency is found to be either partiallycompliant (+ - -) or non-compliant (- - -) with a criterion, the reason for this should be explained. Full or substantial compliance may be impossible for some agencies, owing to restrictions placed on them by the very nature of their work and/or legislation in place in their country(ies) of operation. When considering such cases, the ENQA Board will take mitigating circumstances such as these into account.”

16. A possible option for 15 criteria: grading scale for granting ENQA membership Decided by the Board Proposed by the review panel

17. A possible option for 15 criteria: grading scale for granting ENQA membership Decided by the Board Proposed by the review panel

18. A possible option for 15 criteria: grading scale for granting ENQA membership Decided by the Board Proposed by the review panel

19. A possible option for 15 criteria: grading scale for granting ENQA membership Decided by the Board Proposed by the review panel

20. A possible option for 15 criteria: grading scale for granting ENQA membership Decided by the Board Proposed by the review panel

23. iii. Further proposals - renumbering of ENQA criteria - public list of data - publicity of ENQA evaluation - QA vs. QP

25. Further proposals (out of scope of the title): • ENQA should prepare a list of compulsory data and information about QAA to be posted permanently on member agency’s homepage (partly in English). • Application, self-evaluation report, date of site visit and composition of visiting panel should be posted on the QAA’s homepage like on ENQA homepage. • QA vs. QP: quality assurance is crucial, but quality of the „product” (learning outcomes etc.) is the main aim. Should we move to QP direction?

26. Thank you for your kind attention!

30. ENQA Board decisions:(see Scrutiny form) 1. OAA’s first application for membership: a) candidate membership granted for a maximum of 2 years (documentation or external report based ) or b) full membership granted for 5 years (based an independent external review report only).

31. ENQA Board decisions: (see Scrutiny form) 2. Candidate member’s application for a full membership – based on a full evaluation procedure: - full membership for 5 years provided or - neither fullnor prolonged candidate membership granted, QAA debarred, but remains on the mailing list and has the possibility of a new membership application only after 2 years.

32. ENQA Board decisions: (see Scrutiny form) 3. Full members’ mandatory cyclical review in each 5 year: - full membership re-confirmed for another 5 years or - candidate membership granted in order to conform with full membership criteria in 2 years or - neither full, nor candidate membership is granted?

33. CHAPTER II. APPLICATION FOR MEMBERSHIP Application form and documentation Applications for membership of ENQA shall be in the form specified by the Board (details obtainable from the Secretary General). Applications for membership are considered and decided upon by the Board on the basis either of submitted documentation alone, or of submitted documentation and a visit to the applicant body. Applications for Full Membership will only be consi-dered where an independent external review report on the agency’s conformity with the membership criteria, carried out in a manner and to a standard acceptable to the Board, is received.

34. If, after consideration of the review report by the Board, the Candidate Member is not, in the opinion of the Board, in full compliance with all the criteria, the application will lapse and the applicant will, by the decision of the Board, not be allowed to reapply for membership until a further period of two years has elapsed. During this period the agency will not be a Candidate Member but will remain on the ENQA mailing list to ensure information dissemination on the activities of ENQA.

35. ENQA co-ordinated reviews: Briefing pack for review panel members The level of conformity with the ESG that is expected is ‘substantial compliance’, not rigid adherence. There may, as a result, be occasions where a strict reading of the ESG would indicate substantial rather than total compliance on one or more standards, but the panel may come to the conclusion that this should not jeopardise an overall judgement of compliance. … This does not mean that a clear failure to meet the standard or its intention will be condoned, and it is unlikely that more than one or two examples of ‘substantial’, rather than ‘total’ compliance will be acceptable to a review panel.