‘Innovations in Examinations Assessment’—some principles

1. AHEAD Symposium on A Charter for Inclusive Teaching and Learning in Higher Education 28 April 2009 ‘Innovations in Examinations Assessment’—some principles Peter Cullen Head of Standards, Research and Policy Development Unit HETAC www.hetac.ie

2. HETAC’s functions include • to determine “standards” of knowledge, skill or competence … • to ensure that providers establish procedures for the assessment of learners which are fair and consistent and for the purpose of compliance with standards ; • to review of the effectiveness of quality assurance procedures

3. The national framework of qualifications • National Framework of Qualifications (NFQ) is a system of 10 levels • It is the frame of reference for all education and training qualifications in the state • It is aligned with Bologna framework and currently being referenced to the European Qualifications Framework • The NFQ emphasises (cumulative) learning outcomes(knowledge, skill and competence) for qualifications • The NFQ eschews prescriptions of inputs (it is programme-independent)—this does not downplay the importance of the programme

4. What is a standard? • It’s an educational objective • …a required or specified level of excellence, attainment, … (from Oxford English Dictionary NSOED 1997) • knowledge, skill or competence to be acquired by learnersbefore a higher education and training award may be made (Qualifications Act)

5. What are the uses of standards? • For the provider to use when designing and reviewing intended programme learning outcomes • For learners/teachers—the intended programme learning outcomes constitute the overall learning/teaching objective. • For the awarding body to use when validating programmes • For internal QA to use when reviewing performance • For external examiner to use along side the intended programme learning outcome • For the external quality assurance agency to use when auditing/assessing/accrediting • For the employer when interpreting the significance of qualifications

6. Why not rely on tacit standards? • Traditional approaches to assessment in higher education typically place heavy reliance on tacit understandings of standards and can be strained in new or rapidly changing contexts. • Examples of changing contexts which have encouraged practitioners to look for innovative approaches include • massification, new kinds of teaching and learning, modularisation, plagiarism, learner retention, computer-aided assessment, new kinds of intended learning outcomes, availability of study time, declining resources and greater diversity of learners. • Tacit element is ineluctable • connection with community of practice/learning

7. Types of standards and setters Particular General NQF Award-type descriptor Named award or broad Field of Learning standard Narrow discipline Standard or Regulated Profession Intended programme Learning outcome Nursing Programme X Science Programme Y Honours BachelorDegree Engineering Generic NQAI HETAC approach FETAC approach Tuning/QAA Universities and DIT IOT and other HEI

8. Standards and precision • Precision needs depend on how the standard is to be used. Is it to check that the mean performance level is on target? Or is it to check that the vast majority of students graduate attain or exceed the standard? Is it used for summative assessment of learners? • Note that the QAA benchmark statements contain threshold levels as well as other performance levels) • Standards can be representative (describing characteristics to be possessed by most) • Standards can be threshold (characteristics to be possessed by all) • Standards can be typical (the meaning of this varies) • Can standards that describe the average (mean performance) be used for summative assessment of learners? • What is important is clarity and precision but not too much precision because flexibility is necessary. The nearer the standard is to the ‘programme’ the more precise it should be– the next slide illustrates

9. Synthesis of the intended programme learning outcomes Note that stages emerge naturally from the network of learning opportunities in a learning environment Learning environment (including people interacting) Module Module Module Entrants Graduates Module Module Module Module … …

10. Minimum intended programme learning outcomes (the provider’s standard) • Minimum intended programme learning outcomes describe achievement that any graduate is certified to have attained. • Effective assessment helps learners to attain the intended programme learning outcomes. • Intended programme learning outcomes are developed and maintained by providers. • Programmes are designed to enable learners to achieve intended programme learning outcomes. • Intended learning outcomes are specified for each of a programme’s constituent modules. • The number of learning outcomes in a statement of intended learning outcomes is variable (depending, for example, on the semantics and the level of explicitness used) and it is not a proxy for credit.

11. Implementing standards • Note the implementation gap • Note the implementation lag: assessment is key • Intended learning outcomes are necessarily assessable • What does assessable mean • examine learners’ individual achievement prior to certification • inference from longitudinal research (learners and graduates and employers) • Other… • Student assessment procedures are expected to be designed to measure the achievement of the intended learning outcomes and other programme objectives. (ESG) • Student assessment procedures are expected to have clear and published criteria for marking (ESG)

12. Six assessment principles • Learners are responsible for demonstrating their attainment • Assessment supports standards based on learning outcomes • Assessment promotes and supports effective learning and teaching • Assessment procedures are credible • Assessment methods are reviewed and renewed as necessary to adapt to evolving requirements • Learners are well informed about how and why they are assessed • The following slides approximate: please consult the HETAC website for its definitive principles www.hetac.ie

13. Learners are responsible for demonstrating their learning achievement • A learner who is enrolled on a programme should submit himself or herself to assessment for the purpose of demonstrating attainment of the programme’s intended learning outcomes. • Each learner is expected to strive, with the support of the provider, to develop academic integrity, undertake assessment tasks honestly and truthfully and shun plagiarism and other forms of academic dishonesty or impropriety. • Theprovider should ensure that there are learning opportunities for the programme’s intended learning outcomes.

14. Assessment supports standards based on learning outcomes • Awards are made and classified exclusively on the basis of criterion referenced assessment of learning outcomes (knowledge, skill and competence.) • The learning outcomes required to qualify for HETAC awards are specified by the awards standards issued by HETAC. • The awards standards describe the standard to be acquired by learners before a higher education and training award may be made which implies that they describe the learning required to pass. • The minimum intended programme learning outcomes define the minimum learning outcomes for a particular programme at the programme level. • Credit, while not a proxy for attained learning outcomes, should normally be linked to achievement of minimum intended module or programme learning outcomes.

15. Assessment promotes and supports effective learning and teaching • Effective assessment is intrinsic to effective teaching and learning. • Effective assessment is (i) consistent with, (ii) supportive of, and (iii) derived from the intended programme and module learning outcomes. • Both module assessment strategies and programme assessment strategies are necessary for effective assessment. • The effort required of a learner to complete an assessment task should be proportional to the associated educational benefit to him or her. • Formative assessment supports learning • Teachers and learners share in the responsibilities for effective learning. Involvement of learners in the construction of assessment tasks and criteria can enhance learning. • Authentic assessment supports effective learning.

16. Assessment procedures are credible • Credible assessment is fair and consistent and more specifically it is valid and reliable • Fair assessment is inclusive • Those with a role in assessment should have the necessary competence and there should be no conflict of interest • Learners should be confident about the fairness and objectivity of the assessment procedures. (e.g. anonymous grading) • The intended programme and module learning outcomes and assessment strategies should be plainly written and communicated at the start of a programme to learners and all those involved with teaching and assessment. • The processes for assessment complaints and appeals should meet the same standards of fairness, consistency and fitness for purpose as assessment in general.

17. Assessment methods are reviewed and renewed as necessary to adapt to evolving requirements • Developing and testing new assessment methods, tactics and strategies is necessary for continual enhancement and to cope with emerging challenges.

18. Learners are well informed about how and why they are assessed • Learners need to be familiar with and understand the intended module and programme learning outcomes relevant programme and module assessment strategies and regularly reminded of these and the assessment regulations. • Providers should inform learners (e.g. in a programme handbook and programme web page) appropriately. • Learners should be involved in the periodic review of assessment procedures.

19. Concluding remarks • Standards express the purpose of education • Assessment measures the progress of education

20. www.hetac.ie • Thank you • Please visit our website for more information