Disciplinary standards and Assessment

1. Bologna Expert Colloquium Supporting the Design of Discipline-Specific Learning Outcomes 6 February 2009 Disciplinary standards and Assessment Peter Cullen Head of Standards, Research and Policy Development Unit HETAC www.hetac.ie

2. HETAC’s mandate includes • to determine “standards” of knowledge, skill or competence … • “validation” means the process by which an awarding body shall satisfy itself that a learner may attain knowledge, skill or competence for the purpose of an award made by the awarding body; • Review of the effectiveness of agreed quality assurance procedures review of delegation of authority…

3. What is a standard? • It’s an educational goal • …a required or specified level of excellence, attainment, … (from Oxford English Dictionary NSOED 1997) • knowledge, skill or competence to be acquired by learnersbefore a higher education and training award may be made (Qualifications Act)

4. What is it for? • For learners! • For the provider to use when designing and reviewing intended programme learning outcomes • For the awarding body to use when validating programmes • For internal QA to use when reviewing performance • For external examiner to use along side the intended programme learning outcome • For the external quality assurance agency to use when auditing/assessing/accrediting • For the employer when interpreting the significance of qualifications

5. Why not rely on tacit standards? • Traditional approaches to assessment in higher education typically place heavy reliance on tacit understandings of standards and can be strained in new or rapidly changing contexts. • Examples of changing contexts which have encouraged practitioners to look for innovative approaches include • massification, new kinds of teaching and learning, modularisation, plagiarism, learner retention, computer-aided assessment, new kinds of intended learning outcomes, availability of study time, and declining resources. • Tacit element is ineluctable • connection with community of practice/learning

6. Types of standards and setters Particular General NQF Award-type descriptor Named award or broad Field of Learning standard Narrow discipline Standard or Regulated Profession Intended programme Learning outcome Nursing Programme X Science Programme Y Honours BachelorDegree Engineering Generic NQAI HETAC approach FETAC approach Tuning/QAA Universities and DIT IOT and other HEI

7. Standards and precision • Precision needs depend on how the standard is to be used. Is it to check that the mean performance level is on target? Or is it to check that the vast majority of students graduate attain or exceed the standard? Is it used for summative assessment of learners? • Note that the QAA benchmark statements contain threshold levels as well as other performance levels) • Standards can be representative (describing characteristics to be possessed by most) • Standards can be threshold (characteristics to be possessed by all) • Standards can be typical (the meaning of this varies) • Can standards that describe the average (mean performance) be used for summative assessment of learners? • What is important is clarity and precision but not too much precision because flexibility is necessary. The nearer the standard is to the ‘programme’ the more precise it should be– the next slide illustrates

8. Synthesis of the intended programme learning outcomes Note that stages emerge naturally from the network of learning opportunities in a learning environment Learning environment (including people interacting) Module Module Module Graduates Entrants Module Module Module Module … …

9. Intended programme learning outcomes (the provider’s standard) • Intended programme learning outcomes describe the minimum achievement (in terms of knowledge, skill and competence) that the learner is certified to have attained if he/she successfully completes a particular programme (i.e. passes all the required assessments). • The intended programme learning outcomes are the principal educational goal of the programme and effective assessment helps learners to attain that goal. • Teachers and learners may strive for additional learning outcomes beyond the minimum as expressed by the intended programme learning outcomes. • Intended programme learning outcomes are developed and maintained by providers. • Programmes are designed to enable learners to achieve intended programme learning outcomes. • Intended learning outcomes are specified for each of a programme’s constituent modules. • The number of learning outcomes in a statement of intended learning outcomes is variable (depending, for example, on the semantics and the level of explicitness used) and it is not a proxy for credit.

10. Implementing standards • Note the implementation gap: remember that tacit interpretation operates on all standards • Note the implementation lag: It will take time for providers to align to new standards (unless you choose the least as the standard) and assessment is key • A necessary condition for implementation is being assessable: for if something can be assessed it can be used to control the learning environment appropriately • What does assessable mean • Test for certification related (examine learners) • Test for qualification—how does this differ from certification? • Doing longitudinal research (learners and graduates and employers) • Other… • Student assessment procedures are expected to be designed to measure the achievement of the intended learning outcomes and other programme objectives. (ESG) • Student assessment procedures are expected to have clear and published criteria for marking (ESG)

11. Six assessment principles • Learners are responsible for demonstrating their attainment • Assessment supports standards based on learning outcomes • Assessment promotes and supports effective learning and teaching • Assessment procedures are credible • Assessment methods are reviewed and renewed as necessary to adapt to evolving requirements • Learners are well informed about how and why they are assessed • Please have a look at the ‘Assessment and Standards’ consultation document on the HETAC website

12. Writing standards • HETAC uses the following process • Background research • Standards group (recognized experts) • Formal public consultation • Review (not yet done) • Vague aspirations and hyperbole don’t sit well with standards • Lots of literature to refer to • Note the warnings about over-dependence on taxonomies

13. Generic standards • The award-type descriptors of the NFQ are being used by HETAC as generic awards standards. • Generic standards are the kernel of the awards standards

14. HETAC Awards Standards • Statutory: determine the standard to be attained before an award may be made • Overall standards set by the NQAI through the NFQ • NFQ award-type descriptors are kernels • Broad • Flexible (not all individual outcomes mandatory) • Pairs of standards can be used together • Silent on the curriculum, assessment, duration, except by what is implied by learning • Based on the disciplines of the NCEA boards of study with some additions including standards for regulated professions

15. Pilot standards • Fields of Learning (ISCED 1997) • Art & Design • Business • Computing • Engineering • Nursing • Science • Complementary therapies • Currently looking at (or considering looking at) • Architecture • Social care • Education • Others??? • Have been requested to look at Humanities!?

16. Why broad • Standards should be sufficiently flexible to allow a range of approaches, appropriate to different sub-fields within fields of learning and different providers. • Flexible • Innovative and creative programme design • Avoids mainstreaming • Allows programmes to adapt to changing needs and opps • Tolerates diversity • Facilitates multi-disciplinary programmes • Practical for a small country • Can be reviewed periodically • Reduced likelihood of obsolescence • QAA currently have 55 benchmark statements at honours bachelor degree level alone (there is one for engineering… • Providers can supplement the HETAC broad standards with Tuning and QAA descriptors as well as professional body statements of learning outcomes • Does it work? • HETAC (standards have been in use for four years) • Research will tell • Anecdotal evidence is definitely positive but some modifications are necessary

17. www.hetac.ie • Thank you • Please visit our website for more information