How to be a Good Examiner

1. How to be a Good Examiner Assessment Inset Day - 6 January 2011 Stephen May

2. Papers and Books? • A lot on teaching in HE • Less on assessment, and very methods-oriented • Some on role of external examiners (including Code of Practice) • Very little on the subject/discipline (internal) examiner![Most available lists of relevant qualities relate to public examination roles.]

3. Complementary Roles Internal Examiner External Examiner Student(s)

4. Possible to extract from assessment and staff development literature the key principles which need to be understood, and qualities which are required.

5. Key Elements • Knowledge of subject • Knowledge of students • Knowledge of course, and learning outcome expectations • Assessment knowledge and skills • Good written communication skills (and verbal, if using oral examinations) • Good organisational / team working abilities • The right attitude!

6. Knowledge of Subject • Important for question-setting (and also marking if any subjectivity involved) • Less important, for marking stage, the more objective the test • Ramsden (2003) – rules for better assessments • “Focus on validity before reliability”! (Avoid testing trifling aspects because they are easier to measure.) • “Use multiple choice and other “objective” tests cautiously” • So subject knowledge will always be important in all aspects of professional examinations, • but .... not nearly enough of itself!

7. Knowledge of Students and Course Expectations • Understanding by students of the assessment “game” • Knowledge of learning outcomes, and the implications for assessment • Knowledge • Reasoning • Decision-making • Technical skills • Communication skills • Professional attributes?

8. Assessment Knowledge and Skills • Assessment Purposes (Code of Practice, Section 6) • Evaluating student knowledge, understanding, abilities or skills • Providing a mark or grade .... may also be used to make progress decisions • Enabling the public (including employers) .... to know .... fitness to practise • Promoting studentlearning .... feedback

9. Assessment Knowledge and Skills • Assessment Methods • Subjective v. objective • Holistic v. fragmented • Higher order v. factual information and rote learning • Strengths and weaknesses of different formats Remembering: • Links to learning outcomes • Hierarchical skills development across course

10. Assessment Knowledge and Skills • Assessment Criteria / Marking Schemes • Objective methods (e.g. multiple choice): straightforward, increasingly do not need a marker • Subjectivemethods (e.g. long answer) • Knowledge • Structure of response • Clarity of expression • Level of engagement with question

11. Level of Engagement with Question • Bloom’s taxonomy (1956) • Lower to higher order skills • Biggs’ SOLO taxonomy (1997, 1999) • Ability to structure understand, integrate and reconceptualise • Hatton and Smith (1995) • Evidence-based, but self-critical review of knowledge and skills • Problem is interpretation of separate components, and weighting for aggregation

12. Assessment Knowledge and Skills • Assessment Criteria / Marking Schemes – Problems (Knight 2001) • Difficult to develop statements of learning outcomes • What is “critical thinking”? • Your good communication = my bombast • Precision produces wordy and complex documents and looser statements reduce clarity, so need a balance • Must not confuse criterion-referenced judgements with criterion-determined ones • All criteria subject to social processes by which meanings are constructed (both examiners and students)

13. Assessment Knowledge and Skills • Assessment Criteria / Marking Schemes – Problems • Long answers require clear individual understanding of the weighting process, and common understanding across examiner teams • Hawk / dove effect difficult to avoid • Key = comparable ranking of candidates • Occasional “random markers” exposed by public examining boards (very difficult to train, unless have not understood criteria)

14. Assessment Knowledge and Skills • Assessment Standards – Establishing Borderlines • Norm referencing – based on set percentages on each occasion obtaining certain grades • Criterion referencing – based on linkage to specific, pre-agreed levels of achievement

15. Assessment Knowledge and Skills • Assessment Standards – Establishing Borderlines Criterion referencing • For professional course, external expectations (day one skills) • For other courses, collective academic expectations • “Traditional” Oxbridge undergraduate standard: • Reasonable account of the examiner’s current knowledge 2(ii) • Account intelligently discussing knowledge the examiner once knew 2(i) • Stimulating debate of concepts and issues the examiner did not know, but can recognise as legitimate 1st • But now embedded in progressive marking schemes (such • as 17-point scheme) based on collective agreement

16. Assessment Knowledge and Skills • Assessment Standards – Establishing Borderlines Criterion referencing • In long answer-type assessments, examiners have worked towards 50%, based on perceived degree of difficulty of question, memory of previous groups of similar students and perceived requirement to pass • “Worked” from point of view of examination, but often involved a norm referencing component at individual examiner level • Disastrous approach for objective examinations – must be standard set unless all questions are tested for level of difficulty in advance and mixed in appropriate ratios!

17. Good Written and Communication Skills • Question-Writing Traditional essays • Less work in advance, major workload associated with marking • Writing (relatively) straightforward • Standard setting during/after marking Objective questions (MCQ/EMQ) • All work in advance (including standard setting) • Much more time and care required for question writing; often team-based • Standard setting level to defensibly adjust after the event

18. Good Written and Communication Skills • Question Writing • Need to avoid ambiguity • Need to be succinct (appropriate structuring for MCQ stems and distractors) • Need to anticipate all legitimate interpretations • Need to avoid the “If Ms/Mr/Dr/Professor X, they will be expecting .... answer” if the question does not uniquely demand that answer

19. Good Written and Communication Skills • Question Reading • “It’s important to be able to pick up what the candidate is saying.” • Distinguish between what they actually did mean (which then deserves credit) and what you think they may have said / meant (which deserves no credit)

20. Good Organisational Ability • Getting questions written on time (the reason why most people do not want to chair examination boards) • For internal moderation • For external scrutiny • For final examination papers • Getting the marking done on time • Telling people well in advance if you have a legitimate reason for not being available! • The examination process (like all team activities) is only as good as its weakest link!

21. The Right Attitude • “You don’t want a smart Alec”: selecting examiners to assess doctoral dissertations (Kiley 2009) • Experienced supervisors ensure they know the personality traits of potential examiners • Avoidance of the “bad and the mad” • Need a balance of empathy and understanding, with standards and integrity • Keen to improve: contribute to updating assessment methods and learning to use these effectively

22. A Compulsory Examination for All Assessors! (Modified from Brown 2001) • Answer all questions. Consultation with others (including students) is RECOMMENDED. • How well does your approach to assessment align with your learning outcomes? • Justify and criticise your choice of assessment methods (with reference to relevant research). • Describe, justify and criticise your assessment criteria and methods of grading/marking • Describe, justify and criticise your techniques for overcoming: • Variations in standards – on a single and different occasions • Variations in assessors • How do you ensure that your standards are similar to those in comparable assessments? • What values underlie your approach to assessment, and how are they manifest in your practice?

23. References: • Biggs, J. (1997) Enhancing teaching through constructive alignment. Higher Education 32, 347-364. • Biggs, J. (1999) Teaching for Quality Learning at University. Open University Press, Buckingham. • Bloom, B.S. (1956) Taxonomy of Educational Objectives, the Classification of Educational Goals. Handbook 1: Cognitive Domain. McKay, New York. • Brown, G. (2001) Assessment: Guide for Lecturers. LTSN Generic centre Assessment Series No. 3, LTSN, York. • Hatton, N. and Smith, D. (1995) Reflection in teacher education: towards definition and implementation. Teaching and Teacher Education 11(1), 33-49. • Kiley, M. (2009) “You Don’t Want a Smart Alec”: Selecting Examiners to Assess Doctoral Dissertations. Studies in Higher Education 34, 889-903. • Knight, P. (2001) A Briefing on Key Concepts: Formative and Summative, Criterion and Norm-Referenced Assessment. LTSN Generic Centre Assessment Series No. 7, LTSN, York. • QAA (2006) Code of Practice for the Assurance of Academic Quality and Standards in Higher Education. Section 6: Assessment of Students. QAA, Gloucester. • Ramsden, P. (2003) Learning to Teach in Higher Education, 2nd Edition. Routledge Falmer, London.