Students’ assessments in medical education

1. Students’ assessments in medical education AMAL AL OTAIBI CCP, MME

2. Objectives • Define assessment. • Identify purpose and importance of assessment. • Enumerate different methods used in medical education. • Differentiate between formative & summative assessment.

3. Why Discuss Assessment?

5. What is Assessment? The word ‘assess’ comes from the Latin verb ‘assidere’ meaning ‘to sit with’. In assessment one is supposed to sit with the learner. This implies it is something we do ‘with’ and ‘for’ students and not ‘to’ students (Green, 1999).

6. What is Assessment? Assessment in education is the process of gathering, interpreting, recording, and using information about students’ responses to an educational task. (Harlen, Gipps, Broadfoot, Nuttal,1992)

7. Values and Attitudes about Assessment • Teachers value and believe in students. • Sharing learning goals with the students. • Involving students in self-assessment. • Providing feedback that helps students recognize their next steps and how to take them. • Being confident that every student can improve. • Providing students with examples of what we expect from them.

8. Types oF Assessment • Formative Assessment • Summative Assessment

9. Formative Assessment • Assessment for learning • Taken at varying intervals throughout a course to provide information and feedback that will help improve • the quality of student learning • the quality of the course itself

10. Provides information on what an individual student needs • To practice • To have re-taught • To learn next

12. Key Elements of Formative Assessment • The identification by teachers & learners of learning goals, intentions or outcomes and criteria for achieving these. • Rich conversations between teachers & students that continually build and go deeper. • The provision of effective, timely feedback to enable students to advance their learning. • The active involvement of students in their own learning. • Teachers responding to identified learning needs and strengths by modifying their teaching approach(es). Black & Wiliam, 1998

13. Summative Assessment • Assessment oflearning • Generally taken by students at the end of a unit or semester to demonstrate the "sum" of what they have or have not learned. • Summative assessment methods are the most traditional way of evaluating student work. • "Good summative assessments--tests and other graded evaluations--must be demonstrably reliable, valid, and free of bias" (Angelo and Cross, 1993).

14. Formative and summative assessment are interconnected. • It is widely and empirically argued that formative assessment has the greatest impact on learning and achievement.

15. Formative • ‘… often means no more than that the assessment is carried out frequently and is planned at the same time as teaching.’ (Black and Wiliam, 1999) • ‘… provides feedback which leads to students recognizing the (learning) gap and closing it … it is forward looking …’ (Harlen, 1998) • ‘ … includes both feedback and self-monitoring.’ (Sadler, 1989) • ‘… is used essentially to feed back into the teaching and learning process.’ (Tunstall and Gipps, 1996) Summative • ‘…assessment (that) has increasingly been used to sum up learning…’(Black and Wiliam, 1999) • ‘… looks at past achievements … adds procedures or tests to existing work ... involves only marking and feedback grades to student … is separated from teaching … is carried out at intervals when achievement has to be summarized and reported.’ (Harlen, 1998)

16. Purposes of Assessment FORMATIVE • To provide a profile of what a student has learnt. • To provide feedback to students so they know where to improve their learning. • To motivate students. • To help students to develop their skills of self-assessment. • To improve teaching. SUMMATIVE • To evaluate a student’s strengths and weaknesses. • To pass or fail a student. • To grade or rank a student. • To license to proceed. • To license to practice. • To select for future employment.

17. Importance of Assessment for Learning [Brown & Knight] “Assessment is at the heart of the student experience” [Chalmers] “Assessments define what students see as important” [C. Van derVleuten] -“Tests drive learning”

18. Importance of Assessment for Learning (cont’d) Students take cues from what is assessed rather than from what their lecturers assert is important “If you want to change student learning then change the methods of assessment” [Chalmers]

19. Weaknesses in Assessments • not be representative of the student’s capabilities • not match learning objectives of the course • draw on too narrow a domain (e.g., only written exams used) • be over-weighted with particular skills & underweight others

20. Common Weaknesses in Assessment Systems • Overload of students and staff. • Too many assignments with the same deadline set in the department/school • Insufficient time for students to complete assignments • Insufficient time for staff to mark the assignments • Inadequate or superficial feedback provided to students • Wide variations in assessment demands by different courses/modules • Wide variations in marking across courses/modules. [Brown, Bull and Pendlebury]

21. Other Common Weaknesses in Assessment Systems • Wide variations in marking within a module by tutors • Fuzzy or non-existent criteria. • Students do not know what is expected of them • Students do not know what counts as a good or bad assignment/project. • Assessment viewed by some departments as an extra rather than a recognized use of staff time Brown, Bull and Pendlebury

22. Educational impact Educational impact Reliability Reliability Validity Validity Cost Acceptability Characteristics of instruments

23. Reliability (the degree to which the measurement is accurate and reproducible)

24. Validity (whether the assessment measures what it claims to measure)

25. Behaviour Professional authenticity Cognition A simple model of competence Does Shows how Knows how Knows Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

26. Professional authenticity A simple model of competence Performance or hands on assessment Does Shows how Written, Oral or Computer based assessment Knows how Knows Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

27. Performance assessment in vivo: Masked SPs, Video, Audits….. Does Performance assessment in vitro: OSCE, SP-based test….. Knows how (Clinical) Context based tests: MCQ, essay type, oral….. Knows Factual tests: MCQ, essay type, oral….. Shows how Climbing the pyramid...... Does Shows how Knows how Knows

28. assessment methods • Written examinations • Multiple choice • True/False • Matching • Short answer • Essay • Oral

29. new assessment methods • Modified essay question (MEQ) • Objective Structured Clinical Examination (OSCE) • Mini-Clinical Evaluation Exercise (Mini-CEX) • 360 degree (multi source feedback) • Portfolio

30. Modified essay question (MEQ) • An assessment method where you have a scenario followed by sequential questions. • Strengths • Good reliability. • It measure students’ skills in clinical reasoning and problem solving abilities using what they know about the topic. • Weaknesses • Need resources (human) • Time consuming • A lot of paper work. • It can’t be used alone.

31. What is an OSCE • Objective • Structured • Clinical • Examination Harden RG and Gleeson FA Assessment of clinical competence using an objective structured clinical examination (OSCE) Medical Education,1979, Vol 13: 41-54

32. OSCE-Objective • All the candidates are presented with the same test • Specific skill modalities are tested at each station • History taking • Clinical examination • Procedures

33. OSCE - Structured • The marking sheet for each station is structured. • Structuredinteraction between examiner and student

34. OSCE – Clinical Examination • Test of performance of clinical skills • candidates have to demonstrate their skills, not just describe the theory

35. Skills – which skills Clinical Diagnostic Communication skills Procedural skills

36. OSCE Stations Types • Clinical stations involving interaction between the student & a real or simulated patient. • Practical stations involving performance of technical procedures (e.g. performing CPR). • Static stations do not involve any physical interaction but the task may be very varied.

37. Essential elements of OSCE Stations • A clear definition of the task to be performed by student. • Precise and clear instructions to students,examiners and simulated patients. • A rating form or marking sheet. • A list of requirements (e.g. Equipments,scripts for the simulated patients.

39. Title Sheet

40. Candidate Instruction

41. Examiner Instruction

42. Marking Specification

43. Marking Sheet

44. Requirements List

45. Simulated Patient Instructions

46. Classification

48. Rear Entrance Kitchen Ali Hadwer Mr. Shakil Stock Room CR Auditorium Students ASSEMBLY OSCEExam. Flow Library Control Room Computer Lab. CR Stock Room CR Tutorial Rm. 5 Extra Rest Station for Round - 3 PABX Room PABX Room Station - 7 Tutorial Rm. 6 Station - 6 Tutorial Rm. 4 Station - 1 Tutorial Rm.. 7 Laboratory 1 Tutorial Rm. 3 Station - 2 Laboratory 2 Fire Exit Fire Exit MIT Room Female Waiting Tutorial Rm. 2 Station - 3 Station - 5 Electrical Room Conference Room Tutorial Rm. 1 Station - 4 Secretary Rest & Refreshment Area Secretary CR Main Entrance

49. College of Medicine OSCE Regulation • Each station is 7 minutes. • 1 minute is allowed to students to move between stations & examiners to arrange themselves. • Minimum 7 stations in each exam, maximum 9. • Training of simulated patients & examiners is obligatory. • Feedback session among the examiners & Block Coordinator after the OSCE exam.

50. Advantages of OSCE Every candidate does the same examination Greater opportunity for objectivity OSCE Stations are re-useable   