How to avoid pitfalls in clerkship assessment by medical students?

How to avoid pitfalls in clerkship assessment by medical students? T. Pottecher, M. Lujic, M. Wolf, C. Martin, G. Vicente, B. Ludes Educational research laboratory Faculté de Médecine de Strasbourg

Overview • First step • Assessment of procedure • Changes • And now…

The setting… • 2002, new decennial team • Creation of a unit assessing clinical and theoretical teachings • Appointment of two mission representatives • Instauration of periodical meetings

Key principles • The Faculty has acknowledged the educational role of clinical clerkships • Optimization & coordination of the clerkships: • define endpoints • create a precise logbook which incorporates the endpoints • Transparence & feedback • assess the quality of clinical clerkships • discussion, analysis and diffusion

Organization of clinical clerkship for medical students • Acquisition of endpoints: • Including : • tutorials • clinical case presentations by medical students • Student’s exam (end of clerkship) • presence, motivation, case scenarios, achievement of endpoints • pass or fail grading • Clerkship quality evaluation by students

Endpoints Questions Evaluation sheet

Clinical specialty coordinator Hospital Direction Department coordinator Head of department Medical faculty

Specialty professor defines clinical clerkship endpoints analyses the feedback together with a commission changes the endpoints as required Department Appointed by the head of department Organizes the clinical clerkship: student assignments into groups tutor appointments tutorials and clinical case presentations Final grading on pass/fail basis Ascertains that pre-set endpoints are being met Coordinators:

Feedback processing • Clerkship with minimum duration of 1 month • Assessment sheet • pre-printed (specialty, department, period) • sent to the heads of department before the end of the clerkship period • completed anonymously at the end of the clerkship • Sent back to the Faculty by the (secretary of the) department

Feedback processing(cont’d) • Analysis via optical reader • Manual processing of free comments • Feedback to each head of department and department coordinator

2006-2007 : onset of problems • Low response rate & “silent” departments • Delayed feedback towards the heads of department • Poor reaction by the heads of department and coordinators • Reliability of results? (no responses) • Reorganization of clinical assignments by “poles” (clusters of activity) • Appointed pedagogic referent • Single or multiple specialty poles (clusters of departments) endpoints assessment impossible

Exhaustiveness of feedbacks

But, some improvements…

Critical analysis of the question in assessment sheets Ginsburg S & al. Toward authentic clinical evaluation: pitfalls in the pursuit of competency. Acad Med. 2010 May;85(5):80-6.

2008 : What’s new? • New evaluation sheet, abolishment of the pre-set endpoints • Online assessment of clerkship by students • Feedback quaterly to pedagogic referent • Assessments results published on line • Study assessing individual student motivation • Information campaign

Example of a report

Factors for responding to evaluation sheets

Evaluation is useless..

To conclude…. • Four commandments of educational assessment • - Choose objectives • Sheet • Define Process • Organize Feedback Malhotra A & al. Mind the gap: Learners' perspectives on what they learn in communication compared to how they and others behave in the real world. Patient Educ Couns. 2009 Sep;76(3):385-90. Chun MB. Pitfalls to avoid when introducing a cultural competency training initiative. Med Educ. 2010 Jun;44(6):613-20

How to avoid pitfalls in clerkship assessment by medical students?