Portfolio Assessment in Clerkship

1. Portfolio Assessment in Clerkship Michelle Gibson - Geriatrics (thanks to Chris Frank and Melissa Andrew too)

2. Objective • Participants will discuss the use of portfolios in assessing medical trainees (after a brief review of some basic principles…)

3. Background • Change to the Internal Medicine clerkship in January 2007 • 4 weeks to 6 weeks • Opportune time to review the assessment process in geriatrics • Previously, 8 assessment cards filled out by MD supervisors

4. Challenges with old system • Cards were often filled out the last week of the rotation • Depended solely on supervisor impression • No direct observation was incorporated • Students were assessed using only one modality

5. Reasons for change • Desire for more formative assessment opportunities • Desire for more robust assessment • (Need to meet course requirements!)

6. Formative assessment • Assessment that occurs during instruction to provide feedback to teachers and students • Describes needs for future learning • Assessment for learning

7. Summative Assessment • Assessment that occurs at the end of an instructional unit to document student learning • Certifies learning • Assessment of learning • Is this portfolio formative or summative?

8. High Quality Assessment • Clear & appropriate learning targets • Appropriateness of assessment methods* • Validity • Reliability • Fairness • Positive consequences* • Alignment* • Practicality & efficiency*

9. Reliability & Validity • Reliability: concerned with the consistency, stability, and dependability of the scores • Validity: the appropriateness of the inferences, uses, and consequences of the assessment • Is it assessing what you want it to assess?

10. Portfolio • A systematic collection of student products to assess progress • Structured portfolio • The assessor determines the mandatory content of the portfolio

11. Literature review • Assessment of housestaff in clinical rotations • Portfolio literature reviewed • Specific instruments examined • Portfolio components were selected to provide multiple methods of assessment for each clerkship objective

12. Components by Methodology

13. Mini-CEX

14. TAB

15. Card

16. Medication Review - written • Based on an actual patient, preferably one for whom the student has responsibility • brief summary of patient, • meds he/she is taking including whether or not they are appropriate • suggestion of other medications he/she should be taking

17. Self-assessment of write-up • H & P from early in rotation • Students asked to read and identify • 2 things done well • 2 things that could be improved • Reviewed with undergrad coordinator on the last day

18. Components & CPE Categories

19. Process • Orientation with students on Day 1 • Team, faculty, & residents are oriented to the portfolio • Students meet with undergrad coordinator at mid-term, review progress, identify any gaps, etc. • Last day - review portfolio with students, completion of CPE

20. Evaluation of Pilot • Focus on acceptability • Students and assessors • Student evaluations ongoing - not seen by the undergrad coordinator until collated in April 2008 • Will redo faculty portion in December 2007

21. Challenges noted - 1st time • First rotation ever for clerks • Time spent learning how to be a clerk • Couldn’t assemble half the portfolio in 3 weeks • Exam stress “detracted from our learning” • Supervisors were absent more than usual *all except exam stress have resolved

22. Strengths noted • Multidisciplinary team members provided high-quality feedback • Direct observation occurred • Forms were felt to be user-friendly, acceptable • Students appreciated discussing their medication review and being observed

23. Assessor ratings of forms • TAB • 5/5 for ease of use; 5/5 for usefulness • Mini-CEX • 5/5 for ease of use; 4/5 for usefulness • Time spent by assessor: 13.5 minutes • Encounter card • 4.5/5 for ease of use; 4.5/5 for usefulness • Time spent by assessor: 15 minutes

24. My time? • 5-10 minutes on day 1 for orientation • Plan to develop “talking power point” • 15 min per student at the end of week 3 • Mid-term review • Could be any faculty • 15 min per student on the last day • Final evaluation • Could be any faculty

25. My *impressions* • Students receive more feedback in a timely fashion • Direct observation doesn’t take long, and is very valuable • TAB form is very useful, and team members are pleased to be filling them out • Mid-term meetings are very useful

26. Future directions • Continue with student and assessor feedback • Incorporate other tools? • Peer assessment? • Patient assessment? • Study it more • Implement it for residents

27. Factors to note • Small unit, discrete group of nursing and other staff • Small division • Very workable for us … but for others?

28. Questions to consider • Is this high quality assessment? How could it be improved? • Can it be extended into other areas? • Is there a role for a “whole clerkship” portfolio? “Pre-clerkship” portfolio? • Residents & non-MDs as assessors? • Electronic?

29. Comments about our students • “ .. Is very attentive… he listens well to pt. & staff, & then goes off & evaluates what he’s been told” • “excellent verbal communication observed with all team members” • “better H & P than most residents”