Preparing Final Clerkship Performance Evaluations

1. Preparing Final Clerkship Performance Evaluations A Guide for Clerkship Directors and Evaluation Teams

2. Table of Contents • Data collection • Criteria for Pass, Pass with Distinction • Interpreting evaluation data: • Patient Care (RIME) • Professionalism and Interpersonal Communication • Final grades • Narrative evaluations • Appeals • Resources and Contacts

3. Data Collection

4. Data Collection • Ideally • 100% of potential evaluators • will complete electronic evaluation forms in E*Value. • within a week of an evaluation being assigned.

5. Data collection • In reality • The minimum acceptable return rate is 50%. • Final evaluations must be submitted within 6 weeks of the end of the clerkship. • Some evaluators will not want to use E*Value.

6. Recommendations • Keep an eye on return rates – start reminding and re-requesting early. • Clerkships may need to tailor approaches to data-gathering to suit the needs of different groups, sites, or individual evaluators

7. Recommendations • Avoid going into an evaluation team review or submitting final evaluations with a return rate under 50%. Evaluations based on input from fewer than 50% of potential evaluators are unlikely to stand up in an appeal. • All evaluators who submit information (residents, faculty, fellows, non-MD staff, peers) should be listed as Contributing Evaluators in the final evaluation form.

8. Data Collection • For ideas, resources, and practical support in boosting evaluation return rates, contact • Jen Deitz (jdeitz@stanford.edu) • Gretchen Shawver (gshawver@stanford.edu) or • Elizabeth Stuart (aestuart@stanford.edu)

9. Criteria for Pass and Pass with Distinction

10. Criteria for Pass • Patient Care: • Direct observations of clinical skills complete (2 per clerkship) • RIME Interpreter • Professionalism/Interpersonal Communication: • No significant or consistent concerns • Patient logs, other assignments complete • Exam score above clerkship passing threshold

11. Criteria for Pass with Distinction • All Passing criteria must be met AND • Patient Care: • RIME Manager • Professionalism/Interpersonal Communication: • Multisource feedback requested: 1 non-MD staff member, 1 patient, 1 peer • Evidence of Exceptional Professionalism/IPC • Exam score above clerkship-specific threshold for Pass with Distinction

12. Two paths to distinction Performance Threshold for Distinction Time Students may earn Pass with Distinction by meeting criteria throughout the clerkship OR improving to meet criteria by the end of the rotation

13. Interpreting Evaluation Data:Patient Care (RIME)

14. Expected Transitions MANAGER INTERPRETER REPORTER Core clerkship students are expected to be in the Interpreter stage. POM Core clerkship Sub-I Residency +

15. Interpreting Data from Patient Care (RIME) forms • For a student in the Interpreter stage: • Nearly all evaluators will select Consistently True for items in the Reporter section. • Data will include a mix of Sometimes and Consistently True for items in the Interpreter section.

16. Reporter Items

17. Interpreter Items

18. Alert • If significant number of evaluators - throughout the rotation - have selected Sometimes or Rarely True on Reporting items – the student has not met expectations for performance in a core clerkship and should receive a non-passing grade. • Clerkship directors should speak directly with individual evaluators to confirm that a student is not consistently demonstrating the skills required in the Reporter stage.

19. Pass with Distinction MANAGER INTERPRETER REPORTER POM Core clerkship Sub-I Residency + Pass with Distinction requires consistently strong Reporting and Interpreting (solid bars) with evidence of being in the transition to the Manager stage.

20. Interpreting Data from Patient Care (RIME) forms • For a student in the Manager stage: • Nearly all evaluators will select Consistently True for items in the Reporter and Interpreter sections. • Data will include a mix of Sometimes and Consistently True for items in the Manager section.

21. Manager/Educator Items

22. Broadly • Students in the Manager stage are functioning above the expected level – at the level of a sub-intern or beyond.

23. Q & A Q: Should core clerkship students really be expected to manage patient care?

24. Q & A: “Manager” A: The term Manager is not meant to imply independent implementation of patient care plans. Students who are Managers demonstrate a sense of personal responsibility for knowing as much as possible about their patients and ensuring that they receive optimal care. Managers see themselves – and are seen by others – as patients’ primary providers and advocates.

25. Q & A: M without I? Q: The evaluators for one of my students marked Consistently True for all of the Manager items, but only Sometimes True in the Interpreter section. Does the student meet criteria for being a RIME Manager?

26. Q &A: M without I? A: No. Pass with Distinction requires consistently strong Reporting and Interpreting skills in addition to the skills and attitudes represented by the Manager items on the evaluation form.

27. Q & A: Outliers Q: What if one evaluator checked Rarely or Sometimes True on an item – when all other evaluators marked Consistently True?

28. Q & A: Outliers A: When reviewing data for both Patient Care and Professionalism/Interpersonal Communication, Evaluation Teams should look for trends and themes - over time and across evaluators.

29. Outliers In sorting through outliers, consider: • Role/identity of the evaluator • Setting • Time the evaluator spent with the student • Timing during the rotation

30. Interpreting Evaluation Data:Professionalism and Interpersonal Communication (IPC)

31. Fundamentals of Professionalism and IPC

32. Interpreting data from the Professionalism/IPC form • To Pass, there should be a consistent trend of meeting expectations, with no significant or consistent concerns.

33. Exceptional Professionalism/IPC

34. Interpreting data from the Professionalism/IPC form • To Pass with Distinction, data from multiple evaluators should show a consistent trend of exceptional Professionalism/IPC: • “Yes” answers to the yes/no question • Narrative examples of exceptional professionalism and interpersonal communication

35. Multisource Feedback (MSF) • To meet criteria for Pass with Distinction, students must request multisource feedback from • 1 non-MD staff member • 1 patient • 1 peer • A response by peers, non-MD staff, patients is not required. • Clerkships must put systems in place to record students’ MSF requests.

36. Q & A: Multisource Feedback Q: What if a non-MD staff member reports concerns about professionalism? Does that disqualify the student for PWD? Q: What if a patient or non-MD staff member submits a glowing description of a student’s Interpersonal Communication? Shouldn’t that information count toward Pass with Distinction?

37. Q & A: Multisource Feedback A: For the initial phase of CBEI, the content of multisource feedback is not considered in determining whether a student has met criteria for Pass with Distinction Positive comments from MSF content may be used as examples in the final narrative evaluation.

38. Final Grades

39. Final grade options • As of Period 1, 2010 the following final grade options are available: • N - Continuing • Fail • Marginal Pass • Pass • Pass with Distinction

40. N grade • N is for failed exams. • N should not be used for: • Marginal/non-passing performance in the domains of Patient Care or Professionalism/Interpersonal Communication • Missed time from the clerkship • Incomplete assignments • N is only for failed exams.

41. More on N See MD Program Handbook for additional details.

42. Q & A: N grade Q: I have a student who was admitted to the hospital with appendicitis the night before the shelf exam. Should I wait to submit his evaluation until he takes the exam?

43. Q & A: N grade A: No. The evaluation should be submitted with an N grade within 6 weeks of the end of the clerkship. Evaluations contain important information that should be made available promptly to students. Evaluation submission should not be delayed by failed or missed exams.

44. Marginal Pass vs. Fail See MD Program Handbook for additional details.

45. Q & A: Evaluation Review Shortcuts? Q: Does the full evaluation team need to review files for all students, or just those who seem likely to meet criteria for Pass with Distinction?

46. Q & A: Shortcuts? A: CBEI introduces not only a new set of options for final grades, but also a new process to ensure a fair and balanced review for each student. All students should benefit from the process of a full review.

47. Shortcuts? continued • In addition, the final MSPE will indicate whether a student has met criteria for PWD not only as a final, overall grade, but in each domain of performance. • See the next slide for an example of how performance may be summarized in the MSPE.

48. MSPE Reporting idea All students require a thorough review so that the MSPE can fairly reflect their individual strengths.

49. Q & A: Second year students Q: What about students who started clerkships before Period 1 2010? Their final evaluations should just say Pass, right?

50. Q & A: Second year students A: No. All students’ performance should be assessed in the same manner, regardless of the timing of starting clerkships. All students who meet criteria for Pass with Distinction should be assigned a PWD on the final evaluation form. For students who started clerkships before Period 1, 2010, the Advising Deans will format MSPEs to reflect the pre-CBEI, “ungraded” evaluation system.