Evaluation of Learners

1. Evaluation of Learners Effective Techniques and Strategies Jennie Hsu-Lumetta, M.D. March 16, 2005

2. “Clinical Learning is Experience Examined…” Ende and Davidoff, 1992

3. Goals and Objectives 1. Define the purpose of evaluations. 2. Identify the steps to providing effective evaluations. 3. List some potential barriers/errors to providing effective evaluation .

4. What Will Be Covered. . . 1. Definition, Purposes, Types of Evaluation 2. Pre-Evaluation Activities 3. Completing the Evaluation 4. Writing Good Comments 5. Discussing Evaluation 6. RIME Model for Clinical Evaluation

5. Definition of Evaluation To determine the value or worth of a Program, Product, Performance Process

6. Evaluation of Learners The process by which the teacher assesses the learners’ knowledge, skills, and attitudes, based on criteria related to educational goals.

7. Why Evaluate? Past: Assess performance Present: Improve performance Future: Identify future needs

8. Uses of Medical Student Evaluations Use Purpose Audience Past AssessmentClerkship Director Present Improvement Student Future Dean’s Letter Dean

9. Uses of Resident Evaluations Use Purpose Audience Past Assessment Program Director Present Improvement Resident Future Program Fellowship, Director’s Letter Employers

10. What to Evaluate? Attitude Skills Knowledge

11. ASK’sAttitude, Skills, Knowledge Competency includes the following: • Attitude: (as observed via behaviors) Prepared/Punctuality/Proactive Follow-up with patient/educational issues • Skills – context specific • Knowledge • Has the information • Able to analyze and synthesize information • Able to apply it effectively and efficiently

12. ACGME Competencies 1. Patient Care 2. Medical Knowledge 3. Practice-Based Learning and Improvement 4.Interpersonal and Communication Skills 5. Professionalism 6. Systems-Based Practice

13. ACGME Competencies Patient Care that is compassionate, appropriate, and effective for the treatment of health problems and the promotion of health Medical Knowledge about established and evolving biomedical, clinical, and cognate (e.g. epidemiological and social-behavioral) sciences and the application of this knowledge to patient care

14. ACGME Competencies Practice-Based Learning and Improvement that involves investigation and evaluation of their own patient care, appraisal and assimilation of scientific evidence, and improvements in patient care Interpersonal and Communication Skills that result in effective information exchange and teaming with patients, their families, and other health professionals

15. ACGME Competencies Professionalism, as manifested through a commitment to carrying out professional responsibilities, adherence to ethical principles, and sensitivity to a diverse patient population Systems-Based Practice, as manifested by actions that demonstrate an awareness of and responsiveness to the larger context and system of health care and the ability to effectively call on system resources to provide care that is of optimal value

16. ACGME - Evaluation of Residents • use of dependable measures to assess 6 areas of competency • mechanisms for providing regular and timely performance feedback to residents • a process involving use of assessment results to achieve progressive improvements in residents' competence and performance

17. Types of Evaluation • Formative Evaluation • Summative Evaluation

18. Types of Evaluation • Formative Evaluation • Assess a learner’s progress toward educational goals • Throughout the teaching process • Basis for feedback for the learners and to make teaching adjustments as needed to learner to reach learning goals

19. Types of Evaluation • Summative Evaluation • Final assessment of a learner • End of teaching/learning experience • Used to judge a learner’s mastery of the educational goals

20. How to Evaluate? Assessment/Evaluation • Formative (ongoing) Feedback Evaluation • Summative (end of rotation)

21. Elements of Effective Assessment • Evaluation Prerequisites • Making Good Comments • Aligning Comments with Scaled Questions • Discussing the Evaluation with the Learner Source: David Rubin, MD and James Woodruff, MD. Teaching Skills for the Medical Educator Workshop. University of Chicago, March 23, 2002

22. Effective Assessment 1:Evaluation Prerequisites • Understand expectations of the learner • Understand the evaluation criteria • Understand the survey instrument • Have appropriate expectations • Provide on-going feedback • Provide opportunities for observation

23. Key components of Evaluation • Observation of the Learners • Questioning

24. Key components of Evaluation :Observation of the Learners The teacher can observe learner performance for direct evaluation of knowledge, skills, and attitudes.

25. Key components of Evaluation :The One minute Observation • Explain the purpose of the observation to the learner • Explain how the observation will occur • Select one skill for observation • Inform the patient of what will take place • Observe for a brief period without interrupting • Leave the room and give immediate feedback • Use the information gain to plan your teaching • Repeat to observe learner’s skills over time

26. Key components of Evaluation :Questioning of the Learners • Types of Questions • Levels of Questions

27. Key components of Evaluation :Types of Questions • Open-ended • What? • How? • Why? • Closed-ended • Do? • Can? • Have?

28. Key components of Evaluation :Level of Questions • Recall • Analysis/Synthesis • Application

29. Effective Assessment 2:Completing the Form • Write narrative comments first • Comments are most important!! • Grades are a “blunt instrument” Dr. Rubin, U of Chicago Dept. of Medicine • Give specific examples • Keep comments to focused behaviors • Connect comments to ratings • Check for internal consistency

30. Sources of Comments • Direct observation of learner • Written records • Responses to questions • Homework assignments • Interactions with healthcare team, patients • Learner self-evaluation • Feedback/Midpoint evaluation • Interventions with the learner • Learner’s response to interventions

31. Effective Assessment 3:WritingEffective Comments • Strengths and weaknesses • Areas of concern • Suggestions for improvement • Ensure internal consistency • Focus on behaviors not personalities

32. Effective Assessment 4:Discuss with Learner • Set a time • Allow enough time for discussion • Have the learner perform a self-assessment • Explain your rationale • Emphasize changes in behavior and performance that leads to desired competency • Suggest areas for future improvement • Give evaluation in oral and written forms *Adding feedback element increases the usefulness of the evaluationand avoids surprises

33. Avoid rating errors: • The “halo/horn” effect: • based on reports of prior performance • Restriction range: • using the same rating for all components of the evaluation • Rating nonperformance attributes: • using behavior attributes when rating nonbehavioral qualities • Evaluate performance ONLY

34. RIME Model for Clinical Evaluation • Developed by Dr. Louis Pangaro • Model for student evaluation that was credible and reliable • Student performance -> Progression of 4 developmental steps • Research shows that it is predictive of future performance

35. RIME Model for Clinical Evaluation 4 Stages of Clinical Development: Reporter Interpreter Manager Educator

36. RIME Model for Clinical Evaluation • Reporter • collect and report data (by the M3 year) • Interpreter • interpret history, PE findings and lab results, create & prioritize a problem list and differential diagnosis (by end of M3 year) • Manager • select the most appropriate diagnostic and treatment options (by 2nd & 3rd residents) • Educator • identify and address knowledge gaps (senior residents)

37. Proficient Reporter • Interview & History • Physical examinations • Demonstrates adequate medical knowledge • Generates adequate problem list • Demonstrates respect for patients • Consistently reliable & honest • Interpreter • Interprets basic test data • Integrates medical knowledge with findings • Generates several reasonable explanations • Can discuss issues & process of informed consent

38. Advanced Manager • Develop diagnostic & therapeutic plans • Makes decisions using benefit/risk analysis • Uses evidence-based approach • Performs as a reliable member of the team • Manages relationships with patients

39. Outstanding Educator • Demonstrates leadership • Demonstrates self-directed learning • Exhibits critical reading skills • Demonstrates teaching skills

40. Summary for Effective Evaluators: • Understand learner expectations • Understand the evaluation instrument: • E-value, RIME model • Include thoughtful comments • Connect comments to rating scales • Have multiple opportunities for assessing performance • Evaluate performance ONLY • Discuss evaluation with the learner

41. Special thanks: • Nancy F. Barrett • Carol Packard

42. Case You sit down with your student on her last day of the rotation although you’ve never been comfortable giving evaluations. Knowing evaluations are supposed to be balanced, you try to list several positive things and then list problems. When you mention that the student’s cardiac patient who spent an extra day in the hospital because she didn’t track down a stress test result in time on Friday before the cardiology department closed, she angrily bursts out how difficult it is on a team where no one will give her any help. She also says that you haven’t spent as much time reviewing write-ups with her as other residents.