Module 4: Evaluation and Feedback

1. Module 4:Evaluation and Feedback Residents as Teachers & Leaders Module Created by: Charlene M. Dewey, M.D., M.Ed., FACP Associate Professor of Medicine Teri L. Turner, M.D., M.Ed., MPH Associate Professor of Medicine & Joan Friedland, M.D., MPH Associate Professor of Medicine Web Page and Module Formatting by: Maria Victoria Tejada-Simon, Ph.D. Assistant Professor of Molecular Physiology and Biophysics

2. Welcome • This is Module 4: Evaluation and Feedback. You should have completed a pretest for this module. Did you complete the “honesty pledge” question? • In that pledge you agreed to take the pretest first, then read the module and then take the post-test after reading the module. If you did not complete the pretest, please exit the module now and complete it; then return to the module. Your honesty is appreciated. Click here to continue. Exit Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

3. Welcome • By now you are familiar with the general format of the RATL modules. This module also consists of a pre-assessment, content of the topic and a post-assessment tool. • There are 148 total slides; many are games and interactive activities. Overall it should take about 45-60 minutes to complete this module. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

4. Welcome • However, at the completion of Module 4 we will ask you to complete the Teacher Identity Scale for the second time. • Please remember to provide feedback on the module content and format by completing a module 4 evaluation form found on the main RATL web page. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

5. Introduction • So what are you going to say about these teaching modules? First, you need to compare them to something that is known to be a good example or a gold standard. Then you must decide if they were as good as the gold standard and why. Then you will have to organize your thoughts to provide us with feedback that can help us improve the modules for the future. Lastly, you must convey those thoughts by writing an evaluation. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

6. Introduction • If you do all that and then never send us your comments, we will continue to think these modules are helpful in teaching residents how to teach. If they are not, and you don’t let us know, then this is an educational tragedy! • You do not want to spend a week, a month or a semester teaching learners and never let them know if they made the grade. That would truly be tragic! Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

7. Introduction • What we are talking about? This example covers assessments, feedback and evaluations. All are key concepts in teaching around a learner’s performance. It is part of the teacher identity and if done correctly, you will elevate learners to new levels of knowledge, skill and attitudes. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

8. Introduction • Module 4: Evaluation and Feedback focuses on accurately assessing a learner's level of ability, providing effective feedback and writing appropriate evaluations. • Your role as a teacher will be enhanced by adapting appropriate assessment, evaluation and feedback principles. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

9. Goals • Our goals for Module 4 are that participants will review: • General concepts of learner assessment • ACGME core competencies • How to provide effective feedback • Documenting written evaluations of learners Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

10. Objectives • By the time you complete this module, you should be able to: • Explain the difference between formative and summative feedback and evaluation. • List the 6 ACGME core competencies. • List at least three characteristics of effective feedback. • Listen to examples of feedback. • Discuss the importance of accurate evaluations. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

11. Agenda • In module 4 we will cover the following: • Definitions of assessment, evaluation and feedback • The 6 ACGME core competencies • Assessment of learners • The R.I.M.E model • Providing feedback to learners • Writing learner evaluations Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

12. Evaluation and Feedback • Imagine yourself as a chef in a top fancy restaurant and you are cooking for a food critic for the national newspaper in your area–a very important person. • As you are preparing your dish, you periodically assess the taste of your soup and make changes by adding more ingredients based on your assessment. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

13. Evaluation and Feedback • You also ask a few colleagues in the kitchen to assess your soup and you make adjustments based on their objective opinion. • What has just happened? The first important step here is to gain an understanding of how the soup is progressing as compared to your standard soup – this is assessment. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

14. Evaluation and Feedback • The second important step is seeking an assessment from others in order to improve the soup – this is asking for feedback. • After hearing the comments of other chefs, you contemplate for a while if and how to make your adjustments – this is reflection. • Now your soup is ready to be served and judged by the food critic. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

15. Evaluation and Feedback • You then observe the critic eating your soup. You try to make assumptions on how they liked it based on their facial expressions as they tasted the soup. If you ask their opinion you are seeking feedback. • When they assess your soup based on all the soups they have tasted and you await their final judgment in tomorrow’s newspaper - this is their written evaluation. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

16. Evaluation and Feedback • This process of assessment, providing feedback, reflection and evaluation are the key ingredients to being an effective teacher. • Feedback should be something sought after and used to enhance learner knowledge, attitudes and skills. It should be provided during and after every rotation. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

17. Definitions • Assessment: The comparison of two things; usually the item or behavior being assessed compared to certain standards or criteria. • Ex: When you assess your learner you are observing their behaviors, attitudes, knowledge and skills and comparing them to the “standard” medical student. These characteristics must be directly observed before you can pass judgment. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

18. Definitions • Evaluation: The process of making qualitative judgments or value statements about a person’s work or performance in comparison to a standard or expectations. • Ex: When you deem a student “excellent” you are making a final judgment on their ability (knowledge, attitudes, behaviors and skills) as having exceeded the standard. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

19. Definitions • Feedback: A verbal summary of a learner’s performance that is based directly on an observation of the learner. It is given in an attempt to improve behavior. • Ex: When you tell the learner they are performing above average because their knowledge on most common medical conditions are above the expected levels of knowledge for the average learner. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

20. Definitions • What we evaluate in learners: • Knowledge • Attitudes • Skills • Behaviors • Ex: Basic knowledge of common medical conditions, empathy, appreciation for patients, ability to interpret laboratories, ability to perform accurate physicals or basic procedures and their daily professional behaviors. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

21. Evaluation and Feedback • There are two (2) basic purposes of assessment, evaluation and feedback: 1) To assess learners. 2) To provide a means for changing behavior to prevent repeated errors and reinforce good behaviors-verbally or written. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

22. Evaluation and Feedback • There are two (2) major types of evaluation and feedback: • Formative • Summative • What are the four (4) major foci of evaluation and feedback? Click here for the answer. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

23. Evaluation and Feedback • There are four (4) major foci of evaluation and feedback: • Knowledge • Attitude • Skill • Behavior/performance - professionalism Knowledge Attitude Behaviors Skill Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

24. Definitions • Formative – means during the teaching encounter or “on the spot.” • Summative – means at the end of a teaching rotation or training period. • Both evaluation and feedback can be either formative or summative. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

25. Question • Explain the difference between feedback and evaluation. (Click here for the answer) Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

26. Answer • Evaluation: The process of making qualitative judgments or value statements about a person’s work or performance based on standards or expectations. • Making a judgment on how learners do • Feedback: A verbal summary of a learner’s performance that is based directly on an observation of their performance. It is given in an attempt to improve behavior. • Telling learners what they did well • Telling learners how to improve • Telling learners want not to do in the future Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

27. Evaluation and Feedback • First let’s focus on evaluation and feedback during day to day teaching activities – or formative evaluation and feedback. • We will then return to summative evaluation and feedback latter in the module. Great! I need some feedback on my suturing skills. Let’s discuss how that last procedure just went. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

28. Evaluation and Feedback • Recall the purpose of learner objectives. In module 2 you learned that students have learning objectives for each rotation. These are the skills, knowledge, attitudes and behaviors that they should acquire by the end of each rotation. Thus, the expected outcomes. • Outcomes are the end result. Let’s now introduce the concept of competencies. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

29. Competencies • Competencies are the outcomes and measures the student should achieve. We use competencies to assess how well a student has done – how competent they are for a given area of knowledge, skill, attitude or behavior. • Both residents and students have competencies based on the ACGME topics. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

30. Competencies • The 6 ACGME core competencies are: • Professionalism • Medical knowledge • Patient care • Systems-based practice • Interpersonal communication • Practice based learning and improvement • Leadership was added as a 7th core competency graduation goal for students at the Baylor College of Medicine. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

31. Competencies • The student objectives and core competency graduation goals can be found on the RATL resource page. • We suggest you familiarize yourself with the competencies because these are the standards that your teachers are using to assess your performance and how the school assesses their graduating students. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

32. Competencies • Here is a brief glimpse into the ACGME core competencies document. You may also print them from the RATL resource page or click below to go to the ACGME web page for a shortened version. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

33. Competencies • As a reminder, the 6 core competencies are: • Professionalism (PRO) • Medical knowledge (MK) • Patient care (PC) • Systems-based practice (SBP) • Interpersonal communication (ICS) • Practice based learning and improvement (PBLI) • Use these abbreviations for the following matching game. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

34. Question • Match the examples of ACGME competencies with the correct core competency area: PC, PRO, SBP, PBLI,ICS, MK Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

35. Answers • How did you do? The answers are below. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

36. Assessment • First let’s focus on assessing learners. Keep in mind, you can also use this process for patient performance or adherence to lifestyle modifications. • As the teacher, you will have to assess how your student or patient is doing and then give them the feedback to help them improve. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

37. Assessment • Assessing students requires a basic understanding and familiarity with their course learning objectives and the descriptions of competencies. • Competency goals may be set by national or sub-specialty organizations. • Hemoglobin A1C levels less than 6.5 mg/dl • 15 IV placements • 24 endotracheal intubations • Duty hours must be limited to 80 hours per week, averaged over a four-week period, inclusive of all in-house call activities. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

38. Assessment • You can also use your student evaluation forms to compare your student’s performance with the given scale – helping to determine those with a high level of performance and those with a low level of performance. • For example, if the objective is to accurately measure the blood pressure on several different occasions, then you must observe the student taking the blood pressure. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

39. Assessment • Then you must compare their results to your standard measurements or an experts’ measurements. • How well they measured the blood pressure compared to the standard is their assessment. Are they competent in this skill given the standards or expectations for their level? If no, then you made a judgment or evaluation. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

40. Not competent = Needs Feedback Assessment • If you have directly observed your student’s performance or knowledge and they need improvement, then you must provide them with the feedback to guide them toward improvement and future success. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

41. Assessment • In our example of the chef, they sought out other’s opinions in order to improve the soup. They must have felt comfortable seeking those opinions and they must have valued those individuals they asked for opinions. • As the teacher, you must also make learners feel comfortable asking for your assessment and feedback and you must provide information that is valued. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

42. R.I.M.E. • RIME is a model of assessment of learners in medicine created by Louis Pangaro, M.D.. This model allows residents to assess student learners against a “standard” set of behaviors that is easy to remember. RIME is a mnemonic that stands for: Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

43. Reporter • This learner… • Performs a history & physical efficiently • Gathers & communicates data • Recognizes normal vs abnormal • Identifies and labels new problems • Has a sense of responsibility Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

44. Interpreter • This learner is an… • Effective presenter of information • Can start to assimilate data and study results • Can justify and reason on basic level • Can interpret literature for basic cases Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

45. Manager • This learner can… • Deciding when action needs to be taken • Propose & select among at least 3 diagnostic & therapeutic options • Takes into account particular patient’s circumstances & preferences • Requires more confidence, knowledge & judgment Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

46. Educator • This learner… • Has mastery of the basics • Can identify knowledge gaps and develops a plan for addressing these gaps • Shares new knowledge with others • Defines important questions to research • Takes control • Has drive, insight, skill, • maturity and confidence Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

47. R.I.M.E. • Can you think of student behaviors for each level of the RIME model? • This is an easy way of thinking about the beginner, intermediate and advanced level student. Now let’s think about how we would let the student know what level they have reached. It’s time for feedback! Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

48. Feedback • How you provide feedback can make a significant difference in how your learner receives it and responds to it. Remember, the goal is to improve their performance compared to the gold-standard. • To emphasize how feedback can affect a student, patient or other learner, try playing the feedback game on the next slide. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

49. Feedback • The feedback game is designed to help you better understand feedback and some important points about providing feedback. • Click here to play. • Click here to skip the feedback game. Feedback Game 1 Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

50. Feedback Game: Version 1 • Guess a number between 1 and 10. Use your mouse to click on your guess. You have only three (3) chances to guess the correct number. Then click below to give up. • I give up. Proceed with module. 1 2 3 4 5 6 7 8 9 10 Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.