Module 4: Evaluation and Feedback

1. Module 4:Evaluation and Feedback Residents as Teachers & Leaders Module Created by: Charlene M. Dewey, M.D., M.Ed., FACP Associate Professor of Medicine Teri L. Turner, M.D., M.Ed., MPH Associate Professor of Medicine & Joan Friedland, M.D., MPH Associate Professor of Medicine Web Page and Module Formatting by: Maria Victoria Tejada-Simon, Ph.D. Assistant Professor of Molecular Physiology and Biophysics

2. Welcome • This is Module 4: Evaluation and Feedback. You should have completed a pretest for this module. Did you complete the “honesty pledge” question? • In that pledge you agreed to take the pretest first, then read the module and then take the post-test after reading the module. If you did not complete the pretest, please exit the module now and complete it; then return to the module. Your honesty is appreciated. Click here to continue. Exit Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

3. Welcome • By now you are familiar with the general format of the RATL modules. This module also consists of a pre-assessment, content of the topic and a post-assessment tool. • There are 136 total slides; many are games and interactive activities. Overall it should take about 45-60 minutes to complete this module. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

4. Welcome • However, at the completion of Module 4 we will ask you to complete the Teacher Identity Scale for the second time. • Please remember to provide feedback on the module content and format by completing a module 4 evaluation form found on the main RATL web page. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

5. Introduction • So what are you going to say about these teaching modules? First, you need to compare them to something that is known to be a good example or a gold standard. Then you must decide if they were as good as the gold standard and why. Then you will have to organize your thoughts to provide us with feedback that can help us improve the modules for the future. Lastly, you must convey those thoughts by writing an evaluation. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

6. Introduction • If you do all that and then never send us your comments, we will continue to think these modules are helpful in teaching residents how to teach. If they are not, and you don’t let us know, then this is an educational tragedy! • You do not want to spend a week, a month or a semester teaching learners and never let them know if they made the grade. That would truly be tragic! Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

7. Introduction • What we are talking about is assessments, feedback and evaluations. All are key concepts in teaching and advancing a learners’ performance. It is part of the teacher identity and done correctly will elevate learners to new levels of knowledge, skill and attitudes. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

8. Introduction • Module 4: Evaluation and Feedback focuses on accurately assessing a learner's level of ability, providing effective feedback and writing appropriate evaluations. • Your role as a teacher will be enhanced by adapting appropriate assessment, evaluation and feedback principles. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

9. Goals • Our goals for Module 4 are that participants will review: • General concepts of learner assessment • ACGME core competencies • How to provide effective feedback • Documenting written evaluations of learners Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

10. Objectives • By the time you complete this module, you should be able to: • Explain the difference between formative and summative feedback and evaluation. • List the 6 ACGME core competencies. • List at least three characteristics of effective feedback. • Listen to examples of feedback. • Discuss the importance of accurate evaluations. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

11. Agenda • In module 4 we will cover the following: • Definitions of evaluation and feedback • The ACGME 6 core competencies • Assessment of learners • Providing feedback to learners • Writing learner evaluations Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

12. Evaluation and Feedback • Imagine yourself as a chef in a top fancy restaurant and you are cooking for a food critic for the national newspaper in your area–a very important person. • As you are preparing your dish, you periodically assess the taste of your soup and make changes by adding more ingredients based on your assessment. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

13. Evaluation and Feedback • You also ask a few colleagues in the kitchen to assess your soup and you make adjustments based on their objective opinion. • What has just happened? The first important step here is to gain an understanding of how the soup is progressing as compared to your standard soup – this is assessment. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

14. Evaluation and Feedback • The second important step is seeking an assessment from others in order to improve the soup – this is asking for feedback. • After hearing the comments of other chefs, you contemplate for a while if and how to make your adjustments – this is reflection. • Now your soup is ready to be served and judged by the food critic. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

15. Evaluation and Feedback • You then observe the critic eating your soup. You try to make assumptions on how they liked it based on their facial expressions as they tasted the soup. If you ask their opinion you are seeking feedback. • When they assess your soup based on all the soups they have tasted and you await their final judgment in tomorrow’s newspaper - this is their written evaluation. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

16. Evaluation and Feedback • This process of assessment, providing feedback, reflection and evaluation are the key ingredients to evaluating and providing feedback to learners. • Feedback should be something sought after and used to enhance learner knowledge, attitudes and skills. It should be provided during and after every rotation. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

17. Evaluation and Feedback • Evaluation: The process of making qualitative judgments or value statements about a person’s work or performance in comparison to a standard or expectations. • Feedback: A verbal summary of a learner’s performance that is based directly on an observation of the learner. It is given in an attempt to improve behavior. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

18. Evaluation and Feedback • There are two (2) basic purposes of evaluation and feedback: 1) To assess learners. 2) To provide a means for changing behavior to prevent repeated errors and reinforce good behaviors. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

19. Evaluation and Feedback • There are two (2) major types of evaluation and feedback: • Formative • Summative • There are four (4) major foci of evaluation and feedback: • Knowledge • Attitude • Skill • Overall behavior/performance - professionalism Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

20. Evaluation and Feedback • Formative – means during the teaching encounter or “on the spot.” • Summative – means at the end of a teaching rotation or training period. • Both evaluation and feedback can be either formative or summative. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

21. Question • Explain the difference between feedback and evaluation. (Click here for the answer) Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

22. Answer • Evaluation: The process of making qualitative judgments or value statements about a person’s work or performance based on standards or expectations. • Making a judgment on how learners do • Feedback: A verbal summary of a learner’s performance that is based directly on an observation of their performance. It is given in an attempt to improve behavior. • Telling learners what they did well • Telling learners how to improve • Telling learners want not to do in the future Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

23. Evaluation and Feedback • First let’s focus on evaluation and feedback during day to day teaching activities – or formative evaluation and feedback. • We will then return to summative evaluation and feedback latter in the module. Great! I need some feedback on my suturing skills. Let’s discuss how that last procedure just went. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

24. Evaluation and Feedback • Recall the purpose of learner objectives. In module 2 you learned that students have learning objectives for each rotation. These are the skills, knowledge and attitudes that they should acquire by the end of each rotation. Thus, the expected outcomes. • Outcomes are the end result. Let’s now introduce the concept of competencies. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

25. Competencies • Competencies are the outcomes and measures the student should achieve. We use competencies to assess how well a student has done – how competent they are for a given area of knowledge, skill or attitude. • Both residents and students have competencies based on the ACGME topics. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

26. Competencies • The 6 ACGME core competencies are: • Professionalism • Medical knowledge • Patient care • Systems-based practice • Interpersonal communication • Practice based learning and improvement • Leadership was added as a 7th core competency graduation goal for students at the Baylor College of Medicine. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

27. Competencies • The student objectives and core competency graduation goals can be found on the RATL resource page. • We suggest you familiarize yourself with the competencies because these are the standards that your teachers are using to assess your performance and how the school assesses their graduating students. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

28. Competencies • Here is a brief glimpse into the ACGME core competencies document. You may also print them from the RATL resource page or click below to go to the ACGME web page for a shortened version. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

29. Competencies • As a reminder, the 6 core competencies are: • Professionalism (PRO) • Medical knowledge (MK) • Patient care (PC) • Systems-based practice (SBP) • Interpersonal communication (ICS) • Practice based learning and improvement (PBLI) • Use these abbreviations for the following matching game. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

30. Question • Match the examples of ACGME competencies with the correct core competency area: PC, PRO, SBP, PBLI,ICS, MK Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

31. Answers • How did you do? The answers are below. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

32. Assessment • First let’s focus on assessing learners. Keep in mind, you can also use this process for patient performance or adherence to lifestyle modifications. • As the teacher, you will have to assess how your student or patient is doing and then give them the feedback to help them improve. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

33. Assessment • Assessing students requires a basic understanding and familiarity with their course learning objectives and the descriptions of competencies. • Competency goals may be set by national or sub-specialty organizations. • Hemoglobin A1C levels less than 6.5 mg/dl • 15 IV placements • 24 endotracheal intubations • Duty hours must be limited to 80 hours per week, averaged over a four-week period, inclusive of all in-house call activities. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

34. Assessment • You can also use your student evaluation forms to compare your student’s performance with the given scale – helping to determine those with a high level of performance and those with a low level of performance. • For example, if the objective is to accurately measure the blood pressure on several different occasions, then you must observe the student taking the blood pressure. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

35. Assessment • Then you must compare their results to your standard measurements or an experts’ measurements. • How well they measured the blood pressure compared to the standard is their assessment or evaluation. Are they competent in this skill given the standards or expectations for their level? Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

36. Not competent = Needs Feedback Assessment • If you have directly observed your student’s performance or knowledge and they need improvement, then you must provide them with the feedback to guide them toward improvement and future success. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

37. Assessment • In our example of the chef, they sought out other opinions in order to improve the soup. They must have felt comfortable seeking those opinions and they must have valued those individuals they asked for opinions. • As the teacher, you must also make learners feel comfortable asking for your assessment and feedback and you must provide information that is valued. So let’s focus on providing feedback. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

38. Feedback • How you provide feedback can make a significant difference in how your learner receives it and responds to it. Remember, the goal is to improve their performance compared to the gold-standard. • To emphasize how feedback can affect a student, patient or other learner, try playing the feedback game on the next slide. Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

39. Feedback • The feedback game is designed to help you better understand feedback and some important points about providing feedback. • Click here to play. • Click here to skip the feedback game. Feedback Game 1 Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

40. Feedback Game: Version 1 • Guess a number between 1 and 10. Use your mouse to click on your guess. You have only three (3) chances to guess the correct number. Then click below to give up. • I give up. Proceed with module. 1 2 3 4 5 6 7 8 9 10 Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

41. Feedback Game: Version 1 • You selected choice 1. You are totally wrong! • Back to game Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

42. Feedback Game: Version 1 • You selected choice 2. Not! Try again! Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

43. Feedback Game: Version 1 • You selected choice 3. Next time you try, bring your brain! Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

44. Feedback Game: Version 1 • You selected choice 4. You are not even close! Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

45. Feedback Game: Version 1 • You selected choice 5. Can’t you do better than that? Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

46. Feedback Game: Version 1 • You selected choice 6. Here’s a dime to call your mother – maybe she’ll think you are a doctor! Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

47. Feedback Game: Version 1 • You selected choice 7. That was a stupid guess! Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

48. Feedback Game: Version 1 • You selected choice 8. I have worked with babies with more brains than you! Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

49. Feedback Game: Version 1 • You selected choice 9. You are so far off base you are out of the field! Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.

50. Feedback Game: Version 1 • You selected choice 10. How pathetic! Try again! Created by Culberson, Dewey, Ismail, Friedland, Tejada-Simon & Turner. NIH Funded Relationship-Centered Transformation of Curricula, Baylor College of Medicine, 2006.