Nancy Piro, PhD – No conflicts of interest to disclose

1. Nancy Piro, PhD – No conflicts of interest to disclose Kim Walker, PhD - No conflicts of interest to disclose

2. PC004 BR10Voice of the Coordinator: Your experience on what works and doesn’t work with CCC meetings Nancy Piro, PhD Education Specialist/Program Manager Stanford University Medical Center Kim Walker, PhD Instructional Designer School of Medicine, Stanford University

3. Session Overview • Introduction and background on multi-institution survey • Report on survey results from Program Coordinators at multiple institutions on the CCC process and outcomes • Emergent themes on coordinators’ experiences and their roles in the CCC • Quantitative outcomes • Qualitative outcomes • What coordinators tell us was ineffective in the CCC process • Best practices for coordinators supporting a successful CCC • Action planning for implementation of best practices

4. Objectives & Session Focus • This session will focus on understanding and discussing the results from a multi-institution survey on CCC practices from the Program Coordinators’ perspective. • At the conclusion, participants will be able to: • Review their own practices in light of the survey outcomes • Identify best practices that best suit their program or institution needs • Develop and take home an action plan enabled by these best practices to better facilitate the work of their CCCs

5. Background

6. Milestones

7. Clinical Competency Committee How the CCC does its work is decided by the Program Director

8. TEMPLATE

9. Survey Idea – from Inspiration to Implementation

10. Survey – Fun Facts • What: • A 14 Question* Survey Monkey survey - Both quantitative (8) and (6)open ended/word questions(6) was developed to assess CCC experience so far with respect to CCCs • When and To Whom: • Between November 12, 2014 and January 15, 2015, the survey link was distributed to a wide sample of coordinators (both within Stanford and across the country to coordinators in the 15 largest institutions, and to PC networks in OB/GYN, Anesthesia, and Surgery) • Analytic Methods Used – Both Qualitative and Quantitative Analyses were used. • Response: 325 Program Coordinators (and 135 Program Directors) responded to our survey by January 15, 2015 * Questions detailed on the following slide

11. Survey Questions • How many members are on your CCC? • Were you a part of your CCC meeting ? • Please describe your role: • What’s your program? • How much time (on the average) per resident was spent on the milestone assessments? • How many residents or fellows are in your program? • Do you provide the CCC members any evaluation data to review before the meeting?

12. Survey Questions • There was no way we could have had an effective meeting without completing the pre-work. • Has your program taken any of the following actions to better align your training with your milestones? • Who is on your program’s CCC? Please check as many as apply: • What data did you provide to your CCC? Please choose as many as apply. • Please describe any aspects of the milestone evaluation process that you thought were particularly beneficial: • Are there any changes you’ll make before the next CCC meeting? • Lastly, what did we miss? Are there any other areas of concern or suggestions?

13. How to Apply to Your Institution

14. Q1. How many members are on your CCC?

15. Q2. Were you part of your CCC meeting?

16. Q3. PC’s Role in Meeting: Emergent Themes Assessment Data Collection/Distribution (n=113) Minutes / note taker (n=129)

17. Q3. PC’s Role in Meeting: Emergent Themes Administrative title, other (n=41) • “As a resource member, and as administrative support for the ccc.” • “Program Administrator. Prepared documents, discussed purpose of meeting, emphasized importance and positive impact this will have for both faculty and trainees.” • “Servant”

18. Q3. PC’s Role in Meeting: Emergent Themes Active participant/meeting facilitator (n=31) • “manage agenda with chair, provide materials. problem-solving, participate in discussion” • “Data collection & presentation of data, active participant in discussions, note taking” • “Active Participant and documentation” Provide input on trainee evaluation on pertinent milestones (n=25) • “Participate in discussions on the competencies from an admin perspective and personal interactions with allied health.” • “As per the requirements from the ACGME, I am not a full member. I am ex-officio. However, I attend all meetings, take minutes, and I do participate in the discussions, to the extent that I can contribute certain information and perspectives, regarding the fellows, which are helpful and pertinent to the discussion.”

19. Q3. PC’s Role in Meeting: Emergent Themes Passive Participant (n=7) • “I am a supporting member. I do not have a speaking role, but Ido all meeting prep.” • “observer and note taker” • “Listening”

20. Q4. Distribution of Programs Surveyed

21. Q5. How much time (on the average) per resident was spent on the milestone assessments?

22. Q6. How many residents or fellows are in your program?

23. Q7. Do you provide the CCC members any evaluation data to review before the meeting?

24. Q8. There was no way we could have had an effective CCC meeting without completing pre-work.

25. Q9. Has your program taken any of the following actions to better align your training with your milestones? (1 of 3) Provided additional learning experiences related to milestones? Aligned (or linked) current evaluations with the milestones?

26. Q9. Has your program taken any of the following actions to better align your training with your milestones? (2 of 3) Used spreadsheet software to aggregate and place all evaluation material together for the CCC? Modified the curriculum with respect to clinical rotations?

27. Q9. Has your program taken any of the following actions to better align your training with your milestones? (2 of 3) Added didactic sessions to address milestone subcompetencies or elements? Added new trainee evaluation tools?

28. Q10. Who is on your program's CCC? Please check all that apply.

29. Q11. What data did you provide to your CCC? Please choose as many as apply.

30. Q11. Data included in the CCC Review: Overall Emergent Themes • Medical student evaluations • Mock oral results • Narrative evaluation comments • Peer Evaluations • Presentations • Previous evaluation • Procedure logs • Research projects • Self assessment scores • Spider diagrams with comparison data • Continuity clinic attendance • Aggregate milestone evaluation scores • Board scores • Conference Attendance • Continuity clinic chart review • Curriculum Vitae • Didactic attendance • Duty Hour Summary • Duty Hour reporting compliance • Evaluation completion rate • In service exam scores • Learning modules completion • Medical records outstanding OP reports

31. Q12. Beneficial aspects of the milestone evaluation process CCC faculty assignment and pre-work “Each member reviews and ranks each resident prior to the meeting.”

32. Q12. Beneficial aspects of the milestone evaluation process Discussions; collaborative committee assessment “allows for a rich discussion among a variety of faculty who have interacted with an individual fellow…”

33. Q12. Beneficial aspects of the milestone evaluation process Aggregate and Comparative Data

34. Q12. Beneficial aspects of the milestone evaluation process Residency Management Systems (RMS)

35. Q13. Changes You Will Make: Emergent Themes Modifying or developing new evaluation forms “contrived” “nebulous” “uncalibrated” "what are they really getting at here?"

36. Q13. Changes You Will Make: Emergent Themes Streamline meeting process

37. Q13. Changes You Will Make: Emergent Themes No changes (!!) • “The sub committee was our 3rd iteration of the CCC process and seems to be working well for us now.” • “No. We did a trial run in the spring and did our official one in the fall. We seem to have a good system in place.“ • “I feel prepared- CCC meetings thru 2016 are already scheduled… templates are already created… we have a system in place.”

38. Q14. Other thoughts: Emergent themes Approaches to CCC faculty review process • CCC Subcommittees by training level • Multiple Meetings by training level • Faculty reviews of residents across specific milestones

39. Q14. Other thoughts: Emergent themes “Food for thought” for ACGME: • “We believe the program coordinator's should be allowed to be full members of the CCC.” • “THE ACGME should modify the language and recognize that the coordinator plays a critical role in the success or failure of the CCC. Also, I believe rotating the chair every three years would be helpful and have discussed this with our PD at length. Especially when you have a chair of the CCC who is difficult to work with.” • “For the initial meeting to go back and look at all that data and initially place residents took an ENORMOUS about of pre-work on my part. Hopefully since that first data entry is done it will move easier. But I wish the ACGME addressed that and recognized how much effort that would require.”

40. Q14. Other thoughts: Emergent themes Recognizing time-consuming nature of work … and need for support

41. Overall Results Summary - from Coordinators and Directors • Overall Improved Evaluation and Documentation • Semi-annual evaluations improved • Documentation of trainee strengths and weaknesses • Time Consuming • Need leadership support • Administrative Resources • Ample planning time • CCC Membership Size • Share the burden! • Value of diverse perspectives • Faculty Roles • Responsibilities • Pairing faculty with trainees • Assigning faculty to become “milestone” experts

42. Lessons Learned / Best Practices It’s your turn and your voice that counts!!! How might you change your practice in light of these findings?

43. Questions?

44. Contact Information: • Nancy Piro, PhD; GME Program Manager/Education Specialist • npiro@stanford.edu • Kim Walker, PhD; Instructional Designer • kwalker5@stanford.edu • http://med.stanford.edu/gme/GME_Community.html