managing the acgme core competencies

1. Managing theACGME Core Competencies

3. The ACGME Outcome Project

4. Underlying Principles

9. CREOGCompetency Task Force

10. CREOGCompetency Task Force

11. Best Guide to Determine Expectations

12. What do the RRCs Want?

13. Model Assessment Summary

14. Model Assessment Summary

15. Global Assessment Ratings

18. Patient Care General  Remember: We are good educators and have been doing a good job. We need to develop a �tool box� that is program friendly, resident friendly, and teacher friendly. We need to help programs know where their starting point is located.

19. Patient Care Competency Residents must be able to provide patient care that is compassionate, appropriate, and effective for the treatment of health problems and the promotion of health.

20. Patient Care Residents are expected to: communicate effectively and demonstrate caring and respectful behaviors � gather essential and accurate information � make informed decisions about diagnostic and therapeutic interventions� develop and carry out patient management plans counsel and educate patients and their families use information technology in patient care and patient education perform competently all medical and invasive procedures considered essential for the area of practice provide health care services aimed at preventing health problems or maintaining health work with health care professionals, including those from other disciplines, to provide patient-focused care

21. Tracking Scope of Clinical /Educational Experience General: �Global Assessment� Tools (Overall summaries) Semi-annual Summaries Annual Assessment Exit Interviews Specific: Focused Assessment On-site Evaluations Rotation Evaluations Semi-annual Summaries Annual Assessment Exit Interviews

23. How do we teach surgical skills? Imagery Observation video assisting in the operating room Mental Practice Anatomy review Pelvic models, dog labs Student dependent !!!!

24. How do we teach surgical skills? Systematic Review of Performance Teacher dependent!!!!!

25. Systematic Review  of Performance FEEDBACK, FEEDBACK, FEEDBACK In the OR Real time Constructive Competency based evaluation

26. Technical Skills AssessmentCompetency Based Evaluation Feasibility Reliability Validity Predictive of future performance Content domain Concurrent Construct Face � resembles real life

27. Teaching and Testing Technical SkillsReznick RK, AJS 1993;165:358

28. Teaching Surgical SkillsOSATS (objective structured assessment of technical skills) Reznick RK � Univ. of Toronto 1993- present Validated global and procedure specific tools Used models with reliable results Exported evaluation to other programs with reliable success Transferred evaluation system to o.r. and found positive correlation between time and quality of surgery between resident levels

29. Teaching Surgical SkillsOSATS Barbara Goff � University of WA 2000 - present Validated global and procedure specific tools Used models with reliable results Instructors can evaluate their own residents reliably Exported evaluation to other programs with reliable success

31. Making your own competency evaluation tools Decide on skill/procedure Surgery, Amniocentisis, Biophysical profile Look at textbooks, operative reports Department/Expert Consensus Use animal models, dry lab, or actual procedures for evaluation Evaluate validity and reliability

32. Medical Knowledge Residents must demonstrate knowledge about established and evolving biomedical, clinical, and cognate (e.g. epidemiological and social-behavioral) sciences and the application of this knowledge to patient care. Residents are expected to: demonstrate an investigatory and analytic thinking approach to clinical situations know and apply the basic and clinically supportive sciences which are appropriate to their discipline

33. What we already do well CITROG Examination Annual assessment Large number of items (315) Broad base of information tested High reliability (r=0.91)

34. What we can (easily) do better Chart Stimulated Recall Examination Format familiar from ABOG oral exam Based on examinee�s case list PGY4�s currently collecting information Breadth of knowledge ensured using list of Core Knowledge Areas Use Focused Resident Competency Assessment Form

35. Global Ratings New form Combines both ordinal categories & free form comments Good data supporting reliability of global rating forms for evaluating Medical Knowledge

36. More Challenging Assessments Oral examinations Format familiar from standardized oral exam portion of ABOG exam Hypothetical patient case scenario presented Time intensive Development Training to ensure inter-rater reliability Administration Useful for familiarizing residents with ABOG exam format Based on CREOG learning objectives or list of Core Knowledge Areas

37. Other Assessment Tools OSCE Time intensive Costly 360o Evaluation Validity of assessment Peers Subordinates Other members of the health care team

38. Good News! We are already assessing Medical Knowledge well

39. ACGME Competency Defined

40. Current Assessment Methods

41. ACGME/RRC Think Tank

42. Focused Assessment of Communication Skills

44. C-FAC Implementation

45. Patient and/or Professional Associate Assessment

47. ACGME Definition Residents must demonstrate a commitment to carrying out professional responsibilities, adherence to ethical principles and sensitivity to a diverse patient population. They are expected to: Demonstrate respect, compassion, and integrity: a responsiveness to the needs of patients and society that supercedes self interest: accountability to patients, society and the profession: and a commitment to excellence and ongoing professional development. Demonstrate a commitment to ethical principles pertaining to provision or withholding of clinical care, confidentiality of patient information, informed consent and business practices Demonstrate sensitivity and responsiveness to patient�s culture, age gender and disabilities.

48. Defining Professionalism Perhaps the biggest challenge to meeting this competency Developing a definition or �Code of Conduct� Defining behaviors that match Resources- ACGME, ABIM, CREOG, ACOG

49. Match Behavior to Definition Respect Treats others with respect does not demean, insult or make others feel inferior Uses respectful language when discussing patients Excellence Conscientious effort to exceed ordinary expectations, commitment to life long learning

50. Education *Role-modeling *Environment Discussion of Expectations Critical incident * Case discussion Didactics Readings Role play

51. Evaluation 360 degree evaluation Separate from summative evaluation remove halo effect Discussion of professionalism at formal teaching committee meetings more likely to elucidate professionalism issues Critical incident form/process

52. 360 degree evaluation Evaluate from many different perspectives more effective and instructive Peers Students Patients Nursing/allied health personnel Self Attending

54. Initiating Program Define professionalism/Code of Conduct for your residency Define desired behaviors Develop evaluation form-360� Assess environment Faculty development Component of intern orientation Develop curricula Repeating theme in didactic sessions

55. Culture CREOG objectives significance of conception, birth, marriage, death provide competent care for diverse cultural background Allied health, nursing excellent curricular resources Education: Reading, didactics, portfolio, sessions with interpreters and patients Evaluation- 360�, simulated patients , case discussion

56. Ethics History of education in this area Curricular guidelines in CREOG and ACOG-Ethics in Obstetrics and Gynecology Lecture/case studies/component of case presentations Many sessions necessary to impact care Evaluation-?360, portfolio, simulated cases

57. Conclusion Exemplary professional behavior is and has been an expectation of all physicians We have and have had the responsibility to foster, enhance and evaluate these behaviors Currently it is done informally We need to formalize and document the process

58. Practice-Based Learning and Improvement Practitioners Needs are Changing due to the Information Age Patients have access to unfiltered medical information Up-to-date medical knowledge deteriorates over time if left alone This competency aims to teach skills that residents will need to practice life-long learning

59. Residents are expected to: Analyze their own practice Efficiently locate and evaluate applicable literature Use information technology to assist in locating information Apply knowledge of study design and statistical methods to critically appraise literature Facilitate learning of students and health care staff Clinicians want to provide high quality care Based on high-quality evidence Up-to-dateClinicians want to provide high quality care Based on high-quality evidence Up-to-date

60. Evaluation Tools 1. Global Assessment Form 2. Resident Portfolio: Journal Club Evaluation Forms Resident Research Evaluation Form Clinical Topic Review Log M&M Interesting Clinical Topic

61. Practice-Based Learning and Improvement

62. Practice-Based Learning and Improvement Journal Club Evaluation Everyone uses some elements of this competency in searching and reading the literature Making this activity a natural venue for evaluation

63. Practice-Based Learning and Improvement

64. Journal Club Evaluation Forms

65. Practice-Based Learning and Improvement Journal Club Evaluation Forms The general format for all forms leads the resident through these questions: Are the results of the study valid? What are the results? Are the results likely to apply to my patient or practice?

67. Journal Club Evaluation Forms

68. Research Evaluation Form

69. Clinical Topic Review Log

70. System-Based Practice Residents must demonstrate an awareness of and responsiveness to the larger context and system of health care and the ability to effectively call on system resources to provide care that is of optimal value. Residents are expected to: understand how their patient care and other professional practices affect other health care professionals, the health care organization, and the larger society and how these elements of the system affect their own practice know how types of medical practice and delivery systems differ from one another, including methods of controlling health care costs and allocating resources practice cost-effective health care and resource allocation that does not compromise quality of care advocate for quality patient care and assist patients in dealing with system complexities know how to partner with health care managers and health care providers to assess, coordinate, and improve health care and know how these activities can affect system performance

71. �Residents live in the fault lines of health care systems and give voice to what life is like there.� Paul Batalden. MD

72. �The prominence of physicians in highly interdependent medical systems confers tremendous power on them, individually and as a profession. With this power comes an ethical responsibility to be deeply concerned about medical systems.� Nolan, Annals Intern Medicine, 1998

73. SUGGESTED CURRICULUM CONTENT FOR SBP Quality assurance /Quality improvement Managed care systems Analysis of medical error utilizing a systems based approach Population based medicine

74. RESIDENT SKILLS FOR SBP Effectively access and utilize resources Understand different types of health care systems and how they interact with physicians, patients, and society Utilize evidence based, cost effective strategies to optimize care for individual patients and for populations of patients Utilize a systems oriented approach to identifying and reducing medical error Function effectively in the health care team Advocate for patients to optimize outcomes

75. MEASUREABLE OUTCOMES FOR SBP SKILLS Attending global assessment Interdisciplinary teamwork Patient satisfaction / Patient advocacy Discharge plans / discharge summaries Benchmark of practice, especially preventive measures Addressing systems failures

76. ATTENDING GLOBAL ASSESSMENT FOR SBP Unsatisfactory: Unable to access/mobilize outside resources; actively resists efforts to improve systems of care; does not use systematic approaches to reduce error and improve patient care Superior: Effectively accesses/utilizes outside resources; effectively uses systematic approaches to reduce errors and improve patient care; enthusiastically assists in developing systems� improvement

77. INTERDISCIPLINARY TEAMWORK 360 Degree Assessments Professional Associate Surveys Peer Evaluations

78. PATIENT SATISFACTION / PATIENT ADVOCACY 360 Evaluations Professional associate survey Patient survey Portfolio assessments Write a letter to an HMO requesting a non-formulary medication for a patient Draw a flow chart mapping a patient�s outpt visit and identify all systems involved Perform a cost analysis of a patient�s inpt bill

79. DISCHARGE PLANS/SUMMARIES Record performance review Screen inpatient charts for completeness of plans (medications prescribed, followup appointments confirmed, home health visits scheduled,etc) Monitor number of incomplete medical records to reflect responsibility to the system for complete and accurate records

80. BENCHMARKS OF PRACTICE Record performance review Screen outpatient clinic charts for documentation of preventive care Mammograms, Pap smears, Immunizations, Colorectal cancer screening, etc. Profile resident clinic practices for utilization of ancillary services, cost effective use of medications Review samples of resident outpt billing & coding

81. BENCHMARKS OF PRACTICE Portfolio assessment Assign resident to identify incidence of a specific health problem in their clinic population, assess cost effectiveness of current screening methods, identify impediments to implementation of screening and suggest ways to improve

82. CAUSE AND EFFECT DIAGRAM �FISHBONE� DIAGRAM

83. ADDRESSING SYSTEMS FAILURES Quality Assessment / Quality Improvement Analysis and reduction of medical error

84. QUALITY ASSESSMENT/IMPROVEMENT Portfolio assessment Assign resident to identify a systems problem in a patient care setting, analyze the systems involved, develop at least two interventions to improve the problem and develop a plan to assess outcome of intervention

85. QA/QI PROJECT **********************************************************************

86. ANALYSIS AND REDUCTION OF MEDICAL ERROR Portfolio assessment Analyze an adverse patient outcome or near miss using a root cause analysis approach Discussion based performance assessment Participate in Near Miss Conference discussion

87. EVALUATION OF QA/QI PROJECT The resident chose a project that could significantly improve patient care / the medical environment. UNSATISFACTORY SATISFACTORY SUPERIOR 1 2 3 4 5 6 7 8 9 The resident identified multiple systems impacting this issue. UNSATISFACTORY SATISFACTORY SUPERIOR 1 2 3 4 5 6 7 8 9 The resident identified a team of people to address the issue. UNSATISFACTORY SATISFACTORY SUPERIOR 1 2 3 4 5 6 7 8 9 The resident worked with the QA team and developed a well constructed plan to implement the necessary changes to achieve the goal. UNSATISFACTORY SATISFACTORY SUPERIOR 1 2 3 4 5 6 7 8 9 The resident identified multiple resources within the healthcare system that would be necessary to achieve the goals of the project. UNSATISFACTORY SATISFACTORY SUPERIOR 1 2 3 4 5 6 7 8 9 The resident identified how the results of the planned changes would be monitored to measure the success of the project. UNSATISFACTORY SATISFACTORY SUPERIOR 1 2 3 4 5 6 7 8 9 Overall assessment of project. UNSATISFACTORY SATISFACTORY SUPERIOR 1 2 3 4 5 6 7 8 9

88. ANALYSIS OF MEDICAL ERROR

89. EVALUATION OF ANALYSIS OF MEDICAL ERROR The resident identified a significant adverse outcome or near miss event for analysis. UNSATISFACTORY SATISFACTORY SUPERIOR 1 2 3 4 5 6 7 8 9 The resident evaluated the event from a systems based approach. UNSATISFACTORY SATISFACTORY SUPERIOR 1 2 3 4 5 6 7 8 9 The resident identified multiple �latent� sources of systems errors that contributed to the adverse event. UNSATISFACTORY SATISFACTORY SUPERIOR 1 2 3 4 5 6 7 8 9 The resident identified several possible changes that could be made to prevent a recurrence of this type of adverse event. UNSATISFACTORY SATISFACTORY SUPERIOR 1 2 3 4 5 6 7 8 9 Overall assessment of analysis UNSATISFACTORY SATISFACTORY SUPERIOR 1 2 3 4 5 6 7 8 9