The ACGME Site Visit: Partners in the GME Community

1. The ACGME Site Visit: Partners in the GME Community ARCOS 16th Annual Meeting March 15, 2019 Lee S. Segal, MD ACGME Accreditation Field Representative

2. The ACGME Site Visit: Partners in the GME Community • I have no financial disclosures • I am employed by the ACGME • Former Program Director, Department of Orthopaedic Surgery and Rehabilitation, Penn State University College of Medicine

3. ACGME Site Visit Program Director and Coordinator ACGME Review Committee

4. Department Of Orthopaedic Surgery ACGME Site Visitor

5. Learning Objectives • Describe the elements and objectives of different types of accreditation site visits. • List key attributes of ongoing readiness for a site visit, and key elements of site visit preparation. • Review the Common Program Requirements effective July 2019 as they relate to the Annual Program Evaluation • Prepare for the 10-Year Accreditation Site Visit

6. Types of Site Visits • Application, Re-Application • Pre-Accreditation, Continued Pre-Accreditation (AOA  ACGME) • Initial Accreditation • Continued Accreditation • 10-Year Accreditation • Data-Prompted • Complaint-Prompted

7. Overall Goal of a Site Visit Obtain complete, accurate, meaningful information about the program to assist the Review Committee (RC) in making an accreditation decision

8. Goals for Site Visits • Verify/clarify information provided by the program, by program's ADS site, and by Resident / Faculty Survey results • Address all relevant elements of the Program Requirements • Conduct educationally helpful, non-adversarial interview sessions • Inform program of RC review process and accreditation decision options • Make program improvement suggestions, not decisions or citations

9. Site Visit Preparation – PROGRAM RESPONSIBILITIES • Knowledge of Program Requirements (Common and Specialty-specific) • Provision of requested documents • Accurate completion of program information in ADS • Meaningful responses to all previous citations • Description of subsequent major changes to improve program • Update of faculty and resident rosters • Construct correctly-formatted block diagram • Review of all material by DIO

10. Site Visit Preparation– SITE VISITOR RESPONSIBILITIES • Site Visit Letter of Instruction • Research additional information in program’s ADS site: e.g., program accreditation history, LONs, other RRC correspondence, ACGME Resident and Faculty Survey results, rosters, case lists, institutional accreditation status, core program status, faculty/resident attrition, program director turnover • Study all program-provided information • Plan interview sessions and discussion items for each

11. Resident / Fellow Interviews Trainees may be interviewed in a single group, in 2 groups (seniors vs. juniors), or individually Trainee interviews occur early in the site visit day, followed by other interviews to confirm the resident perspective

12. Two Sources ofConversation-starters • Resident and Faculty ACGME annual surveys • Confidential resident and faculty “consensus lists” of 5 strengths and 5 areas for improvements are requested before most site visits

13. Resident Survey Review Challenges • Small programs (< 4 )  multi-year aggregated results +/- helpful • Recent program graduates are not interviewed • PGY-1 Residents, fellows in 1-yr programs did not complete the RS available for review • Residents unwillingness to be open with site visitor may require reorganization of resident interviews on day of the site visit • RC thresholds of concern not known and RS items weighted differently • RS considered by some to be more than a screening tool • Results not PGY-stratified

14. Resident Survey Review Challenges • Corrected concern carry-over • Use as a retaliation tool • Expression of program, department, institutional unrest • Items/questions not understood by residents • Compare to national aggregate data, not specialty data • My understanding that RS will be changing

15. Resident Survey Review  How has the program addressed the RS results? • Were RS results discussed with trainees/faculty? • What RS-stimulated improvements were made?

16. Document Review • Focused document review to reduce burden and increase meaning • Some documents (e.g., evaluations) will be reviewed in resident files or online

17. Lauren M. Byrne, Rebecca S. Miller, and Thomas J. Nasca (2016) Implementing the Next Accreditation System: Results of the 2014–2015 Annual Data Review. Journal of Graduate Medical Education 2016; 8(1): 118-123. Internal Drivers for Program Improvement Number of Citations

18. Next Accreditation System (2013)Nasca TJ, et al: NEJM 2012: 366: 1051-1056 • Former accreditation process prescriptive, not promote innovation • Programs were meeting, not raising educational standards • Transition from episodic SV (4-5 years) to annual data collection and 10 year SV, with a self-study done before scheduled 10 year SV. • Continuous process of quality improvement, educational outcomes based.

19. 10 year Site Visit / APE Citations - Internal Drivers for Program Improvement ACGME – External Driver for Program Improvement Self-Study complements the Annual Program Evaluation

20. New Common Program RequirementsJuly 1, 2019 IV.A.: The curriculum must contain the following educational components - a set of program aims consistent with the Sponsoring Institution’s mission, the needs of the community it serves, and the desired distinctive capabilities of its graduates V.C.1.b.: The PEC’s responsibilities must include a review of current operating environment to identify strengths, challenges, opportunities, and threats as related to the program’s mission and aims – SWOT analysis V.C.2.: The program must complete a Self-Study prior to its 10- Year Accreditation Site Visit

21. Program Aims • What does the program want their program graduates to look like? • Align with the community of patients for which the program provides care • Aims can change over time • Intentionality – “mission statement”

22. Context / SWOT Analysis/APE • Current operating environment (context) • Strengths, challenges, opportunities, and threats • Devise strategic plan - interventions • Area(s) for improvement • Advance strength, eliminate weakness, take advantage of an opportunity, mitigate a threat

23. Annual Program Evaluation Interventions address: • Program requirements • What will take us to the “next level” • Long-term projects • Use PDSA approach (Plan, Do, Study, Act) • What is program trying to change, what interventions, how to measure outcomes • Improvement model – onus on each program to raise educational standard.

24. APE template

25. Self Study Notification of Self-Study, Submit SS Summary, Self-Study Visit usually two years later … Intentionality: Where have we been? Where are we now? Where are we going?

26. Self Study • Have 5+ years of APE’s to incorporate • Self Study Group ≠ PEC (increase stakeholders, residents) • Strategic plan (Aims, SWOT, Interventions/strategies, action plans) • Opportunity to set direction and priorities, communicate mission/aims, alignment of program. Simplify decision-making.

27. The 10-Year Site Visit A full site visit with review of all applicable program requirements Submit Summary of Achievements prior to scheduled 10-year SV 2-4 areas where improvements have been achieved Review both the Self-Study Summary and the program improvements described in the Summary of Achievements

28. Site visitor provides feedback to program leadership • Aspects of the Self-Study may be inserted into Site Visitor’s Report (SVR) • Site visitor provides feedback to program leadership • How well your Aims fit your program • Achievement areas link to aims and operating environment (SWOT) • Use of data to measure outcomes • Stakeholder engagement

29. ACGME Review Committee • Reviews to SVR, ADS documents, Surveys, SSS, Summary of Achievement • Makes accreditation status decision • May commend the program • Issues any citations • May request progress report • May note areas for concern / improvement • Sends Letter of Notification (LON)

30. Questions?

31. Thank You