Pieces of the Puzzle

1. Pieces of the Puzzle Perspectives on the Accreditation Visit……

2. Central Theme • Accreditation and Assessment are critically linked in contemporary accreditation models. • They were parallel themes in recent past but have “collided” to form a new accountability in higher education. Ratcliff JL et al, How Accreditation Influences Assessment, Josey Bass 2001

3. How does accreditation and assessment come together? • Two key questions: • How should learner outcomes be demonstrated in the accreditation process? • How should assessment information be used to show improvement in programs, services, and student/resident learning?

4. Accreditation Purposes • Fostering excellence through the development of criteria and guidelines for assessing effectiveness • Encouraging improvement through ongoing self-study and planning • Ensuring external constituents that a program has clearly defined goals and appropriate objectives, maintains faculty and facilities to attain them, demonstrates it is accomplishing them, and has the prospect for continuing to do so Ratcliff JL et al, How Accreditation Influences Assessment, Josey Bass 2001

5. Accreditation Purposes • Provides advice and counsel to new and established programs in the accrediting process • Ensures that programs receive sufficient support and are free from external influence that may impede their effectiveness and the freedom of inquiry Ratcliff JL et al, How Accreditation Influences Assessment, Josey Bass 2001

6. Program Accreditation • Needs to fulfill two dimensions of program quality: • There should be some evidence that the programs have clear goals and courses of study to attain them • The process should demonstrate that responsibilities associated with the goals are being carried out. • Peers interpret quality in the context of the program’s own aims and activities • Central issue: Program review is conducted as a self study process. • Central assumption: Evaluators from outside cannot effectively examine and judge until those inside have reached a judgment about its quality. Ratcliff JL et al, How Accreditation Influences Assessment, Josey Bass 2001

7. Program Review – Audit Style • Issues identified by “outside party” • Buy-in from program director? • Reactive – • Fear about the inspection • Approach – “cover the basics, get by this one and on to the next” • Not stimulated to think about “points of excellence” – only points of deficiency • Outcome from Inspection: • Fear relieved – “get on to the next one.” • Yardstick: STANDARDS – Basic/Minimal

8. Program Review Assessment Style • Self Study and Peer assessment against target goals: • Minimal standards = musts • Growth goals • Improvement goals • Outcome goals • Buy-in for Program Director • Proactive – ownership of goals • Outcome of Inspection • External validation of internal assessment (agree or not) • How much progress has been made against goals? • Yardstick: Goals achieved or improvement beyond minimal standards – Points of Excellence

9. Practicality? What about the Residents? OPTI Standards Basic Standards For I R Time? Last Inspection? Family? How am I Doing as a PD? Patient Care? Federal and Local Rules Inspector Factor ? Specialty College Standards Productivity? Salary? Malpractice?

10. What am I supposed To do with this? OPTI Standards Basic Standards For I R ??? ??? YES!! What role? Federal and Local Rules Specialty College Standards

11. ? OPTI Standards Do they agree??? Basic Standards For I R Which one takes precedence? ? ? Federal and Local Rules ? Specialty College Standards

12. Experience with the Audit Program Review…. • Attempts to help program directors and DMEs to prepare for upcoming inspections. • Attempts to find “common areas” of support need across OPTI programs of same discipline • Based on BASIC STANDARDS – developed “Crosswalks” to assist in review of individual standards and collective standards.

13. Standards Crosswalk and Worksheet Completed by:_____________ Date:_____ Specialty:______________

14. Program Pre-Inspection Findings Hospital_______________ Completed by:________________

15. Standards often do not: • Consider practical issues • A complete research paper per year of residency (human subject approvals take almost half that time). • Ability of community hospital to afford “out rotations” since the BBA 1997 • Ability of trainer to (again) teach subjects which were dropped from practice in time past. • Consider federal rules or procedures • Institutional Review Board and HIPPA approval for human subject research • Documentation standards

16. Standards often do not: • Reflect the need to include the following in the curriculum and assess effectiveness: • Ethics • Communication skills • Require the assessment of outcomes of study • Clinical skill development or OSCEs • In Service Examination and Analysis (individual and program wide). • Progress exams or oral exams • Require that a program develop an ongoing quality assessment program and benchmarks for process and learning outcomes goals.

17. Program Review 40 “areas of concern” – standards not met in crosswalk Affiliation agreements for major components not completed (for over 10 years) Manual in disarray…never distributed to residents for past three years PD mostly practices at alternate hospitals (where residents not welcome) for >70% time Etc. Inspector “Wonderful program with dedicated PD and hospital” No major deficiencies Recommend increase in number of trainees Commendable PD attention to program Recommend full five year accreditation. Experiences with Accreditation Visits – an OPTI view ..example: Net result:: “OPTI is too tough…. see, We are OK!!”

18. Program Review 36 “areas of concern” on Standards Crosswalk No resident manual in print/no written curriculum Resident work hours = 2 weeks on call continuously 2 yr program required by PD/1 yr by specialty college….few recent trainees complete the two years. Certificates difficult to obtain. Certification rates = <%50 pass rate on first attempt. Inspector exit notes: “Excellent program which should be a model for others” This program should be expanded to train more residents. No major deficiencies Experiences with Accreditation Visits – an OPTI view ..example: Net result:: PD who had been working with OPTI to correct findings has stopped. “We are doing fine….see how well we did on the inspection? Why won’t you sponsor more positions?”

19. Experiences with Accreditation Visits – the “Inspector Factor” • Inspectors • Often not familiar with Basic Standards for Interns and Residents or OPTI standards • Often spend as little as 4-6 hours in the entire process • Do not “dig deep enough” to see issues • Accept inspectors workbook without verifying content. • Inspectors workbooks and standards often do not reflect the same “intent” or priority for the issue in question – produce confusion • Ranking of standards is evident by threat of automatic probation if question answered “NO”. • “Inter-rater reliability” seems low

20. Conclusion – Audit Review • “Dissatisfaction” with end result • Gets through the current “audit” inspector approach • Fails to stimulate long term goals or real improvement • Something is missing • May promote a punitive rather than a supportive atmosphere –

21. “Begin with the end in mind”- What is the goal? • Get from one accreditation visit to the next? • Minimal level thinking • Threshold is the lowest possible point to keep the program afloat (?) • Continuous improvement over time against declared goals. • Determine a base line • Set realistic but challenging goals • Measure progress….set new goals • (meeting minimal thresholds is never a problem)

22. Goal –get from one accreditation visit to the next? • Data driven • Minimalist thinking • “audit mentality” • Promotes bashing and blaming • Sets interested parties at odds • Promotes a pessimistic environment and inhibits creative thinking • Goals are at the minimal threshold – “Basic”

23. Goal – Assessment for CQI and Outcomes • Markers of Excellence • “Internalized goals” born in self study and assessment • “Buy-in” for all stakeholders

24. Goals for the Workshop • Recreate the “Basic Standards” to allow and encourage growth, goal setting and a reach for higher goals. • Revise and recreate basic standards to ensure basic elements are present and to foster agreement between: • External standards/Internal standards • External validation/Internal validation • To foster better understanding of the “fit” between the AOA standards and laws which apply to residency training.

25. Basic Standards For I R OPTI Standards Federal and Local Rules Specialty College Standards Goal