ACGME Competencies: Web-based Evaluation and Documentation

1. ACGME Competencies: Web-based Evaluation and Documentation Charles O. Frazier, M.D. STFM Theme Day - Competencies

2. Objectives • ACGME Toolbox • Using a web-based, database-driven application • Demonstration • Wrap-up STFM Theme Day - Competencies

3. ACGMECompetencies STFM Theme Day - Competencies

4. ACGME Competencies • Clinical effectiveness? (clinical outcomes) • Competence (knows how) vs. performance (shows how) STFM Theme Day - Competencies

5. Chart stimulated recall oral exam Checklist evaluation OSCE Patient surveys Simulations and models Standardized oral exam Standardized patients 360 ° Evaluation Instrument Global rating of recorded performance Procedure/Case logs Portfolios Record Review Written exam ACGME “Toolbox” STFM Theme Day - Competencies

6. ACGME “Toolbox” • What did I see when I saw the toolbox? Reams of nearly useless paper… STFM Theme Day - Competencies

7. ACGME “Toolbox” • 360° evaluation instrument • “Implementing an electronic system should make the 360-degree-evaluation feasible” • Procedure, case logs • “Electronic recording devices and systems facilitate the collection and summarization of patient cases or procedures performed” STFM Theme Day - Competencies

8. ACGME “Toolbox” • Patient Surveys • “It may be possible to improve feasibility by … using computers to collect and summarize survey data” • Simulations and models • “computer-automated scoring rules to record the examinees’ actions” STFM Theme Day - Competencies

9. Riverside Family Practice Residency STFM Theme Day - Competencies

10. Advising / Evaluation System • Faculty Advisor / Educational Partner • Quarterly advisor meetings with resulting Educational Prescriptions • Semiannual director meetings • Evaluation forms reflect competencies; criteria clear and widely distributed • Expectations defined for residents and faculty • Comp-E-Val™ pulls it all together STFM Theme Day - Competencies

11. Web Based Competency Evaluation Comp-E-Val™ STFM Theme Day - Competencies

12. Introduction • Why database-driven? • Why web-based? • Our methods • Brief technical description • Demonstration STFM Theme Day - Competencies

13. Why database-driven? • To get rid of a lot of the paper • To allow collation, computation, and comparison STFM Theme Day - Competencies

14. Why web-based? • Universal access • Real-time data acquisition • To facilitate PDA input • To get rid of a lot of the paper STFM Theme Day - Competencies

15. Methods • Aligning competencies / tools • Dissemination among faculty, residents, and staff • Choosing when data collected • Data analysis and review STFM Theme Day - Competencies

16. STFM Theme Day - Competencies

17. X X X X X X X X

18. Methods • Aligning competencies / tools • Dissemination among faculty, residents, and staff • Choosing when data collected • Data analysis and review STFM Theme Day - Competencies

19. Schedule STFM Theme Day - Competencies

20. STFM Theme Day - Competencies

21. Methods • Aligning competencies / tools • Dissemination among faculty, residents, and staff • Choosing when data collected • Data analysis and review STFM Theme Day - Competencies

22. Technical description - Intranet • Compaq Proliant Server • Microsoft Windows NT 4.0 • Microsoft Internet Information Services 5.0 • Microsoft FrontPage 2000 • Active Server Pages • Microsoft Internet Explorer STFM Theme Day - Competencies

23. Active Server Pages • Script (programming) is executed on the server, and resulting HTML is sent back to and displayed on browser • Connects to database • Underlying script secure STFM Theme Day - Competencies

24. Technical Description - Intranet • Security • NT Server Authentication • Cost • Server • Software • Labor (estimate) 450 hours STFM Theme Day - Competencies

25. Technical Description - Internet • VCU IIS Public Information Server - June 2002 • SSL encryption security with ASP database passwords • No direct cost to residency • Domain name registration STFM Theme Day - Competencies

26. Demonstration STFM Theme Day - Competencies

27. Current Status • 40 residencies all across the USA • Not all are using yet • 1 neurology; 1 radiology • Full toolbox, with custom tools available • Conference and procedure surveillance; faculty evaluation • ERAS Import • No scheduling STFM Theme Day - Competencies

28. All 6 VCU FP programs Riverside Transitional UCSD FMR WVU FMR Loma Linda U. FPR Indiana U. FPR Carilion FPR Fort Wayne Med Ed Prg East Jefferson GH FPR Mountainside FPR Med. Coll. Ohio FPR Crozer-Keystone FPR Residencies STFM Theme Day - Competencies

29. Comp-E-Val™ • Acquired by the American Board of Family Practice • Combine with resident tracking program • Pilots early 2004 • Production 2005 • Free to all FP residencies STFM Theme Day - Competencies

30. Comp-E-Val™ • https://www.pubinfo.vcu.edu/webeval Examalyzer • https://www.pubinfo.vcu.edu/webeval/examalyzer Riverside Family Practice Residency • http://www.riverside.vcu.edu • 757-594-3878 • (FAX) 757-591-9021 • charles.frazier@rivhs.com STFM Theme Day - Competencies

31. The future • Ability to study The Outcomes Project • Evaluate Program Curriculum • Ability to describe expected “Natural History” or “Growth Curve” for resident • Add clinical effectiveness STFM Theme Day - Competencies

32. Summary • ACGME outcomes-based education/evaluation may improve our educational outcomes • A web-based, database-driven, competency-based evaluation system can facilitate those efforts, and may aid evaluation of the ACGME Outcomes Project STFM Theme Day - Competencies

33. Questions? STFM Theme Day - Competencies