Analysis of Pre-Clinical Student Narratives Progress in Assessment of ACGME Core Competencies

1. Analysis of Pre-Clinical Student Narratives Progress in Assessment of ACGME Core Competencies James S. Powers, MD Josh Denny, MD, MS

2. Geriatric Competencies • ACGME Core Competencies (6) • AAMC Geriatric Competencies (26) 8 domains • IM/FP Geriatric Competencies for residents (26/7) • Challenges in defining, operationalizing, and measuring competencies

3. Reflection (Narratives) • Identify educational needs • Develop clinical reasoning • Address professional challenges • Develop skills to cope with clinical complexity

4. Written Exercises Promote • Understanding of concepts • Ownership of the learning experience • Enhancement of professionalism and communication skills • Age appropriate competencies

5. Narratives-Basis • Piaget & Erikson's constructive nature of identity • Continuous process of evaluation, selection, organization “world making” • Communication of feelings and thoughts makes trainee aware of these • Different type of learning from other coursework

6. Narratives-Benefits • Knowledge gained through process of personalization and discernment of values, hopes, fears, actions • Analysis is qualitative, themes emerge • Provides insight into thinking, maturity, reflective ability • May operationalize competencies

7. Narrative Example: First Year Medical Student Preceptorship-Small Group Discussion Topics • Physicians as patients • Ethical dilemmas • Uncertainty • Cultural attitudes and healthcare • Physician lifestyles • Delivering bad news • The student’s role • Patient compliance • * First Year VU Medical Students after field experience (N=424)

8. Innovations Knowledge Map – a concept-based curriculum management system Learning Portfolio– extends Knowledge Map by to track experience in the clinical years POGOe - www.pogoe.org - a free resource of geriatric education products

9. Natural Language Processing Principle: Education documents and most of clinical narratives are in “natural language” Goal: We want to harness this text for computer information systems Better tracking of what is taught Better knowledge of what a trainee saw Natural Language Processing (NLP) systems convert these “natural language” human language texts into machine-readable data 9

10. Web server Document Conversion Server KM Concept Identifier Document Processing Document uploaded by lecturer, placed in queue Document Conversion Server pulls next document off queue, converts to HTML and Text HTML & PDF versions Text version placed in queue Identified concepts indexed for searching PDF, MS Word, WordPerfect, HTML, PowerPoint, etc

11. Learning Portfolio Electronic Medical Record Section tagging Preprocessor KnowledgeMap Indexer Database of concepts Document indexed by patient Student types a note in the EMR Clinical Portfolio Mentors evaluate and:1. Give feedback2. Assign learning objectives Appropriate mentors emailed Document indexed by student

12. Narrative Example: Geriatric Competency Learning – Medical Clerkship Portfolios • EMR H&P’s of Third Year Medical Students • E-Learning Portfolio (LP) • E-Knowledge Map catalogue of curriculum (KM) • Detect learning opportunities re: Advance Directives and Altered Mentation • Messages to student’s Vanderbilt.edu account in real time with educational prompts

13. Survey of Residents and Attending physicians Analysis of student write-ups • Evaluated 142 write-ups using 13-point rating scale • Performance on “Assessment and Plan” ratings improved in electronic feedback group (p=0.036) Spickard, III, AS, Gigante J, Stein GS, Denny JC. A Randomized Study of Feedback on Student Write-ups Using an Electronic Portfolio. J Gen Int Med. 2008

14. Enhanced Narrative Evaluations • Competency-based “learning opportunities” identified by LP • Trainee messaged by email with link to KM/POGOe • Reference to: control group, pre-post standardized MCE, OSCE (SP)

15. Enhanced Narrative Evaluations • Operationalize competencies • Evaluate competencies • New EMT tools can be utilized to provide just-in-time training as well as evaluate trainees at point of patient care

16. Methods Three geriatric educators independently reviewed the narratives and graded each core competency topic as present or absent.  These ratings served as the reference to which subsequent computer searches were compared. For each ACGME Core Competency we derived a list search terms that mapped to the vocabulary of the Unified Medical Language System We searched student narratives, comparing these results with those of the reference ratings of the narratives for each Core Competency

17. Table 1 A Sample of the UMLS Concepts used for the ACMGE Competency “Systems Based Practice” Activities of Daily Living Advocating (action) Agencies - Case Management Ambulatory care services Arrange for appropriate home care services, as needed Assist patient and/or family in making informed decisions regarding health care Cares, Self (Rehabilitation) Case Management Services Collaborate with the physician, patient/family/significant others, and other health team members in planning for continuity of health care Discharge planning Financial cost Health Care Team

18. Example Perhaps the biggest challenge is lack of exposure to these populations. But by having us meet these patients and work with them, we may become more comfortable and competent in caring for them.

19. Results Table 2 KM Competency Searches for MMC Students (n=100 documents, 2009) for Second Year Course Narratives ACGME # Docs Experts # Docs KM/LP Sens Spec PPV NPV SBP 27 27 0 Pt Care 12 58 . 56 .45 . 35 .75 Profess 73 79 .42 .41 .78 .13 PBLI 29 45 .77 0 .94 0 Med Know 13 38 .40 .64 .44 .58 Comm 15 34 .48 .76 .59 .68 KM/LP KnowledgeMap Learning Portfolio PPV Positive predictive value NPV Negative predictive value

20. Conclusions We found that UMLS lacks rich vocabulary that clearly describes ACGME competencies.   Embellishment of better search terms will be important as institutions seek authentic and efficient means, such as automated searches of electronic portfolios, to monitor trainees’ progress in attaining ACGME competencies