Enhancing Medical Education: Externally Informed Self-Assessment Program

1. Self-Assessment within a Learner-Prescribed CME/CE Curriculum Destry Sulkes M.D., Vice President, Educational Services Medscape LLC

2. From We Move: Judy Blazer From Outcomes, Inc.: Mazi Abdolrasulnia, PhD Linda Casebeer, PhD Hamid Doroodchi, MD Nancy Roepke, MBA From Medscape LLC: Martin Irvine, PhD Acknowledgements • We wish to acknowledge the following for their contributions and hard work:

3. Disclosure This activity was made possible by an unrestricted educational grant from Boehringer-Ingelheim Pharmaceuticals, Inc.

4. Sometimes things aren’t quite what they seem…

5. Starting Point • CME has been shown to improve knowledge, skills (cognitive) and attitude1 • Self-assessments have been shown to be inaccurate when not externally informed2 • Feedback from externally informed self-assessments, provided to both clinicians and Faculty, can be used to increase effectiveness of CME3 • Evid Rep Technol Assess (Full Rep). 2007 Jan;(149):1-69. Effectiveness of continuing medical education. Marinopoulos SS, Dorman T, Ratanawongsa N, Wilson LM, Ashar BH, Magaziner JL, Miller RG, Thomas PA, Prokopowicz GP, Qayyum R, Bass EB. • JAMA. 2006 Sep 6;296(9):1094-102. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. • Med Teach. 2008 Mar;30(2):124-45.The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10. Colthart I, Bagnall G, Evans A, Allbutt H, Haig A, Illing J, McKinstry B. NHS Education for Scotland.

6. Current Model of Online Medical Education • Medical education programs for healthcare professionals are designed using broad needs assessments across target audiences • Individual activities within courses are not typically developed for specific segments of learners • The courses clinicians select to maintain their professional licensure may or may not be appropriate to address gaps in specific individuals’ clinical knowledge, skills and attitude (K/S/A)

7. Hypothesis for an Externally Informed Self Assessment Learners who participate in an externally informed self-assessment and engage in the corresponding prescribed educational activities are more likely to: • Engage in the educational activities • Enhance their K/S/A relative to those who find the activities in an undirected fashion

8. Methodology • To address K/S/A gaps identified through a needs assessment of Medscape members, an online CME/CE curriculum of activities was developed in collaboration with We Move and Outcomes Inc • One part of the curriculum is a CME/CE-certified self-assessment activity that measures K/S/A related to recognition, diagnosis, and non-pharmacologic and pharmacologic management of RLS • The online self-assessment provides automated activity recommendations based on the learners’ responses to K/S/A questions • Post-assessments follow each activity to assess the relative efficacy of this new educational approach vs the standard, undirected approach, to improving clinical management of RLS

9. Course Formats/Titles/Focus * prescribed courses

10. Curriculum Design

11. Technical Platform Supporting Individualized Feedback Feedback What: • Each learner receives an email listing the recommended activities based on the question responses provided in the self-assessment. How: • Each night, a script that queries the database for any new self-assessment participants. • The script evaluates each participant’s answer choices and maps the choices to the recommended CME activities. • The data is fed to an email application where it is formatted and sent to each participant with individualized activity recommendations, plus a link to the curriculum page.

12. Sample immediate feedback / email message

13. Process Schematic Prescribed Learning: Learner sent recommendation from Externally Informed Self Assessment Automated response directing learner to specific K, S, or A course Post Test A1: Includes mapped Attitude questions from Self Assessment Attitude Course 1 Knowledge Course 1 Post Test K1: Includes mapped Knowledge questions from Self Assessment Learner Undirected Learning: Learner finds curriculum and/or individual activities by standard search criteria Skill Course 1 Post Test S1: Includes mapped Skills questions from Self Assessment

14. Datasets and Minimums Required for Analysis Prescribed Post Assessment A1 N=~100 Undirected Post Assessment: A N= ~1400 over 7 activities Undirected Post Assessment A1 N=~100 Non-Medscape Membership Baseline Survey: N=400 (Identical to E-I self-assessment) Externally Informed Self-Assessment on Medscape: N=400 (Identical to baseline survey) Undirected Post Assessment: K N= ~1400 over 7 activities Prescribed Post Assessment K1 N=~100 Undirected Post Assessment K1 N=~100 Prescribed Post Assessment S1 N=~100 Undirected Post Assessment S1 N=~100 Undirected Post Assessment: S N= ~1400 over 7 activities Outcomes 1 – Measures the courses prescribed vs. the courses actually completed in a three month time period, click-thru rate of A1/K1/S1 prescriptions that resulted in accessing the activities Outcomes 2 – n=~100; Measures the Prescribed Post Assessment to the E. I. Self Assessment by Knowledge, Skills and Attitude Outcomes 3 – n=~100; Measures the Prescribed Post Assessment to the Baseline Survey by Knowledge, Skills and Attitude Outcomes 4 – n=~100; Measures the Prescribed Post Assessment to the Undirected Post Assessment by Knowledge, Skills and Attitude Outcomes 5 – n=~100; Measures the Undirected Post Assessment (A1, K1, S1) to the Baseline survey by Knowledge, Skills and Attitude Outcomes 6 – n=200; Measures the Undirected Post Assessment (4-10) to the Baseline Survey by Knowledge, Skills, Attitude

15. Learnings to date… • K/S/A model helpful and could benefit from further refinement incorporating established competencies • Email feedback is somewhat effective, but Medscape learners access activities primarily from their specialty homepage • 65%: specialty homepage • 25%: outbound email with editorial focus (note: different than the prescriptive emails in this initiative) • 10%: external search • Recognition that current CME/CE activity “tagging” for site-location and rotation purposes can be enhanced to include competencies, thus allowing all activities (>3000 in 2007) to be targeted as needed in the future, perhaps on learners’ specialty homepages, and perhaps without explicitly communicating the targeting