Research Objective

1. Development and Evaluation of a Cognitive Simulator for Improving the Diagnostic Proficiency of Generalist, Community-Based Pathologist for Malignant Melanoma Dana Grzybicki, PhD, MD1, Rebecca Crowley, MD, MS2, Drazen Jukic, MD, PhD2,3, Stephen Raab, MD1, Olga Medvedeva, MS2, Eugene Tseytlin, MS2, Melissa Castine, BA2 Department of Pathology, University of Colorado Denver Health Sciences Center1, Center for Biomedical Informatics, University of Pittsburgh2, Department of Pathology, UPMC3 Expert mode monitors your work in the background, providing hints and analyzing errors Background Preliminary Findings A correct diagnosis allows you to move onto the next tab Intelligent Tutoring Systems (ITS) are computer-based knowledge communication systems that provide individualized instruction by incorporating methods from artificial intelligence1. Despite many successes in other domains, few intelligent tutoring systems have been developed for use in medical fields2. However, through previous empirical studies we have successfully developed two different tutors, SlideTutor and ReportTutor, that have shown to improve learning among resident pathologists3,4. SlideTutor focuses on teaching the set of diagnostic findings that are needed in coming to a correct final diagnosis, whereas ReportTutor focuses on teaching the relevant prognostic factors that are required to report during the sign-out of a case. In our current system, we have integrated and modified SlideTutor and ReportTutor into a Pathology Simulation Tutor (SimTutor) that provides appropriate guidance and instruction for non-trainee, generalist pathologists with varying expertise in the diagnosis of melanocytic skin lesions. Using Protégé OWL, a much more complex and comprehensive knowledge base was successfully created that can accurately represent difficult melanocytic skin lesions, including lesions previously associated with diagnostic errors. As a result of information derived from pilot studies, extensive modifications to the previous cognitive simulator interface, case authoring tool, and tutor functionality were necessary in order to generate a simulation tool acceptable and user-friendly for practicing pathologists. The new system reflects the higher level of diagnostic expertise, different diagnostic workflow requirements, and changes in tissue section visual and cognitive assessments with increasing time in practice. Based on results from the initial baseline testing scores of the volunteer subjects, a high level of variability in diagnostic expertise for melanocytic skin lesions exists among practicing community pathologists. Novice mode monitors your work and will give immediate feedback for any incorrect action Critical diagnostic errors will immediately show up as red text and enter you into novice mode Research Objective Order additional levels and/or immunohistochemical stains The research objective is to examine the effectiveness of cognitive simulation as a method for improving generalist community practice pathologists’ diagnostic proficiency for difficult pigmented skin lesions. We will determine whether an ITS improves patient safety by decreasing diagnostic errors. Elements of the current system have been previously evaluated and shown to produce dramatic learning gains among pathology residents and fellows3,4. Conclusions Ask for help and get a series of progressive hints that will walk you through the problem We have been able to successfully develop and pilot test a cognitive simulator suitable for use by experienced, practicing pathologists for effectiveness testing as an educational tool to improve histopathologic diagnostic skills for one of the most litigated diagnostic errors in anatomic pathology: false positive or false negative diagnoses of malignant melanoma. General pathologists who have volunteered to serve as subjects in this study show a high level of variability in their level of diagnostic expertise for malignant melanoma. Add multiple authored slides per case Study Design Implications Hint messages will help guide you how to properly evaluate and write the important prognostic features required for a given diagnosis A controlled clinical evaluation of the effectiveness of use of the modified intelligent tutoring system simulator as an educational intervention for improving diagnostic accuracy compared to the currently used conventional forms of continuing education (e.g. reviewing one or more current textbooks, reviewing some current medical literature, and working through publically available online review cases). A group of 20 volunteer private practice, community generalist pathologists, matched for years in practice, estimated annual total number of surgical cases examined, and estimated annual total number of pigmented lesions examined will be randomly assigned to the experimental condition (simulation tutor group) or the control condition (self-study group). Results of the study currently underway will provide a first summative evaluation of a medical cognitive tutoring system. The use of cognitive simulation could be a highly convenient and effective tool for both CME and proficiency testing purposes that could be expanded to many domains in Pathology. References Communication between the diagnostic and prognostic parts of the tutor allow the final diagnosis and findings to be carried over to the report 1. Anderson JR, Corbett AT, Koedinger KR et al. Cognitive tutors: Lessons Learned. J Learning Sci 1995, 4: 167-207. 2. Lillehaug S-I and LaJoie SP. AI in medical education, another grand challenge for medical informatics. AIIM 1998 12: 197-225. 3. Crowley RS, Legowski E, Medvedeva, OM, Tseytlin E, Roh E and Jukic D. Evaluation of an Intelligent Tutoring System in Pathology: Effects of External Representation on Performance Gains, Metacognition, and Acceptance. JAMIA 2007 14(2):182-190. 4. Saadawi GM, Tseytin E, Legowski E, Jukic D, Castine M, and Crowley RS. A natural language intelligent tutoring system for training pathologists: implementation and evaluation. Adv Health Sci Educ Theory Pract. 2007 Oct 13. Experimental Group Timeline Control Group Timeline The report tab of the tutor acts much like a word processor while parsing the text